Ms3

**ms3 requires Python 3.10**

Extraction of all lyrics

This version enables the extraction of lyrics with all verses. Previously, only the last verse\'s syllable for any given position was extracted. The lyrics now can be found in [lyrics\_\[verse\]]{.title-ref} columns in the chords facet, where [lyrics_1]{.title-ref} corresponds to the first or only verse.

Extraction of figured bass

Figured bass labels can now be found in the chords facet tables. Score that include at least one figure will have a `thoroughbass_duration` column and each layer of figures comes in a separate `thoroughbass_layer_` column. For example, if all stacks of figures have only layer, there will be only the column `thoroughbass_layer_1`.

Extraction of custom-named spanners

Spanners with adjusted \"Begin text\" property get their own columns in the chords tables, containing the relevant subselection of IDs. For example, if a score contains normal `8va` spanners and others where the \"Begin text\" has been set to `custom`, all IDs will be combined in the column `Ottava:8va` as before, but the subset pertaining to the custom spanners is additionally shown in the column `Ottava:8va_custom`.

Including and excluding paths

It is now possible to specify specific directories to be included or excluded from a view, not only folder names.

New methods and properties

- `Parse.get_facet()` (singular)
- `Corpus.fnames`
- `Corpus.add_dir()`
- first version of `utils.merge_chords_and_notes()`

**ms3 requires Python 3.10**

* Refines the new "writing score headers" functionality and makes it non-default. User needs to
set ``ms3 metadata --prelims`` which replaces the flag ``--ignore`` that had been introduced in 1.1.1.
* A couple of bug fixes, including a very important one regarding conversion of fifths introduced with b0ce8a1d

* enables updating score headers from the respective `metadata.tsv` columns
(`title_text`, `subtitle_text`, `composer_text`, `lyricist_text`, and `part_name_text`)
* Parse, Corpus, and Piece now come with the method keys()

This version does not throw errors when trying to parse files created by MuseScore 4. Parsing these files has not
sufficiently been tested but so far it was looking good. The fact that MuseScore 3 is able to read such files shows
that not much has changed in the file format itself.

The command ``ms3 convert`` has been updated to support MuseScore 4 executables. With the current MuseScore 4.0.0 this
is not quite straightforward because conversion to ``.mscz`` via the commandline isn't currently working and conversion
to ``.mscx``, if it works at all, deletes the contents of the target directory
([issue 15367](https://github.com/musescore/MuseScore/issues/15367#issuecomment-1369783686)). The new function
``utils.convert_to_ms4()`` offers a workaround that creates temporary directories to store the
"Uncompressed MuseScore folder" and then copies the ``.mscx`` file to the target directory (default) or zips the
temporary directory into an ``.mscz`` file (parameter ``--format mscz``). For all other target formats, the output
will correspond to what the MuseScore 4 executable yields.

**ms3 requires Python 3.10**

ms3 has gotten a makeover and does not quite like it did before. The major changes are:

- The library is now optimized for one particular folder structure, namely `[meta-corpus ->] corpus -> piece`.
- ms3 now comes with a full-fledged \"views\" feature which lets you subselect files in manifold ways.
- The TSV outputs have gained additional columns. In particular, all TSV files now come with the column `quarterbeats` reflecting each event\'s offset from the piece\'s beginning.
- Warnings concerning irregularities, e.g. wrong measure numbering due to a cadenza, can now be sanctioned by copying them into an IGNORED_WARNINGS file.

New features

- Each object that the user interacts with, `Parse, Corpus, and Piece`, comes with at least two views, called \"default\" and \"all\". The \"default\" view disregards review files, scores in convertible formats, and scores that are not listed in the top-level `metadata.tsv` file.
- `metadata.tsv` files, by the virtue of their first column `fname`, now serve as authority on what is included in the corpus and what belongs together. This column is always unique and supposed to be used as index.
- Suffixed `metadata_<suffix>.tsv` files are loaded as available views based on the column `fname` (other columns are disregarded).
- The Parse object now detects if the passed directory contains individual corpora or if it is a corpus itself.
- Parse objects perform operations by iterating over Corpus objects.
- Corpus objects perform operations by iterating over Piece objects.
- Corpus objects reflect exactly one folder, the `corpus_path`, and always discover all present files (which can be filtered before the actual parsing). Default output paths are derived from it.
- Piece objects unite the various files pertaining to the same `fname` and are able to keep multiple versions of the same type apart (e.g., scores or annotation files) and pick one automatically, if necessary, or ask for user input.
- The command `ms3 review` combines the functionalities of `ms3 check`, `ms3 extract`, and `ms3 compare`, and is now the only command used in the new `dcml_corpus_workflow` action. For each score that has DCML harmony labels, it stores another score and TSV file with the suffix `_reviewed` in the folder `reviewed`.
- The score has all out-of-label tones colored in red and
- the TSV file contains a report on this coloring procedure. Both files are stored in the folder `reviewed` on the top level of the corpus.
- **(1.0.2)** In addition, if any warnings pop up, they are stored in the top-level `warnings.log` file.
- Inserting labels into scores is accomplished using the new method `load_facet_into_scores()` which comes with the optional parameter `git_revision` which allows loading TSVs from a specific commit.
- Therefore, `ms3 compare` (and hence, `ms3 review`) is now able to compare the labels in a score with those in a TSV file from an older git revision.
- `ms3 extract -F` extracts form labels and expands them into a tree-like view in the output TSV.

Changes to the interface

- Many things have been renamed for the benefit of a more homogeneous user interface.
- Methods previously beginning with `output_` were renamed to `store_`.
- Parse.parse_mscx() =\> Parse.parse_scores()
- The properties for retrieving DataFrames from `Score` objects:
- are now methods and accept the parameters `unfold` and `interval_index`.
- return None when a facet is not available.
- Parsed scores and dataframes are always returned with File object that identifies the parsed file in question. This is particularly relevant when using the `get_facet()` methods that may return facets from parsed TSV files or extract them from the scores, according to availability.
- Gets rid of the argument `simulate` except for writing files.
- logger_cfg now as \*\*kwargs
- **(1.0.3)** Currently the ``-d/--dir`` argument to ``ms3`` commands accepts only one directory, not several.


Changes to the outputs

- **(1.0.1)** When unfolding repeats, add the column `mn_playthrough` with disambiguated measure Numbers (\'1a\', \'12b\', etc.).
- The column `label_type` has been replaced and disambiguated into `harmony_layer` (0-3, text, Roman numeral, Nashville, guitar chord) and `regex_match` (containing the name of the regular expression that matched first).
- Notes tables now come with the two additional columns `name` (e.g. \"E4\") and `octave`. For unpitched instruments, such as drumset, the column `name` displays the designated instrument name (which the user can modify in MuseScore), and have no value in the `octave` columns.
- For pieces that don\'t have first and second endings, the TSVs come without a `volta` column.
- Extracted metadata
- **(1.0.1)** come with the new columns last_mc_unfolded, last_mn_unfolded, volta_mcs, guitar_chord_count, form_label_count, ms3_version, has_drumset
- uses the column `fname` as index
- comes with a modified column order
- renames the previous column `rel_paths` to subdir, whereas the new column `rel_path` contains
- include the text fields included in a score. Columns are `composer_text`, `title_text`, `subtitle_text`, `lyricist_text`, `part_name_text`.
- Upon a full parse (i.e. if the view has default settings), each facet folder gets a `csv-metadata.json` file following the CSVW standard. This file indicates the version of ms3 that was used to extract the facets. The version is also included in the last row of the README.

Other changes

Many, many bugs have died on the way. Also:

- Most functions and methods now come with type hints.
- New unittest suite that makes use of the DCMLab/unittest_metacorpus repo and enforces it to be at the correct commit.
- The parser is now more robust against user-induced strangeness in MuseScore files.
- **(1.0.1)** Repetitions are unfolded for checking the integrity of DCML phrase annotations in order to deal with voltas correctly.
- **(1.0.3)** Pedal notes that have multiple (volta) endings, although still not being correctly propagated into each ending,
get propagated into the first ending, and don't cause propagation nor the integrity check to fail anymore

**ms3 requires Python 3.10**

ms3 has gotten a makeover and does not quite like it did before. The major changes are:

- The library is now optimized for one particular folder structure, namely `[meta-corpus ->] corpus -> piece`.
- ms3 now comes with a full-fledged \"views\" feature which lets you subselect files in manifold ways.
- The TSV outputs have gained additional columns. In particular, all TSV files now come with the column `quarterbeats` reflecting each event\'s offset from the piece\'s beginning.
- Warnings concerning irregularities, e.g. wrong measure numbering due to a cadenza, can now be sanctioned by copying them into an IGNORED_WARNINGS file.

New features

- Each object that the user interacts with, `Parse, Corpus, and Piece`, comes with at least two views, called \"default\" and \"all\". The \"default\" view disregards review files, scores in convertible formats, and scores that are not listed in the top-level `metadata.tsv` file.
- `metadata.tsv` files, by the virtue of their first column `fname`, now serve as authority on what is included in the corpus and what belongs together. This column is always unique and supposed to be used as index.
- Suffixed `metadata_<suffix>.tsv` files are loaded as available views based on the column `fname` (other columns are disregarded).
- The Parse object now detects if the passed directory contains individual corpora or if it is a corpus itself.
- Parse objects perform operations by iterating over Corpus objects.
- Corpus objects perform operations by iterating over Piece objects.
- Corpus objects reflect exactly one folder, the `corpus_path`, and always discover all present files (which can be filtered before the actual parsing). Default output paths are derived from it.
- Piece objects unite the various files pertaining to the same `fname` and are able to keep multiple versions of the same type apart (e.g., scores or annotation files) and pick one automatically, if necessary, or ask for user input.
- The command `ms3 review` combines the functionalities of `ms3 check`, `ms3 extract`, and `ms3 compare`, and is now the only command used in the new `dcml_corpus_workflow` action. For each score that has DCML harmony labels, it stores another score and TSV file with the suffix `_reviewed` in the folder `reviewed`.
- The score has all out-of-label tones colored in red and
- the TSV file contains a report on this coloring procedure. Both files are stored in the folder `reviewed` on the top level of the corpus.
- **(1.0.2)** In addition, if any warnings pop up, they are stored in the top-level `warnings.log` file.
- Inserting labels into scores is accomplished using the new method `load_facet_into_scores()` which comes with the optional parameter `git_revision` which allows loading TSVs from a specific commit.
- Therefore, `ms3 compare` (and hence, `ms3 review`) is now able to compare the labels in a score with those in a TSV file from an older git revision.
- `ms3 extract -F` extracts form labels and expands them into a tree-like view in the output TSV.

Changes to the interface

- Many things have been renamed for the benefit of a more homogeneous user interface.
- Methods previously beginning with `output_` were renamed to `store_`.
- Parse.parse_mscx() =\> Parse.parse_scores()
- The properties for retrieving DataFrames from `Score` objects:
- are now methods and accept the parameters `unfold` and `interval_index`.
- return None when a facet is not available.
- Parsed scores and dataframes are always returned with File object that identifies the parsed file in question. This is particularly relevant when using the `get_facet()` methods that may return facets from parsed TSV files or extract them from the scores, according to availability.
- Gets rid of the argument `simulate` except for writing files.
- logger_cfg now as \*\*kwargs
- **(1.0.3)** Currently the ``-d/--dir`` argument to ``ms3`` commands accepts only one directory, not several.


Changes to the outputs

- **(1.0.1)** When unfolding repeats, add the column `mn_playthrough` with disambiguated measure Numbers (\'1a\', \'12b\', etc.).
- The column `label_type` has been replaced and disambiguated into `harmony_layer` (0-3, text, Roman numeral, Nashville, guitar chord) and `regex_match` (containing the name of the regular expression that matched first).
- Notes tables now come with the two additional columns `name` (e.g. \"E4\") and `octave`. For unpitched instruments, such as drumset, the column `name` displays the designated instrument name (which the user can modify in MuseScore), and have no value in the `octave` columns.
- For pieces that don\'t have first and second endings, the TSVs come without a `volta` column.
- Extracted metadata
- **(1.0.1)** come with the new columns last_mc_unfolded, last_mn_unfolded, volta_mcs, guitar_chord_count, form_label_count, ms3_version, has_drumset
- uses the column `fname` as index
- comes with a modified column order
- renames the previous column `rel_paths` to subdir, whereas the new column `rel_path` contains
- include the text fields included in a score. Columns are `composer_text`, `title_text`, `subtitle_text`, `lyricist_text`, `part_name_text`.
- Upon a full parse (i.e. if the view has default settings), each facet folder gets a `csv-metadata.json` file following the CSVW standard. This file indicates the version of ms3 that was used to extract the facets. The version is also included in the last row of the README.

Other changes

Many, many bugs have died on the way. Also:

- Most functions and methods now come with type hints.
- New unittest suite that makes use of the DCMLab/unittest_metacorpus repo and enforces it to be at the correct commit.
- The parser is now more robust against user-induced strangeness in MuseScore files.
- **(1.0.1)** Repetitions are unfolded for checking the integrity of DCML phrase annotations in order to deal with voltas correctly.
- **(1.0.3)** Pedal notes that have multiple (volta) endings, although still not being correctly propagated into each ending,
get propagated into the first ending, and don't cause propagation nor the integrity check to fail anymore