Djtools

[ENHANCEMENT] Optimize Pytest suite

Why?
Testing time should be minimized.

What?
Factor out `rekordbox.xml` loading into a session scoped fixture. Use
`tmpdir_factory` to create a temporary directory which may be session scoped;
use this temporary directory, in combination with the factored out
`rekordbox.xml` loading fixture, in the `test_xml` fixture. Mock `os.system`
calls in `test_upload_log` and `test_download_xml`. Minimize the amount of
content in `rekordbox.xml` so as to speed up reading and writing that file.

[FIX] Catch async.exceptions.TimeoutErrors (https://github.com/a-rich/DJ-Tools/pull/89)

Why?
The underlying `aiohttp` calls for the `spotify.playlist_builder` module
occasionally timeout. These `TimeoutError` exceptions need to be caught
so that potentially successful Reddit submissions retrievals can occur.

What?
Wrap the `asyncpraw` method that returns an AsyncGenerator with another
generator which applies try / except logic that differentiates between
`StopAsyncIteration` exceptions and other exceptions.

Why?
Combiner playlists that include playlist selectors previously did not include tracks that were added to those playlists in the same run.

What?
Delay the creation of the playlist selector tag -> track lookup until after any other TagParser implementations have been run.

In addition, this MR inserts all auto-playlists into a folder called AUTO_PLAYLISTS so as to make it simpler to bulk import these playlists.

Fixed broken image links for READMEs.

* [FEATURE] Add script for moving Comments field "My Tag" data to Genre field

Why?
Some users may have captured Genre information in the "My Tag" system
which adds clutter to their tracks' Comments fields.

What?
Given a list of Genre tags that are populated by the My Tags system and
written to the Comments field, write those tags to the Genre field.

Note: these tags aren't removed from the Comments field since that can
easily be done within the Rekordbox software itself.

* [FEATURE] Create Spotify playlist from Discord webhook output

Why?
Users may want to preview the tracks uploaded by another user so they
can choose which, if any, they want to download.

What?
Adds a `playlist_from_upload` function to the
`spotify.spotify_playlist_builder` module.

This function can read the output of the Discord webhook from the system
clipboard (or from a file) and search Spotify for those tracks creating
a playlist called "<USER> Uploads" where <USER> is a top-level user
directory sampled from the webhook output.

A new configuration option `PLAYLIST_FROM_UPLOAD` has been added which
accepts a boolean or a file path. Discord webhook output must be in the
system clipboard if set to `true`. If set to a file path, that file
obviously must exist and contain the output of the Discord webhook.

* [FEATURE] Download Beatcloud tracks from Spotify a playlist

Why?
Users may not want to download the entire upload of another user.

What?
Given a Spotify playlist with a subset of tracks from a user's upload,
perhaps generated with `PLAYLIST_FROM_UPLOAD`, compare those tracks
with Beatcloud files and add the paths of matching tracks to
`DOWNLOAD_INCLUDE_DIRS`.

* [ENHANCEMENT] Refactor packages and configuration

Why?
The membership of functions to their parent subpackages was misplaced
in many places. Also, the primary modules accessed as features from the
top-level should be stripped down to the core functions with all other functions
moved into helper modules. Reorganizing things to minimize the amount of
cross-package dependency, and reducing the amount of code in the
feature-oriented modules, makes the code easier to read and maintain.

`configs` should be made a first-class package with config building,
validation, and CLI argument parsing included instead of in living in `utils`.

The inter-package dependencies should be minimized to improve readability and
maintainability of the code base; this also improves usability for end-users as
it improves clarity with respect to which config options / CLI args map to
which features of the library.

Configuration files should be in YAML format rather than JSON because YAML
requires fewer characters and is easy to read and write. This reduces the
surface area for user-error when editing the configuration files.

Pydantic models should be used to perform validation at the point of loading
automatically and encourage separation of concern between the different
packages' configuration options.

Main function needs unit testing.

What?
Isolate code between the different packages as much as possible. Move all
non-core functions in the feature-oriented modules into helper modules.

Make `configs` a first-class Python package. Move the config-related utilities
from the `utils` folder to `configs.helpers`.

Move `COPY_TRACKS` feature from `utils` to `rekordbox`. Rename configuration
options for more clarity and consistency. Replace underscore characters in CLI
args with dashes to make them easier to type.

Replaced all JSON configuration files with equivalent YAML files. `config.yaml`
options are now grouped by their respective package. Options that either do not
apply to a particular package or else apply to more than one package are
assigned to the `configs` package.

Each package's configuration options are loaded into Pydantic models that
perform validation. All exception handling that can be validated at the point
of loading (which is almost all of it) is not being performed during Pydantic
model loading instead of in the `build_config` function or at the points of
execution.

Refactor the main function out of `dj_tools.py` and into a separate module.

* [BUGFIX] Fix python-Levenshtein package version

Why?
A pip bug is causing a name conflict when importing python-Levenshtein.

What?
Fix python-Levenshtein version at 0.12.2.

More info:
https://github.com/maxbachmann/python-Levenshtein/issues/1

(1) `rekordbox` package

Why?
Creating a package the distinguishes Rekordbox operations from platform-
agnostic operations encourages development that focuses on each category.

What?
Relocate modules that operate on Rekordbox's XML database files
(`utils.randomize_tracks` and `utils.generate_genre_playlists`) to a
`rekordbox` package.

Proposal:
In order to port the functionality of the `rekordbox` package for, e.g., Serato
users, a new package called `serializers` should be created which is
responsible for converting the databases of various DJ software tools into an
XML format that allows the existing features of the `rekordbox` package to be
reused.

(2) `MyTagParser` playlist builder

Why?
Automatically generating playlists from Rekordbox's "My Tag" system makes
these tags portable so they can be accessed on CDJs, other users' laptops,
or even non-Pioneer systems via the `utils.copy_playlists_tracks` module. In
addition, parsing "My Tag" data allows these tags to be used with the
`Combiner`.

What?
Specifies an implementation of the `TagParser` abstraction which uses
regular expressions to parse out the "My Tag" data written to the Comments
field.

NOTE: "My Tag" data is only written to the Comments field if
`Preferences > Advanced > Browse > Add "My Tag" to the "Comments"` is
enabled.

Including the key `MyTagParser` in `configs/rekordbox_playlists.json` allows
users to specify a desired playlist folder structure for "My Tag" data. The
supported feature set is the same as with the `generate_genre_playlists`
module which has been rebranded as the `GenreTagParser`.

The `MyTagParser` returns a list of tags which the calling class,
`rekordbox.rekordbox_playlist_builder.PlaylistBuilder` uses to assemble a
tag -> track ID lookup data structure which is later used to generate
Rekordbox playlists.

(3) `GenreTagParser` playlist builder

Why?
To be consistent with the design of the `TagParser` abstraction in the
`rekordbox.tag_parsers` module, used by the
`rekordbox.rekordbox_playlist_builder` module, the
`generate_genre_playlists` module was refactored.

What?
Specifies an implementation of the `TagParser` abstraction which splits the
Genre field on `/` forward slash characters.

Including the key `GenreTagParser` in `configs/rekordbox_playlists.json`
allows users to specify a desired playlist folder structure for Genre data. The
supported features set is the same as with the `generate_genre_playlists`
module.

The `GenreTagParser` returns a list of tags which the calling class,
`rekordbox.rekordbox_playlist_builder.PlaylistBuilder` uses to assemble a
tag -> track ID lookup data structure which is later used to generate
Rekordbox playlists.

NOTE: the former options `GENERATE_GENRE_PLAYLISTS` and
`GENERATE_GENRE_PLAYLISTS_REMAINDER` have been renamed
`REKORDBOX_PLAYLISTS` and `REKORDBOX_PLAYLISTS_REMAINDER`
respectively. These options now apply to all `TagParser` implementations.
The `GENERATE_GENRE_PLAYLISTS_PURE` option has been renamed to
`GENRE_PLAYLISTS_PURE`. Additionally, the option
`GENRE_TAG_DELIMITER` has been removed and the delimiter `/` is assumed
to be consistent with the in-built "My Tag" delimiter.

(4) `Combiner` playlist builder

Why?
Rekordbox's "Track Filter" system is very limited. The `Combiner` greatly
improves upon the "Track Filter" in the following ways:

(a) Portability:
The "Track Filter" can only be used on a laptop running Rekordbox and,
furthermore, can only be used with the Master Database USB meaning that
using your USB as a Device on someone else's laptop bars you from
accessing this feature. The `Combiner` generates ordinary Rekordbox
playlists which can be exported to a Device. You can even combine these
generated playlists with the `utils.copy_playlists_tracks` module to enable
easier access to tracks on non-Pioneer systems giving users maximum
portability.

(b) Operands:
The "Track Filter" only operates on "My Tag" data. The `Combiner` can be
extended to operate on any data that can be parsed from a Rekordbox XML.
Currently supported operands are:
- Genre tags parsed with the `GenreTagParser`, e.g. `Liquid Funk`
- "My Tag" tags parsed with the `MyTagParser`, e.g. `Peak Time`
- Playlist selectors, e.g. `{My Favorite Tracks}`
- Rating selectors (0 <= numbers <= 5), e.g. `[0, 2-5]`
- BPMs selectors (numbers > 5), e.g. `[80-88, 140, 150-174]`

(c) Operators:
The "Track Filter" only supports `AND` and `OR` logic. The `Combiner`
adds the `NOT` operator into the mix which allows users to subtract tracks
belonging to the successive operand from the resulting playlists.

(d) Managed complexity:
The "Track Filter" only supports filtering a maximum of three tags using two
logic operators. The `Combiner` allows users to write boolean algebra
expressions with an arbitrary number of operands / operators to create
incredibly rich filtering logic. Subexpressions are enclosed in parentheses to
give users clear control over the order of operations in their playlist building
logic.

What?
Including the key `Combiner` in `configs/rekordbox_playlists.json` allows
users to specify a desired set of `Combiner` playlists. Unlike the other playlist
builders based on `TagParser` implementations, the `Combiner` does not
support nested folders, the "Other" playlist / folder, or the "_ignore" key.
Instead, users must specify "playlists" as a flat list of playlist names which
are, themselves, the boolean algebra expressions they wish to be evaluated to
generate the desired playlists.

The `rekordbox.rekordbox_playlist_builder.PlaylistBuilder` class manages the
overall construction of Rekordbox playlists by utilizing the various `TagParser`
implementations to extract tag information from a Rekordbox XML. The way
this class handles the `Combiner` is a bit different. It first instantiates the
`Combiner`, employs the set of `TagParser` instances, and then applies the
`Combiner` on the reduced data structure generated by the `TagParser`
instances. When the `Combiner` is initialized, it checks if any of the
`Combiner` playlists contain selectors (playlist, rating, and BPM) and, if so,
performs a pre-scan of the tracks to initialize the data structure with the
proper track IDs treating each selector as a tag for lookup.

Once the tag -> track ID lookup is completely assembled, each `Combiner`
playlist has its name evaluated as a boolean logic parse tree. Parentheses
symbolize subexpressions which create new child nodes in the parse tree. The
operators `&`, `|`, `~` (i.e. `AND`, `OR`, and `NOT`, respectively) are
appended to a list of operators belonging to a node. All other stripped values
between operators / parentheses are appended as tags to be looked up in the
tag -> track ID data structure. When a (sub)expression is terminated by a
closing parenthesis, or by reaching the end of the playlist name, the
(sub)expression is evaluated by dequeuing tags and operators from the
respective lists belonging to that node. Sets of track IDs evaluated in
subexpressions are propagated upward towards the root of the parse tree
until a final set of track IDs is returned representing the fully evaluated
`Combiner` playlist.

(5) Removed `utils.get_genres` module and the related `config.json` options

Why?
Getting genre information from the ID3 tag fields of MP3s is not useful as that
information already exists in the Rekordbox XML.

What?
Remove the `utils.get_genres` module and related `config.json` options.

(6) Add `utils.copy_playlists_tracks` module

Why?
Users may need to create an additional USB containing just the tracks from
one or more playlists. For example, one may need to create a backup for an
event, provide a USB with tracks to another DJ for a back-to-back, etc.

What?
Searches `XML_PATH` for each playlist name in `COPY_PLAYLISTS_TRACKS`
(NOTE: if playlist names are not unique, the first playlist found will be used).
Audio files stored at the tracks' `Location` field are copied to
`COPY_PLAYLISTS_TRACKS_DESTINATION`. A new XML file called
`relocated_<NAME>`, where `<NAME>` is the basename of `XML_PATH`, is
generated containing only these copied tracks with updated `Location` fields
that point to `COPY_PLAYLISTS_TRACKS_DESTINATION`.

New `config.json` options: `COPY_PLAYLISTS_TRACKS` and
`COPY_PLAYLISTS_TRACKS_DESTINATION`.

(7) Add parameterized download location

Why?
Users should not need to have their `USB_PATH` existing to use the
`utils.youtube_dl` module.

What?
Adds the option `YOUTUBE_DL_LOCATION` to `configs/config.json`. If left
empty, files will be downloaded to the current directory.

(8) Add unit tests and `pytest-cov` Action

Why?
Unit testing is obviously a good thing. As other users begin contributing, unit
testing becomes more essential. Ensuring 100% coverage on `push` and
`pull_request` events reduces / eliminates the need to manually test features
and bugfixes or manage beta release periods.

What?
Add `rekordbox.xml` test data and unit tests for 100% total coverage.
Additionally, `.github/workflows.test.yaml` has been added to trigger a
`pytest-cov` runner upon `push` and `pull_request` events.

Hot-fix for tuple paths