Djtools

- Bug fixes:
- config: only run arg_parse function if executing from CLI or test suite (i.e. scripts shouldn't inadvertently trigger arg_parse when reusing the build_config function)
- config: make ARTIST_FIRST a global config option so it can be set via the CLI for both sync and utils operations
- copying playlists: when multiple playlists exist with the same name, it was possible for one or more of the playlists to be inadvertently removed from the output collection
- playlist_builder: when ordinary operands and track sets were considered separately, the information about the order in which the different operand types appeared got lost resulting in incorrect evaluation
- playlist_builder: lower precision date inequalities weren't evaluated properly (e.g. {date:>2022} would match a track added any time after 2022-01-01 because `2022` converted into a datetime object becomes 2022-01-01)
- playlist_builder: globbing strings with * characters would match substrings as if implicit ^ and $ characters weren't being used -- use `match` instead of `search` and add implicit `$` character
- playlist_buidler: MinimalDeepTechFilter should only be applied to playlists underneath a "Techno" or "House" playlist
- sync: upload_log logic fix for which files are removed from the logs directory
- sync: handle errors decoding bytes from stdout when syncing (replace bytes that fail decoding with empty strings)
- Fixes for Python 3.8 - 3.12 and Windows compatibility
- CI:
- Pytest and pylint with the latest Python versions for 3.8, 3.9, 3.10, 3.11, and 3.12
- Pytest and pylint on Windows in addition to Ubuntu
- Docs:
- Add tagging guide
- Enhancements:
- playlist_builder: add filtering and aggregation steps to combiner playlists in addition to tags playlists
- playlist_builder: rename "Other" playlist / folder to "Unused Tags"
- playlist_builder: remove pre-existing playlist_builder playlists
- playlist_builder: gracefully handle missing playlists when using playlist selectors
- playlist_builder: gracefully handle exceptions during parsing of combiner expressions
- utils: parallelize audio export in process_recording
- Features:
- playlist_builder: configurable PlaylistName (allows playlist names to be different from their combiner expressions or tags)
- playlist_builder: TransitionTrackFilter and ComplexTrackFilter (playlists from tracks whose comments contain special syntax for BPM or genre transitions, playlists containing tracks with many "other" tags)
- process_recording: trim shift heuristic to automatically remove leading silence of recording
- Refactor:
- collections: split abstractions and implementations for Collection, Playlist, and Track into their own modules
- Scripts:
- cluster_tracks.py: generate playlists by clustering tracks on tag data
- get_tags_from_spotify.py: add interactive mode and --mode flag to switch between interactive and bulk processing
- update_collection.py: sync local Collection data to a remote master Collection


v.2.7.12
ci: format code
fix: remove logs properly

v.2.7.11
Fix packaging errors caused by switch to pyproject.toml.

docs: add developer docs, playlist_filters module
refactor: cleanup the exported members
refactor: show full BaseConfig if not on the CLI execution path
script: rename files
build: update to pyproject.toml

feat: Make `MinimalDeepTechFilter` more robust (https://github.com/a-rich/DJ-Tools/pull/124)

Why?
Successful application of the `MinimalDeepTechFilter` is sensitive to
the relative positioning of tags matching `Techno` (case-insensitive).
Instead, the track should only need to have any tag containing the
substring `techno` to be placed under a playlist called `Techno` and,
similarly, any tag containing the substring `house` to be placed under a
playlist called `House`. Furthermore, if a track matches both substrings
, it should be placed under both playlists.

What?
Check explicitly for `house` and `techno` substrings using regex.
Explicitly identify a `House` parent playlist rather than simply
`not Techno`.

feat: configurable PlaylistFilter classes (https://github.com/a-rich/DJ-Tools/pull/125)

Why?
Hardcoding the `PlaylistFilter` implementations into the
`playlist_builder` will likely cause unexpected behavior for users who
aren't hyper-aware of the source code. Also, as more `PlaylistFilter`
implementations come into existence, it only makes sense to support
configuring them.

What?
Add `COLLECTION_PLAYLIST_FILTERS` to the `collection` config.
Dynamically instantiate the configured `PlaylistFilter` implementations using
`getattr` on `djtools.collection.helpers`.

feat: normalize audio and process recordings (https://github.com/a-rich/DJ-Tools/pull/129)

Why?
Audio files in the Beatcloud should be standardized. Splitting recording
files manually is error prone and time consuming.

What?
Add two new `utils` features:
- `--normalize-audio` which globs `--local-dirs` and normalizes the
peak amplitude using `--normalize-audio-headroom`
- `--process-recording` which takes in `--recording-file` and
`recording-playlist` and chops a file into chunks, normalizes the
peak amplitude, names the files using title and artist data, and
populates the tag data

feat: generalize string selectors (https://github.com/a-rich/DJ-Tools/pull/132)

Why?
There are multiple attributes of tracks that ought to be selectable when
using build_combiner_playlists.

What?
Re-brand the former BPM and rating selector as numerical selectors and
add support for release year.
Re-brand the former playlist selector as string selectors and add
support for:
- artist
- comment
- date added
- key
- label
- playlist

refactor: Collections serialize to same location (https://github.com/a-rich/DJ-Tools/pull/135)

Why?
With the upcoming GUI work item, having to update the COLLECTION_PATH
config with the "auto_" collection output when serializing would be
annoying.

What?
Have collections serialize to the same path they are read from.

feat: supported nested combiner playlists (https://github.com/a-rich/DJ-Tools/pull/141)

Why?
To make organizing those projects easier and more flexible.

What?
Make the parts of the build_combiner_playlists call stack that touch
PlaylistConfigContent.playlists work recursively.

feat(scripts): analyze Rekordbox My Tags by user (102)

Why?
It's cool to see the distribution of tags by users of my Beatcloud.

What?
Group tracks in an XML by the part of the `Location` path that
corresponds with username, search the `Comments` field for the regex
matching `My Tags` data, counting the frequency of each tag, and then
plot histograms per user.

feat(scripts): move "My Tags" data around

Why?
Users would like the ability to do bulk edits to their "My Tags" data.

What?
An r/rekordbox user asked for a way to move "My Tag" data that encodes genre info into the `Genre` field.

I wanted to be able to sort my `Comments` field to sort by energy level, so I added the ability to move particular tags to the front of the `Comments` field.

feat(rekordbox): deprecate registered_users.yaml

Why?
Users shouldn't have to maintain a list of registered users. The whole
point of this file is to facilitate `Location` field manipulations as
part of `rewrite_xml` but, since the `Location` field has a hardcoded
substring of "DJ Music", we can infer all users' `USB_PATH`.

What?
Remove the usage of `registered_users.yaml`.

feat(sync): parameterize track title/artist order (105)

Why?
Some users may already have a collection stored in the `ARTIST_FIRST`
format. These users should still be able to use the `--check-tracks` and
`--playlist-from-upload` features without having to rename all their
files.

What?
Adds the `sync` package configuration option, `ARTIST_FIRST`, which has
Spotify API calls return tracks as `artist - track` instead of
`track - artist`. Additionally, when using with
`CHECK_TRACKS_LOCAL_DIRS`, Beatcloud tracks will have their filenames
temporarily reversed to be the opposite of however they're stored as in
order to facilitate `ARTIST_FIRST` comparison.

feat(version): add single source package version

Why?
A `__version__` variable and `--version` option should be available.

What?
Add a `version.py` module containing a `__version__` variable which is:
- used to setup the package
- exported from the `__init__` module
- printed with the `--version` option

refactor(configs): improve config print upon execution

Why?
Interpreting the config in the console is difficult.

What?
Override the `__repr__` method of `BaseConfig` to print only the
attributes of the sub-class. Format the representation with new lines
and indentations to improve readability.

feat(playlist_builder): Minimal Deep Tech logic (107)

Why?
The genre "Minimal Deep Tech" straddles "House" and "Techno". Users
should be able to control which tracks appear in that genre playlist
depending on some conditions.

What?
A prefix genre tag of "Techno" followed by "Minimal Deep Tech" controls
which tracks belong in playlists called "Minimal Deep Tech" with a parent
playlist called "Techno".

feat(playlist_builder): automatic pure playlists

Why?
Users should not have to configure the playlist builder to apply "pure"
playlist logic.

What?
Automatically find playlists prefixed with "Pure " and feed those
suffixes into the TagParser.

feat(playlist_builder): print ASCII histograms of tag statistics (109)

Why?
To quickly iterate on the proper Combiner playlist expressions, it would
be helpful if the CLI immediately output useful information to help
guide users in the right direction.

What?
Print simple ASCII tables for each TagParser implementation for each
playlist in the set of Combiner playlists generated in the run. Tables
show the frequency of each tag's apperance in each playlist. Tag
frequencies are scaled to a maximum value to constrain table heights.

TODO: Investigate vertical X-axis labels to constrain histogram width.
Investigate splitting histograms into chunks when there is a large
number of X-axis labels.

feat(collections): create collection abstractions

Why?
The `rekordbox` package offers many modules that are only useful to
Rekordbox users because they make assumptions about the structure of
collections, playlists, and tracks. Abstractions should be created for
these structures so that implementations can be made for different DJ
software platforms. This decouples the database (de)serialization logic
from the modules like the `PlaylistBuilder`.

What?
Rename the `rekordbox` package to `collection`. Create abstract classes
for Collection, Playlist, and Track. Implement these abstractions for the
Rekordbox use-case. Replace all Rekordbox-specific implementation
details in the `collection` package with the methods defined on the
abstractions. Add a configuration option for DJ software.

Renamed a bunch of the CLI args.

fix: More elegant music sync (116)

Why?
The carriage return prints made by `awscli` which contain information
about the amount of data and number of files transferred was getting
lost due to the use of `readline()` which splits on "\n" characters.

What?
Read stdout one character at a time so the carriage return data can be
displayed.

Also remove the usage of `shell=True` from all `Popen` calls.

refactor: CLI sub-commands (117)

Why?
The `--help` menu is verbose and hard to read.

What?
Break down all the CLI args into sub-commands based on the package they
pertain to.

refactor: Filter list args and parse JSON args

Why?
Previously, the `--spotify-playlist-subreddits` arg would accept YAML
strings and parse them into dictionaries. Formatting an arg as YAML is
annoying since whitespace matters. JSON is much easier to use.

Overriding list type args that have a default value was not possible
without editing the config that defines that default value.

What?
Revert `parse_yaml` back to `parse_json` for the
`--spotify-playlists-subreddits` argument.

Iterate parsed args and apply a `list(filter(None, ...` expression to
values that are an instance of `list`.

* [FIX] Delay playlist selector lookup creation (86)

Why?
Combiner playlists that include playlist selectors previously did not include tracks that were added to those playlists in the same run.

What?
Delay the creation of the playlist selector tag -> track lookup until after any other TagParser implementations have been run.

In addition, this MR inserts all auto-playlists into a folder called AUTO_PLAYLISTS so as to make it simpler to bulk import these playlists.

* [FIX] Catch async.exceptions.TimeoutErrors (89)

Why?
The underlying `aiohttp` calls for the `spotify.playlist_builder` module
occasionally timeout. These `TimeoutError` exceptions need to be caught
so that potentially successful Reddit submissions retrievals can occur.

What?
Wrap the `asyncpraw` method that returns an AsyncGenerator with another
generator which applies try / except logic that differentiates between
`StopAsyncIteration` exceptions and other exceptions.

* Optimize Pytest suite (91)

Why?
Testing time should be minimized.

What?
Factor out `rekordbox.xml` loading into a session scoped fixture. Use
`tmpdir_factory` to create a temporary directory which may be session scoped;
use this temporary directory, in combination with the factored out
`rekordbox.xml` loading fixture, in the `test_xml` fixture. Mock `os.system`
calls in `test_upload_log` and `test_download_xml`. Minimize the amount of
content in `rekordbox.xml` so as to speed up reading and writing that file.

* [ENHANCEMENT] Replace os.path with pathlib.Path (93)

Why?
When looking at code using os.path operations, developers have to
evaluate expressions the same way as the Python interpreter
(inside -> out). By comparison, pathlib.Path uses method chaining which
is much more intuitive. In addition, pathlib.Path accounts for the
eccentricities of Windows paths under the hood; this means all the
ridiculous `.replace(os.sep, "/")` calls can be removed.

What?
Replace all uses, where appropriate, of `os.path` with `pathlib.Path`.

* [ENHANCEMENT] Collapse djtools directory (97)

Why?
test_data was in the src directory creating an unnecessary level
between the root and the library source.

What?
Move test_data to the project root, collapse src/djtools into
djtools.

* [ENHANCEMENT] Better logging for check-tracks (98)

Why?
The `--check-tracks` feature logs too verbosely and also omits info that
would be nice to have such as the count of local files. There's also an
uncaught error when trying to zip emtpy lists of tracks.

What?
Simplify logging and include local file counts. Skip comparison loop if
there aren't any tracks.

* [ENHANCEMENT] Diataxis docs

Why?
There are no docs!

What?
Create docs. Use the Diataxis framework to organize the docs and use
`mkdocs` and `mkdocstring` to generate the docs. Host the docs on GitHub
Pages. Added a `.pylintrc` and linted the entire repo's `.py` files.

A few functions were renamed for clarity.
* `randomize_playlists` is not `shuffle_playlists`
* `set_tag` is now `set_track_number`
* `rekordbox_playlists` is now `build_playlists`
* `get_playlist_track_locations` is now `get_playlist_tracks`

The `Combiner` class was removed from `tag_parsers.py`, made to not inherit from `TagParser`, and moved into it's own module `playlist_combiner.py`.

[ENHANCEMENT] Optimize Pytest suite

Why?
Testing time should be minimized.

What?
Factor out `rekordbox.xml` loading into a session scoped fixture. Use
`tmpdir_factory` to create a temporary directory which may be session scoped;
use this temporary directory, in combination with the factored out
`rekordbox.xml` loading fixture, in the `test_xml` fixture. Mock `os.system`
calls in `test_upload_log` and `test_download_xml`. Minimize the amount of
content in `rekordbox.xml` so as to speed up reading and writing that file.