Annif

This release introduces a new [Simplemma analyzer](https://github.com/NatLibFi/Annif/wiki/Analyzers#simplemma-analyzer), support for multiple configuration files in a directory, and support for Python 3.10; support for Python 3.7 is removed.

[Simplemma](https://github.com/adbar/simplemma) is a lightweight multilingual lemmatizer, which currently supports 38 languages; an analyzer based on Simplemma is now implemented as a core feature of Annif. Using multiple project configuration files is made possible by implementing support for a project configuration directory: Annif reads all files matching pattern `*.cfg` and `*.toml` in the directory and merges their contents. The default name of the configuration directory is `projects.d`, but any directory can be selected with `-p/--projects` command option or `ANNIF_PROJECTS` environment variable.

Python 3.10 support is reached by updating multiple dependencies; retraining of existing projects should not be necessary. The language filtering optional feature is not yet available on Python 3.10, because of the lack of support of pycld3 for Python 3.10.

New features:
* 584/585 Support for multiple configuration files in a directory
* 590/591 Add simplemma analyzer

Improvements:
* 589/592 Add Python 3.10 support & update dependencies

Maintenance:
* 594 Upgrade Simplemma to version 0.7
* 587 Update GitHub Actions
* 588 Delete .coveragerc configuration file
* 598 Pin flake8 to version 4.x to avoid pytest-flake8 breakage

Bug fixes:
* 586 Fix readthedocs documentation builds

581/582 Add tags trigger to GH Actions CI/CD workflow

Training of NN ensemble models can now be performed in parallel (running suggest operations simultaneously for all source projects) on multiple CPUs; this is controlled by using the `--jobs` parameter of the `train` command. The compatibility of Annif with [DVC](https://dvc.org/) is improved by supporting TOML file format for configuring Annif projects. The `--force` option is added to the `loadvoc` command that can be used to replace an existing vocabulary instead of updating it. This release includes many small maintenance tasks for the CI/CD pipeline, e.g. migrating Docker image builds to GitHub Actions from the Drone platform.

Omikuji, TensorFlow and Connexion dependencies are upgraded to the latest available versions; retraining of projects should not be necessary.

New features:
526/567 Add --force option to loadvoc CLI command

Improvements:
429/568 Perform suggest operations in parallel using multiprocessing in nn_ensemble
547/560 Support TOML as a configuration file format alongside CFG/INI for DVC compatibility

Maintenance:
570 Use fulltext corpus in MLLM tests which is much faster
571 Docker builds on GitHub Actions CI/CD

550 Drop Python 3.6 support
541 Remove Maui and Vowpal Wabbit multi backends
551 Remove swagger-tester dependency
542/555 Add CITATION.cff file
553 Update Scrutinizer config
561 Set a 10 minute timeout for GitHub Actions CI jobs
565 Avoid coverage 6.3 as it causes some tests to hang

This release introduces a new spaCy analyzer and takes care of many maintenance tasks. The CLI usage is improved by shortening the startup time of some commands, the Docker images are now easier to customize, there are improvements to the eval command, and minor bugs are fixed.

The [spaCy analyzer](https://github.com/NatLibFi/Annif/wiki/Analyzers#spacy-analyzer) enables support for some new languages and can improve subject suggestion results. The spaCy analyzer and the language-specific models need to be [installed separately](https://github.com/NatLibFi/Annif/wiki/Optional-features-and-dependencies#spacy-analyzer). The Docker image distributed via quay.io includes the spaCy analyzer and the English language model, but no other languages.

The maintenance tasks include upgrading many dependencies, notably Omikuji to [v0.4](https://github.com/tomtung/omikuji/releases/tag/v0.4.0). The Omikuji upgrade brings faster training and predictions as well as reduced memory usage, but the Annif projects using the omikuji backend need to be retrained. The projects using other backends should not require retraining, although warnings may be shown in some cases.

The support for Python 3.6 is removed, which is necessitated by the dependency upgrades.

This release also removes the Maui and vw_multi (Vowpal Wabbit) backends.


New features:
374/527/563 spaCy analyzer

Improvements:
514/544 Optimize startup time using local & lazy imports
548 Allow selecting installed optional dependencies in Docker build
545/558 Select metrics for eval command using an option
546/557 Output eval metrics as a JSON file compatible with DVC

Bug fixes:
552/554 LMDB can overflow (credit: mo-fu)
562 Add missing import of annif.eval in MLLM backend

Maintenance:

This release includes a new [language filtering](https://github.com/NatLibFi/Annif/wiki/Transforms#filter_lang-transform) feature. This input-transform filters out sentences of the intput text whose language is different than the project language. The language detection is performed with [Compact Language Detector v3](https://github.com/google/cld3) via [`pycld3`](https://pypi.org/project/pycld3/). `pycld3` is an optional dependency of Annif, see the [installation page](https://github.com/NatLibFi/Annif/wiki/Optional-features-and-dependencies#language-filtering-with-pycld3).

Also minor bug fixes and dependency updates are included.

The [Maui](https://github.com/NatLibFi/Annif/wiki/Backend%3A-Maui) and [vw_multi](https://github.com/NatLibFi/Annif/wiki/Backend%3A-vw_multi) (Vowpal Wabbit) backends have been marked as deprecated in this release and they will be removed in the next release 0.56. Removing is motivated by making codebase more compact and thus easier to maintain. The [MLLM](https://github.com/NatLibFi/Annif/wiki/Backend%3A-MLLM) and [nn_ensemble](https://github.com/NatLibFi/Annif/wiki/Backend%3A-nn_ensemble) backends offer similar functionality as Maui and vw_multi.

Note that the notes for the previous release ([Annif 0.54](https://github.com/NatLibFi/Annif/releases/tag/v0.54.0)) initially missed to mention the added support for the [input-transform](https://github.com/NatLibFi/Annif/wiki/Transforms) feature.


New features:
464/507 Language filtering in input text

Improvements:
536 Allow rdflib version 6.*

Bug fixes:
533/534 Adjust flask and click versions to avoid dependency mismatches

Maintenance:
530 Add deprecation warning to Maui & vw_multi train commands
492/529 Update Docker base image to Debian Bullseye to upgrade Voikko library