Annif

647/661 Order of projects when using project configuration directory

Maintenance:
609/640 Use black code style
641 Use isort to order import statements
656 Install linting tools with Poetry in CI/CD pipeline
624 Increase timeout of test and publish GH Actions jobs
653 Add CodeQL workflow for GitHub code scanning
599/650 Avoid using pytest-flake8 plugin
 657/662 Upgrade GitHub Actions
636 Better set up for docker-compose

This release includes improvements and maintenance updates in particular to the Web UI and REST API as well as some new functionality, especially related to multilingual support. The Web UI no longer relies on jQuery, as the last parts that were used were replaced with Axios. The REST API and Web UI updates are by UnniKohonen, who has joined NatLibFi as a trainee in the Annif & Finto development teams.

It is now possible to override the language for subject suggestion labels instead of always using the project language: when using the `annif suggest` command by giving the new `--language/-L` option, and when using the REST API suggest method by the new optional `language` parameter.

A new resource is added to the root of the REST API (i.e. `http://<annif_host>/v1/`) that gives basic information on the API (a title for the API and the version of Annif being used). Also, the REST API spec has been updated to OpenAPI 3.0. In the Web UI it is now possible to see detailed information about a project (language, backend type, modification timestamp etc.). 

Multiprocessing support for Mac OS and Windows environments has been improved by supporting the 'spawn' multiprocessing mode.

The language detection is now performed with [Simplemma](https://github.com/adbar/simplemma) instead of pycld3. This functionality is now installed by default instead of being an optional extra.

New code style tools Black and isort are now used to help maintaining good code quality; see [CONTRIBUTING.md](https://github.com/NatLibFi/Annif/blob/master/CONTRIBUTING.md) how they can be used and instructions to how best participate in Annif development.

Many dependendencies have been updated to their most recent versions.

Note also that we are preparing for Annif 1.0 release. For this purpose we have opened the [issue 616](https://github.com/NatLibFi/Annif/issues/616) for discussing the expectations of backward compatibility and Semantic Versioning in releases beyond 1.0.


Backward compatibility:

* Models trained with Annif v0.59 should remain working; the warnings by SciKit-learn are harmless
* The `annif loadvoc` command has been removed, as in the previous release it was deprecated and replaced by the `annif load-vocab` command.

New features:
628/630 Allow overriding subject label language in CLI and REST suggest operations
637/638 Add support for spawn multiprocessing mode
654 Add project info to web UI
655/658 Add REST API root resource

Improvements:
593/626 Use Simplemma instead of pycld3 for language detection
643 Add CONTRIBUTING.md file
645 Use tailored user-agent in requests by HTTP-backend
644/649 Upgrade REST API spec to OpenAPI 3
627 Upgrade joblib to 1.2.x

This release makes many changes to how Annif handles vocabularies.

First, the vocabularies are now multilingual: projects with different languages can share the same vocabulary by using a common vocabulary id in the project configurations. The vocabulary id should no longer include a language specifier, which has been the practice until now. The language of the labels of subject suggestions is now defined by the project's language setting, or it can be overridden in a project by giving the language code in parentheses after the vocabulary id (e.g. `vocab=lcsh(en)` in a Finnish language project). These changes break the backward compatibility of existing projects and vocabularies.

The CLI command for loading a vocabulary has changed: the command is now `annif load-vocab` to align with the other annif commands and its first argument is a vocabulary id instead of a project id. When loading a vocabulary from a TSV file the `--language` option needs to be given to set the language. A command `annif list-vocabs` is introduced for listing vocabularies. The old `annif loadvoc` command still works in this release, but it has been deprecated and will be removed in the next Annif release.

The CLI commands are now documented in a [page on the ReadTheDocs](https://annif.readthedocs.io/en/stable/source/commands.html) instead of the Annif wiki. The development installations of Annif now use [Poetry](https://python-poetry.org/) for managing Python virtual environments and dependencies. There are also a few other minor changes, including an upgrade to Simplemma v0.8 series that introduced support for new languages.

Note also that we are starting to prepare for Annif 1.0 release. For this purpose we have opened the [issue 616](https://github.com/NatLibFi/Annif/issues/616) for discussing the expectations of backward compatibility and Semantic Versioning in releases beyond 1.0.

Backward compatibility
The changes in the vocabulary functionality require **reloading of previously loaded vocabularies** and **retraining of existing models**.

New features
559/600 Make vocabularies multilingual
602/614 Implement `load-vocab` and `list-vocabs` commands
603/610 Store vocabs in AnnifRegistry so they are shared between projects
597 Include labels without language tag and concepts without labels in vocabulary

Improvements
617/618 Upgrade to simplemma 0.8 and disable unnecessary cache
595/611 Autogenerated CLI commands documentation on ReadTheDocs
612 Add Annif logo to ReadTheDocs sidebar
608 Multilingual SubjectIndex backed by CSV file
604 Refactor SubjectSuggestion to store subject_id - not uri, label, notation

Maintenance
607 Remove language suffixes from vocabulary ids in example config
606 Refactor SubjectSet and Document to store subject IDs instead of URIs and labels
601/605 Switch to Poetry for dependency management
621 Remove curl from Docker image
622 Remove Poetry cache from Docker image

Fixes
613 Restore ability to use vocab language different from project language
619 Allow use of hyphens in vocabulary IDs
620 Make NN ensemble suggest operations silent

This release introduces a new [Simplemma analyzer](https://github.com/NatLibFi/Annif/wiki/Analyzers#simplemma-analyzer), support for multiple configuration files in a directory, and support for Python 3.10; support for Python 3.7 is removed.

[Simplemma](https://github.com/adbar/simplemma) is a lightweight multilingual lemmatizer, which currently supports 38 languages; an analyzer based on Simplemma is now implemented as a core feature of Annif. Using multiple project configuration files is made possible by implementing support for a project configuration directory: Annif reads all files matching pattern `*.cfg` and `*.toml` in the directory and merges their contents. The default name of the configuration directory is `projects.d`, but any directory can be selected with `-p/--projects` command option or `ANNIF_PROJECTS` environment variable.

Python 3.10 support is reached by updating multiple dependencies; retraining of existing projects should not be necessary. The language filtering optional feature is not yet available on Python 3.10, because of the lack of support of pycld3 for Python 3.10.

New features:
* 584/585 Support for multiple configuration files in a directory
* 590/591 Add simplemma analyzer

Improvements:
* 589/592 Add Python 3.10 support & update dependencies

Maintenance:
* 594 Upgrade Simplemma to version 0.7
* 587 Update GitHub Actions
* 588 Delete .coveragerc configuration file
* 598 Pin flake8 to version 4.x to avoid pytest-flake8 breakage

Bug fixes:
* 586 Fix readthedocs documentation builds

581/582 Add tags trigger to GH Actions CI/CD workflow

Training of NN ensemble models can now be performed in parallel (running suggest operations simultaneously for all source projects) on multiple CPUs; this is controlled by using the `--jobs` parameter of the `train` command. The compatibility of Annif with [DVC](https://dvc.org/) is improved by supporting TOML file format for configuring Annif projects. The `--force` option is added to the `loadvoc` command that can be used to replace an existing vocabulary instead of updating it. This release includes many small maintenance tasks for the CI/CD pipeline, e.g. migrating Docker image builds to GitHub Actions from the Drone platform.

Omikuji, TensorFlow and Connexion dependencies are upgraded to the latest available versions; retraining of projects should not be necessary.

New features:
526/567 Add --force option to loadvoc CLI command

Improvements:
429/568 Perform suggest operations in parallel using multiprocessing in nn_ensemble
547/560 Support TOML as a configuration file format alongside CFG/INI for DVC compatibility

Maintenance:
570 Use fulltext corpus in MLLM tests which is much faster
571 Docker builds on GitHub Actions CI/CD