Fractal-server

* API:
* Validate all user-provided strings that end up in pip-install commands (\2003).

* Task collection
* Now both the local and SSH versions of the task collection use the bash templates (\1980).
* Update task-collections database logs incrementally (\1980).
* Add `TaskGroupV2.pinned_package_versions_string` property (\1980).
* Support pinned-package versions for SSH task collection (\1980).
* Now `pip install` uses `--no-cache` (\1980).
* API
* Deprecate the `verbose` query parameter in `GET /api/v2/task/collect/{state_id}/` (\1980).
* Add `project_dir` attribute to `UserSettings` (\1990).
* Set a default for `DatasetV2.zarr_dir` (\1990).
* Combine the `args_schema_parallel` and `args_schema_non_parallel` query parameters in `GET /api/v2/task/` into a single parameter `args_schema` (\1998).

> WARNING: As of this version, all extras for `pip install` are deprecated and
> the corresponding dependencies become required.

* Database:
* Drop `TaskV2.owner` column (\1977).
* Make `TaskV2.taskgroupv2_id` column required (\1977).
* Dependencies:
* Make `psycopg[binary]` dependency required, and drop `postgres-pyscopg-binary` extra (\1970).
* Make `gunicorn` dependency required, and drop `gunicorn` extra (\1970).
* Testing:
* Switch from SQLite to Postgres in the OAuth Github action (\1981).

> WARNING: This release comes with several specific notes:
>
> 1. It requires running `fractalctl update-db-data` (after `fractalctl set-db`).
> 2. When running `fractalctl update-db-data`, the environment variable
> `FRACTAL_V27_DEFAULT_USER_EMAIL` must be set, e.g. as in
> `FRACTAL_V27_DEFAULT_USER_EMAIL=adminfractal.yx fractalctl
> update-db-data`. This user must exist, and they will own all
> previously-common tasks/task-groups.
> 3. The pip extra `postgres` is deprecated, in favor of `postgres-psycopg-binary`.
> 4. The configuration variable `DB_ENGINE="postgres"` is deprecated, in favor of `DB_ENGINE="postgres-psycopg"`.
> 5. Python3.9 is deprecated.

* API:
* Users and user groups:
* Replace `UserRead.group_names` and `UserRead.group_ids` with `UserRead.group_ids_names` ordered list (\1844, \1850).
* Deprecate `GET /auth/group-names/` (\1844).
* Add `DELETE /auth/group/{id}/` endpoint (\1885).
* Add `PATCH auth/group/{group_id}/user-settings/` bulk endpoint (\1936).
* Task groups:
* Introduce `/api/v2/task-group/` routes (\1817, \1847, \1852, \1856, \1943).
* Respond with 422 error when any task-creating endpoint would break a non-duplication constraint (\1861).
* Enforce non-duplication constraints on `TaskGroupV2` (\1865).
* Fix non-duplication check in `PATCH /api/v2/task-group/{id}/` (\1911).
* Add cascade operations to `DELETE /api/v2/task-group/{task_group_id}/` and to `DELETE /admin/v2/task-group/{task_group_id}/` (\1867).
* Expand use and validators for `TaskGroupCreateV2` schema (\1861).
* Do not process task `source`s in task/task-group CRUD operations (\1861).
* Do not process task `owner`s in task/task-group CRUD operations (\1861).
* Tasks:
* Drop `TaskCreateV2.source` (\1909).
* Drop `TaskUpdateV2.version` (\1905).
* Revamp access-control for `/api/v2/task/` endpoints, based on task-group attributes (\1817).
* Update `/api/v2/task/` endpoints and schemas with new task attributes (\1856).
* Forbid changing `TaskV2.name` (\1925).
* Task collection:
* Improve preliminary checks in task-collection endpoints (\1861).
* Refactor split between task-collection endpoints and background tasks (\1861).
* Create `TaskGroupV2` object within task-collection endpoints (\1861).
* Fix response of task-collection endpoint (\1902).
* Automatically discover PyPI package version if missing or invalid (\1858, \1861, \1902).
* Use appropriate log-file path in collection-status endpoint (\1902).
* Add task `authors` to manifest schema (\1856).
* Do not use `source` for custom task collection (\1893).
* Rename custom-task-collection request-body field from `source` to `label` (\1896).
* Improve error messages from task collection (\1913).
* Forbid non-unique task names in `ManifestV2` (\1925).
* Workflows and workflow tasks:
* Introduce additional checks in POST-workflowtask endpoint, concerning non-active or non-accessible tasks (\1817).
* Introduce additional intormation in GET-workflow endpoint, concerning non-active or non-accessible tasks (\1817).
* Introduce additional intormation in PATCH-workflow endpoint, concerning non-active or non-accessible tasks (\1868, \1869).
* Stop logging warnings for non-common tasks in workflow export (\1893).
* Drop `WorkflowTaskCreateV2.order` (\1906).
* Update endpoints for workflow import/export (\1925, \1939, \1960).
* Datasets:
* Remove `TaskDumpV2.owner` attribute (\1909).
* Jobs:
* Prevent job submission if includes non-active or non-accessible tasks (\1817).
* Remove rate limit for `POST /project/{project_id}/job/submit/` (\1944).
* Admin:
* Remove `owner` from `GET admin/v2/task/` (\1909).
* Deprecate `kind` query parameter for `/admin/v2/task/` (\1893).
* Add `origin` and `pkg_name` query parameters to `GET /admin/v2/task-group/` (\1979).
* Schemas:
* Forbid extras in `TaskCollectPipV2` (\1891).
* Forbid extras in all Create/Update/Import schemas (\1895).
* Deprecate internal `TaskCollectPip` schema in favor of `TaskGroupV2` (\1861).
* Database:
* Introduce `TaskGroupV2` table (\1817, \1856).
* Add `timestamp_created` column to `LinkUserGroup` table (\1850).
* Add `TaskV2` attributes `authors`, `tags`, `category` and `modality` (\1856).
* Add `update-db-data` script (\1820, \1888).
* Add `taskgroupv2_id` foreign key to `CollectionStateV2` (\1867).
* Make `TaskV2.source` nullable and drop its uniqueness constraint (\1861).
* Add `TaskGroupV2` columns `wheel_path`, `pinned_package_versions` (\1861).
* Clean up `alembic` migration scripts (\1894).
* Verify task-group non-duplication constraint in `2.7.0` data-migration script (\1927).
* Normalize `pkg_name` in `2.7.0` data-migration script (\1930).
* Deprecate `DB_ENGINE="postgres"` configuration variable (\1946).
* Runner:
* Do not create local folders with 755 permissions unless `FRACTAL_BACKEND_RUNNER="slurm"` (\1923).
* Fix bug of SSH/SFTP commands not acquiring lock (\1949).
* Fix bug of unhandled exception in SSH/SLURM executor (\1963).
* Always remove task-subfolder compressed archive (\1949).
* Task collection:
* Create base directory (in SSH mode), if missing (\1949).
* Fix bug of SSH/SFTP commands not acquiring lock (\1949).
* SSH:
* Improve logging for SSH-connection-locking flow (\1949).
* Introduce `FractalSSH.fetch_file` and `FractalSSH.read_remote_json_file` (\1949).
* Use `paramiko.sftp_client.SFTPClient` methods directly rathen than `fabric` wrappers (\1949).
* Disable prefetching for `SFTPClient.get` (\1949).
* Internal:
* Update `_create_first_group` so that it only searches for `UserGroups` with a given name (\1964).
* Dependencies:
* Bump fastapi to `0.115` (\1942).
* Remove pip extra `postgres`, corresponding to `psycopg2+asyncpg` (\1946).
* Deprecate python3.9 (\1946).
* Testing:
* Benchmark `GET /api/v2/task-group/` (\1922).
* Use new `ubuntu22-slurm-multipy` image, with Python3.12 and with Python-version specific venvs (\1946, 1969).
* Get `DB_ENGINE` variable from `os.environ` rather than from installed packages (\1968).

* Database
* Fix use of naming convention for database schema-migration scripts (\1819).
* Testing:
* Test `alembic downgrade base` (\1819).
* Add `GET /api/v2/task/` to benchmarks (\1825).

* API:
* Introduce `GET /auth/current-user/viewer-paths/` endpoint (\1816).
* Add `viewer_paths` attribute to `UserGroup` endpoints (\1816).
* Database:
* Add `viewer_paths` column to `UserGroup` table (\1816).
* Runner:
* Anticipate `wait_thread.shutdown_callback` assignment in `FractalSlurmExecutor`, to avoid an uncaught exception (\1815).