Fractal-server

> **WARNING**: The remove-remote-venv-folder in the SSH task collection is broken (see issue 1633). Do not deploy this version in an SSH-based `fractal-server` instance.

* API:
* Fix incorrect zipping of structured job-log folders (\1648).

This release includes a bugfix for task names with special characters.

> **WARNING**: The remove-remote-venv-folder in the SSH task collection is broken (see issue 1633). Do not deploy this version in an SSH-based `fractal-server` instance.

* Runner:
* Improve sanitization of subfolder names (commits from 3d89d6ba104d1c6f11812bc9de5cbdff25f81aa2 to 426fa3522cf2eef90d8bd2da3b2b8a5b646b9bf4).
* API:
* Improve error message when task-collection Python is not defined (\1640).
* Use a single endpoint for standard and SSH task collection (\1640).
* SSH features:
* Remove remote venv folder upon failed task collection in SSH mode (\1634, \1640).
* Refactor `FractalSSH` (\1635).
* Set `fabric.Connection.forward_agent=False` (\1639).
* Testing:
* Improved testing of SSH task-collection API (\1640).
* Improved testing of `FractalSSH` methods (\1635).
* Stop testing SQLite database for V1 in CI (\1630).

This release includes two important updates:
1. An Update update to task-collection configuration variables and logic.
2. The first released version of the **experimental** SSH features.

Re: task-collection configuration, we now support two main use cases:

1. When running a production instance (including on a SLURM cluster), you
should set e.g. `FRACTAL_TASKS_PYTHON_DEFAULT_VERSION=3.10`, and make sure
that `FRACTAL_TASKS_PYTHON_3_10=/some/python` is an absolute path. Optionally,
you can define other variables like `FRACTAL_TASKS_PYTHON_3_9`,
`FRACTAL_TASKS_PYTHON_3_11` or `FRACTAL_TASKS_PYTHON_3_12`.

2. If you leave `FRACTAL_TASKS_PYTHON_DEFAULT_VERSION` unset, then only the
Python interpreter that is currently running `fractal-server` can be used
for task collection.

> WARNING: If you don't set `FRACTAL_TASKS_PYTHON_DEFAULT_VERSION`, then you
> will only have a single Python interpreter available for tasks (namely the
> one running `fractal-server`).

* API:
* Introduce `api/v2/task/collect/custom/` endpoint (\1607, \1613, \1617, \1629).
* Task collection:
* Introduce task-collection Python-related configuration variables (\1587).
* Always set Python version for task collection, and only use `FRACTAL_TASKS_PYTHON_X_Y` variables (\1587).
* Refactor task-collection functions and schemas (\1587, \1617).
* Remove `TaskCollectStatusV2` and `get_collection_data` internal schema/function (\1598).
* Introduce `CollectionStatusV2` enum for task-collection status (\1598).
* Reject task-collection request if it includes a wheel file and a version (\1608).
SSH features:
* Introduce `fractal_server/ssh` subpackage (\1545, \1599, \1611).
* Introduce SSH executor and runner (\1545).
* Introduce SSH task collection (\1545, \1599, \1626).
* Introduce SSH-related configuration variables (\1545).
* Modify app lifespan to handle SSH connection (\1545).
* Split `app/runner/executor/slurm` into `sudo` and `ssh` subfolders (\1545).
* Introduce FractalSSH object which is a wrapper class around fabric.Connection object.
It provides a `lock` to avoid loss of ssh instructions and a custom timeout (\1618)
* Dependencies:
* Update `sqlmodel` to `^0.0.19` (\1584).
* Update `pytest-asyncio` to `^0.23` (\1558).
* Testing:
* Test the way `FractalProcessPoolExecutor` spawns processes and threads (\1579).
* Remove `event_loop` fixture: every test will run on its own event loop (\1558).
* Test task collection with non-canonical package name (\1602).

This release streamlines options for the Gunicorn startup command, and includes
two new experimental features.

> NOTE 1: you can now enable custom Gunicorn worker/logger by adding the following
> options to the `gunicorn` startup command:
> - `--worker-class fractal_server.gunicorn_fractal.FractalWorker`
> - `--logger-class fractal_server.gunicorn_fractal.FractalGunicornLogger`

> NOTE 2: A new experimental local runner is available, which uses processes
> instead of threads and support shutdown. You can try it out with the
> configuration variable `FRACTAL_BACKEND_RUNNER=local_experimental`

> NOTE 3: A new PostgreSQL database adapter is available, fully based on
> `psycopg3` (rather than `pyscopg2`+`asyncpg`). You can try it out with the
> configuration variable `DB_ENGINE=postgres-psycopg` (note that this requires
> the `pip install` extra `postgres-psycopg-binary`).


* API:
* Add extensive logs to `DELETE /api/v2/project/{project_id}` (\1532).
* Remove catch of `IntegrityError` in `POST /api/v1/project` (\1530).
* App and deployment:
* Move `FractalGunicornLogger` and `FractalWorker` into `fractal_server/gunicorn_fractal.py` (\1535).
* Add custom gunicorn/uvicorn worker to handle SIGABRT signal (\1526).
* Store list of submitted jobs in app state (\1538).
* Add logic for graceful shutdown for job slurm executors (\1547).
* Runner:
* Change structure of job folders, introducing per-task subfolders (\1523).
* Rename internal `workflow_dir` and `workflow_dir_user` variables to local/remote (\1534).
* Improve handling of errors in `submit_workflow` background task (\1556, \1566).
* Add new `local_experimental` runner, based on `ProcessPoolExecutor` (\1544, \1566).
* Database:
* Add new Postgres adapter `psycopg` (\1562).
* Dependencies
* Add `fabric` to `dev` dependencies (\1518).
* Add new `postgres-psycopg-binary` extra (\1562).
* Testing:
* Extract `pytest-docker` fixtures into a dedicated module (\1516).
* Rename SLURM containers in CI (\1516).
* Install and run SSH daemon in CI containers (\1518).
* Add unit test of SSH connection via fabric/paramiko (\1518).
* Remove obsolete folders from `tests/data` (\1517).

This release fixes a severe bug where SLURM-executor auxiliary threads are
not joined when a Fractal job ends.

* App:
* Add missing join for `wait_thread` upon `FractalSlurmExecutor` exit (\1511).
* Replace `startup`/`shutdown` events with `lifespan` event (\1501).
* API:
* Remove `Path.resolve` from the submit-job endpoints and add validator for `Settings.FRACTAL_RUNNER_WORKING_BASE_DIR` (\1497).
* Testing:
* Improve dockerfiles for SLURM (\1495, \1496).
* Set short timeout for `docker compose down` (\1500).

> NOTE: This version changes log formats.
> For `uvicorn` logs, this change requires no action.
> For `gunicorn`, logs formats are only changed by adding the following
> command-line option:
> `gunicorn ... --logger-class fractal_server.logger.gunicorn_logger.FractalGunicornLogger`.

* API:
* Add `FRACTAL_API_V1_MODE` environment variable to include/exclude V1 API (\1480).
* Change format of uvicorn loggers (\1491).
* Introduce `FractalGunicornLogger` class (\1491).
* Runner:
* Fix missing `.log` files in server folder for SLURM jobs (\1479).
* Database:
* Remove `UserOAuth.project_list` and `UserOAuth.project_list_v2` relationships (\1482).
* Dev dependencies:
* Bump `pytest` to `8.1.*` (1486).
* Bump `coverage` to `7.5.*` (1486).
* Bump `pytest-docker` to `3.1.*` (1486).
* Bump `pytest-subprocess` to `^1.5` (1486).
* Benchmarks:
* Move `populate_db` scripts into `benchmark` folder (\1489).