Fractal-server

This version comes with streamlined options for the Gunicorn startup command
options, and with two new experimental features.

> NOTE 1: you can now enable custom Gunicorn worker/logger by adding the following
> options to the `gunicorn` startup command:
> - `--worker-class fractal_server.gunicorn_fractal.FractalWorker`
> - `--logger-class fractal_server.gunicorn_fractal.FractalGunicornLogger`

> NOTE 2: A new experimental local runner is available, which uses processes
> instead of threads and support shutdown. You can try it out with the
> configuration variable `FRACTAL_BACKEND_RUNNER=local_experimental`

> NOTE 3: A new PostgreSQL database adapter is available, fully based on
> `psycopg3` (rather than `pyscopg2`+`asyncpg`). You can try it out with the
> configuration variable `DB_ENGINE=postgres-psycopg` (note that this requires
> the `pip install` extra `postgres-psycopg-binary`).


* API:
* Add extensive logs to `DELETE /api/v2/project/{project_id}` (\1532).
* Remove catch of `IntegrityError` in `POST /api/v1/project` (\1530).
* App and deployment:
* Move `FractalGunicornLogger` and `FractalWorker` into `fractal_server/gunicorn_fractal.py` (\1535).
* Add custom gunicorn/uvicorn worker to handle SIGABRT signal (\1526).
* Store list of submitted jobs in app state (\1538).
* Add logic for graceful shutdown for job slurm executors. (\1547)
* Runner:
* Change structure of job folders, introducing per-task subfolders (\1523).
* Rename internal `workflow_dir` and `workflow_dir_user` variables to local/remote (\1534).
* Improve handling of errors in `submit_workflow` background task (\1556, \1566).
* Add new `local_experimental` runner, based on `ProcessPoolExecutor` (\1544, \1566).
* Database:
* Add new Postgres adapter `psycopg` (\1562).
* Dependencies
* Add `fabric` to `dev` dependencies (\1518).
* Add new `postgres-psycopg-binary` extra (\1562).
* Testing:
* Extract `pytest-docker` fixtures into a dedicated module (\1516).
* Rename SLURM containers in CI (\1516).
* Install and run SSH daemon in CI containers (\1518).
* Add unit test of SSH connection via fabric/paramiko (\1518).
* Remove obsolete folders from `tests/data` (\1517).

This release fixes a severe bug where SLURM-executor auxiliary threads are
not joined when a Fractal job ends.

* App:
* Add missing join for `wait_thread` upon `FractalSlurmExecutor` exit (\1511).
* Replace `startup`/`shutdown` events with `lifespan` event (\1501).
* API:
* Remove `Path.resolve` from the submit-job endpoints and add validator for `Settings.FRACTAL_RUNNER_WORKING_BASE_DIR` (\1497).
* Testing:
* Improve dockerfiles for SLURM (\1495, \1496).
* Set short timeout for `docker compose down` (\1500).

> NOTE: This version changes log formats.
> For `uvicorn` logs, this change requires no action.
> For `gunicorn`, logs formats are only changed by adding the following
> command-line option:
> `gunicorn ... --logger-class fractal_server.logger.gunicorn_logger.FractalGunicornLogger`.

* API:
* Add `FRACTAL_API_V1_MODE` environment variable to include/exclude V1 API (\1480).
* Change format of uvicorn loggers (\1491).
* Introduce `FractalGunicornLogger` class (\1491).
* Runner:
* Fix missing `.log` files in server folder for SLURM jobs (\1479).
* Database:
* Remove `UserOAuth.project_list` and `UserOAuth.project_list_v2` relationships (\1482).
* Dev dependencies:
* Bump `pytest` to `8.1.*` (1486).
* Bump `coverage` to `7.5.*` (1486).
* Bump `pytest-docker` to `3.1.*` (1486).
* Bump `pytest-subprocess` to `^1.5` (1486).
* Benchmarks:
* Move `populate_db` scripts into `benchmark` folder (\1489).

* API:
* Add `GET /admin/v2/task/` (\1465).
* Improve error message in DELETE-task endpoint (\1471).
* Set `JobV2` folder attributes from within the submit-job endpoint (\1464).
* Tests:
* Make SLURM CI work on MacOS (\1476).

* Add `FRACTAL_SLURM_SBATCH_SLEEP` configuration variable (\1467).

> WARNING: This update requires running a fix-db script, via `fractalctl update-db-data`.

* Database:
* Create fix-db script to remove `images` and `history` from dataset dumps in V1/V2 jobs (\1456).
* Tests:
* Split `test_full_workflow_v2.py` into local/slurm files (\1454).