Fractal-server

This release provides a bugfix (PR 1239) and a workaround (PR 1238) for the
SLURM runner, which became relevant for the use case of processing a large
dataset (300 wells with 25 cycles each).

* Runner:
* Do not include `metadata["image"]` in JSON file with task arguments (\1238).
* Add `FRACTAL_RUNNER_TASKS_INCLUDE_IMAGE` configuration variable, to define exceptions where tasks still require `metadata["image"]` (\1238).
* Fix bug in globbing patterns, when copying files from user-side to server-side job folder in SLURM executor (\1239).
* API:
* Fix error message for rate limits in apply-workflow endpoint (\1231).
* Benchmarks:
* Add more scenarios, as per issue \1184 (\1232).

* API:
* Add `GET /admin/job/{job_id}` (\1230).
* Handle `FileNotFound` in `GET /project/{project_id}/job/{job_id}/` (\1230).

* Remove CORS middleware (\1228).
* Testing:
* Fix `migrations.yml` GitHub action (\1225).

* API:
* Add rate limiting to `POST /{project_id}/workflow/{workflow_id}/apply/` (\1199).
* Allow users to read the logs of ongoing jobs with `GET /project/{project_id}/job/{job_id}/`, using `show_tmp_logs` query parameter (\1216).
* Add `log` query parameter in `GET {/api/v1/job/,/api/v1/{project.id}/job/,/admin/job/}`, to trim response body (\1218).
* Add `args_schema` query parameter in `GET /api/v1/task/` to trim response body (\1218).
* Add `history` query parameter in `GET {/api/v1/dataset/,/api/v1/project/{project.id}/dataset/}` to trim response body (\1219).
* Remove `task_list` from `job.workflow_dump` creation in `/api/v1/{project_id}/workflow/{workflow_id}/apply/`(\1219)
* Remove `task_list` from `WorkflowDump` Pydantic schema (\1219)
* Dependencies:
* Update fastapi to `^0.109.0` (\1222).
* Update gunicorn to `^21.2.0` (\1222).
* Update aiosqlite to `^0.19.0` (\1222).
* Update uvicorn to `^0.27.0` (\1222).

> **WARNING**:
>
> This update requires running a fix-db script, via `fractalctl update-db-data`.

* API:
* Improve validation of `UserCreate.slurm_accounts` (\1162).
* Add `timestamp_created` to `WorkflowRead`, `WorkflowDump`, `DatasetRead` and `DatasetDump` (\1152).
* Make all dumps in `ApplyWorkflowRead` non optional (\1175).
* Ensure that timestamps in `Read` schemas are timezone-aware, regardless of `DB_ENGINE` (\1186).
* Add timezone-aware timestamp query parameters to all `/admin` endpoints (\1186).
* API (internal):
* Change the class method `Workflow.insert_task` into the auxiliary function `_workflow_insert_task` (\1149).
* Database:
* Make `WorkflowTask.workflow_id` and `WorfklowTask.task_id` not nullable (\1137).
* Add `Workflow.timestamp_created` and `Dataset.timestamp_created` columns (\1152).
* Start a new `current.py` fix-db script (\1152, \1195).
* Add to `migrations.yml` a new script (`validate_db_data_with_read_schemas.py`) that validates test-DB data with Read schemas (\1187).
* Expose `fix-db` scripts via command-line option `fractalctl update-db-data` (\1197).
* App (internal):
* Check in `Settings` that `psycopg2`, `asyngpg` and `cfut`, if required, are installed (\1167).
* Split `DB.set_db` into sync/async methods (\1165).
* Rename `DB.get_db` into `DB.get_async_db` (\1183).
* Normalize names of task packages (\1188).
* Testing:
* Update `clean_db_fractal_1.4.1.sql` to `clean_db_fractal_1.4.2.sql`, and change `migrations.yml` target version (\1152).
* Reorganise the test directory into subdirectories, named according to the order in which we want the CI to execute them (\1166).
* Split the CI into two independent jobs, `Core` and `Runner`, to save time through parallelisation (\1204).
* Dependencies:
* Update `python-dotenv` to version 0.21.0 (\1172).
* Runner:
* Remove `JobStatusType.RUNNING`, incorporating it into `JobStatusType.SUBMITTED` (\1179).
* Benchmarks:
* Add `fractal_client.py` and `populate_script_v2.py` for creating different database status scenarios (\1178).
* Add a custom benchmark suite in `api_bench.py`.
* Remove locust.
* Documentation:
* Add the minimum set of environment variables required to set the database and start the server (\1198).

> **WARNINGs**:
>
> 1. This update requires running a fix-db script, available at https://raw.githubusercontent.com/fractal-analytics-platform/fractal-server/1.4.2/scripts/fix_db/current.py.
> 2. Starting from this version, non-verified users have limited access to `/api/v1/` endpoints. Before the upgrade, all existing users must be manually set to verified.

* API:
* Prevent access to `GET/PATCH` task endpoints for non-verified users (\1114).
* Prevent access to task-collection and workflow-apply endpoints for non-verified users (\1099).
* Make first-admin-user verified (\1110).
* Add the automatic setting of `ApplyWorkflow.end_timestamp` when patching `ApplyWorkflow.status` via `PATCH /admin/job/{job_id}` (\1121).
* Change `ProjectDump.timestamp_created` type from `datetime` to `str` (\1120).
* Change `_DatasetHistoryItem.workflowtask` type into `WorkflowTaskDump` (\1139).
* Change status code of stop-job endpoints to 202 (\1151).
* API (internal):
* Implement cascade operations explicitly, in `DELETE` endpoints for datasets, workflows and projects (\1130).
* Update `GET /project/{project_id}/workflow/{workflow_id}/job/` to avoid using `Workflow.job_list` (\1130).
* Remove obsolete sync-database dependency from apply-workflow endpoint (\1144).
* Database:
* Add `ApplyWorkflow.project_dump` column (\1070).
* Provide more meaningful names to fix-db scripts (\1107).
* Add `Project.timestamp_created` column, with timezone-aware default (\1102, \1131).
* Remove `Dataset.list_jobs_input` and `Dataset.list_jobs_output` relationships (\1130).
* Remove `Workflow.job_list` (\1130).
* Runner:
* In SLURM backend, use `slurm_account` (as received from apply-workflow endpoint) with top priority (\1145).
* Forbid setting of SLURM account from `WorkflowTask.meta` or as part of `worker_init` variable (\1145).
* Include more info in error message upon `sbatch` failure (\1142).
* Replace `sbatch` `--chdir` option with `-D`, to support also slurm versions before 17.11 (\1159).
* Testing:
* Extended systematic testing of database models (\1078).
* Review `MockCurrentUser` fixture, to handle different kinds of users (\1099).
* Remove `persist` from `MockCurrentUser` (\1098).
* Update `migrations.yml` GitHub Action to use up-to-date database and also test fix-db script (\1101).
* Add more schema-based validation to fix-db current script (\1107).
* Update `.dict()` to `.model_dump()` for `SQLModel` objects, to fix some `DeprecationWarnings`(\1133).
* Small improvement in schema coverage (\1125).
* Add unit test for `security` module (\1036).
* Dependencies:
* Update `sqlmodel` to version 0.0.14 (\1124).
* Benchmarks:
* Add automatic benchmark system for API's performances (\1123)
* App (internal):
* Move `_create_first_user` from `main` to `security` module, and allow it to create multiple regular users (\1036).