Highlighted Features Added
* `toil debug-job` now has `--retrieveTaskDirectory <dir>` which will set up a job's downloaded files under `<dir>` and try to stop the job after doing the downloads. Jobs can call `self.files_downloaded_hook()` to provide a stopping point for this mode. (4815)
* `toil debug-job` can now reconstruct the inside-the-container environment for CWL and WDL tasks. (4815)
* Added support for caching on Slurm and other HPC schedulers (4775)
* Replace all instances of boto2 with boto3 for all Toil AWS code (4718)
* Add support for Python 3.12 (4718)
* Add support for Python 3.13 (5145)
* Ceph input/output errors from file locking functions are now tolerated. (4874)
* Toil now uses `flock` to enable directory locks to work properly (4924)
* Added support to get Slurm partitions and automatically send jobs to GPUs on Slurm (4833) (supports both CWL and WDL)
* New `--symlinkJobStoreReads=False` option lets you force local-node copies (possibly in the cache) even when reading directly from a FileJobStore is possible, potentially reducing shared filesystem IO. (4673)
* Toil now supports reading and writing MiniWDL's call cache. (4797)
* Toil now supports running CWL and WDL workflows from [Dockstore](https://dockstore.org/), by using either a Dockstore page URL or TRS ID as the URL/filename of the workflow to run. Since these often contain `?` or `#`, remember to quote them on the command line! (5049)
* Add support for parallel file imports (5114)
* New argument `--importWorkersThreshold`. This specifies the threshold where files will begin to be imported on individual jobs. Small files will be batched into the same import job up to this threshold.
* `--importWorkersDisk` defaults to 1 MiB. Should be increased when download streaming is not possible on a worker.
Breaking Changes
CWL
* Prevent simultaneous Singularity container pulls in `toil-cwl-runner` (4990)
* Added support to import files on workers for toil-cwl-runner (5025)
* `--runImportsOnWorkers` to enable importing files on workers
* `--importWorkersDisk` to control how much disk space the import worker will use
* Don't error when passing through input as the output (5138)
* CWL jobs with dynamic requirements now have input type checking properly protected by their conditionals. (4930)
* Fixed a LoadListing bug with CWL workflows (5149)
* Fix CWL Workflow Slurm memory test (5151)
* workDir and jobStore defaults to tmp-outdir-prefix (5154)
* CWL container prepull: no reason to if extensions are enabled, they are now supported by cwl-utils 0.36+ (5188)
* CWL container prepull: skip if --no-containers is specified (5188)
WDL
* Update WDL conformance tests on CI (4875)
* Added support to run task only WDL files (4960)
* Added support for the gpu field in WDL (4949)
* Support passing inputs into `toil-wdl-runner` for task only WDLs (4977)
* `toil-wdl-runner` will now carry through task exit codes (4978)
* `toil-wdl-runner` will respect explicit null values for optional inputs (4981)
* toil-wdl-runner will not immediately error on nonexistent coerced files until outputted (4994)
* File? type for string to file coercion is now supported (will be nullified)
* WDL output files will now live in directories named after their tasks instead of UUID directories (5008)
* Fixed a bug with conditional statements inside a WDL scatter (5055)
* `toil-wdl-runner` now correctly finds and returns outputs from tasks in scatters and conditionals when a WDL workflow lacks an `output` section. (5094)
* `toil-wdl-runner` has a new `--allCallOutputs` option to allow including all calls' outputs in a workflow's output. (5093)
* `toil-wdl-runner` can now detect and try not to delete the outputs of a workflow that is meant to use the [Cromwell Output Organizer (`croo`)](https://github.com/ENCODE-DCC/croo). Note that `croo` still can't actually work on the output of `toil-wdl-runner`. (#5093)
* --allCallOutputs no longer discards WDL workflow outputs section outputs. (5106)
* File virtualization in toil-wdl-runner now only happens at task boundaries (5028)
* File to String coercion should be supported
* Added support to import files on workers for toil-wdl-runner (5103)
* Support WDL 1.1 disk specification as per spec (5001)
* Fixed a bug with WDL file imports (5121)
Kubernetes
Dependencies
* Toil can now use connexion 4 (5196)
* Toil now uses htcondor 23.6 or 24, which are still on PyPI
Misc
* Makefile: use isolated builds, add dist target (sdist+wheel) and deprecate the sdist target. (4820) (4826)
* Toil will now wait `--jobStoreTimeout` seconds (default: 30) to see an update to/removal of a job that was run, and will not let the job succeed unless it is seen to make progress. (3814)
* Toil job descriptions no longer have a `command` field, and we track the link to the job body and the command to invoke the Toil worker separately. (4811)
* Several typos in the docs were fixed (4889)
* Add a test to ensure batchsystem plugins are installable (4879)
* Fix Toil utils to work without the AWS extra (4953)
* Print commit hash with `toil --version` when installed from source. Before: `7.1.0a1`. After: `7.1.0a1-ccf57e6071e32675daabdcbacb91988e871745a9` (4954)
* Fixed a broken URL and an omitted variable in CI tests (4974)
* Generate default config correctly (5014)
* Use the latest setuptools when running cactus. (5017)
* Toil will refuse to proceed if it detects that its coordination directory or a Singularity cache directory it needs to lock is on Ceph, to prevent hanging the Ceph MDS (4972)
* Fix a NotImplementedError in the Grid Engine batchsystem (5061)
* Added basic Grid Engine CI tests
* Update Cactus on CI to 2.9.0 (5062)
* Separate out create/delete iam role functions into lib.
* Remove deprecated pipes module (5122)
* New `--slurmTime`/`TOIL_SLURM_TIME` setting to set the time limit on Slurm jobs in a way Toil itself understands. (5010)
* New `--slurmPE` argument to allow setting a parallel-job Slurm partition without using `TOIL_SLURM_PE` (5010)
* New `--slurmArgs` argument to allow specifying extra Slurm submission arguments without using `TOIL_SLURM_ARGS` (5010)
* For non-GPU jobs on Slurm, Toil will submit the job to a partition with a time limit long enough to accommodate the configured runtime (from `--slurmTime`). (For GPU jobs, the lowest-priority GPU partition is still always used.) (5010)
* Toil now has a `--slurmDefaultAllMem` option to run jobs lacking their own memory requirements with Slurm's `--mem=0`, so they get a whole node's memory. (4971)
* `toil-cwl-runner` now has `--no-cwl-default-ram` (and `--cwl-default-ram`) to control whether the CWL spec's default `ramMin` is applied, or Toil's own default memory logic is used. (4971)
* The `--dont_allocate_mem` and `--allocate_mem` options have been deprecated and replaced with `--slurmAllocateMem`, which can be `True` or `False`. (4971)
* Added WDL unit tests to CI (5110)
* Mesos build updated. (5049)
* CWL and WDL argument parsing revised for Python 3.12. (5049)
* Organize stats and logging files into `stats/inbox` and `stats/archive` and avoid a circular rename. (1727)
* Added proper FTP support for jobstores (5134)
* URL existence and size gets/checks are now done with HEAD requests (5134)
* Dependabot configuration should now pass schema validation and is itself under CI (5175)
* Toil now tests a version of Cactus that ought to run on Python 3.13. (5184)
* WDL conformance tests on Kubernetes may now run for 30 minutes. (5185)
* When importing files on workers, fall back to importing on the leader when file sizes are not obtainable (5135)
Thank you to our contributors: stxue1, DailyDreaming, adamnovak, mr-c, gmloose, davidjsherman!