Toil

Highlighted Features Added

* WDL and CWL task standard output and standard error logs that are not captured by the workflow will now be logged at INFO level and stored in the `--writeLogs`/`--writeLogsGzip` directory. (4657)
* Use a default log limit of 100MiB (4788)

Breaking Changes

* Stats and logging system again uses job display name (4755)
* `--disableProgress` is once again a flag that doesn't take an argument (4758)


CWL

* Don't clear out user-provided values for the --default-container option (4730)

WDL

* WDL job names now include numbers for scatters (4755)
* Multi-line WDL placeholder substitutions no longer interfere with de-indenting WDL command blocks (https://github.com/chanzuckerberg/miniwdl/issues/665)
* Standard error for failed tasks is now always logged to the worker log somewhere (4781)

Kubernetes

Dependencies

* Deps: removed the ruaml.yaml.string plugin dependency for a simpler solution (4760)

Misc

* Toil will no longer warn about a missing XDG_RUNTIME_DIR (4769)
* Read the Docs and CI docs builds should have Graphviz installed (pending CI image rebuild) (4734)
* Add more Python3.12 compatibility by replacing the one function from distutils that we use, `strtobool()`. (4765)
* Set default cache folders to be accessible between toil-wdl-runner workflows (Same as MiniWDL/Singularity defaults) (4761)
* Set toil-wdl-runner cache folders on Toil managed clusters to be at `/var/lib/toil` (4761)
* Fall back to assuming machine has 1 core when CPU count is unavailable. (4545)
* `FileJobStore` now supports filenames that get modified when percent-encoded (4779)

Thank you to our contributors:

DailyDreaming mr-c stxue1 adamnovak app/dependabot

**Full Changelog**: https://github.com/DataBiosphere/toil/compare/releases/6.0.0...releases/6.1.0

NOTE!
We now have a config file! https://toil.readthedocs.io/en/latest/running/cliOptions.html#the-config-file


Breaking Changes
* Removed the parasol batch system
* Removed the TES batch system (this is now a plugin)
* Removed our WDL compiler in favor of an interpreter (we still support WDL, we just do it differently now)
* We no longer support python3.7


CWL
* Support CWL 1.2.1 (4682)
* CWL Pipefish compatibility (4636)
* Support per-task preemptibility in CWL (4551)
* Fix configargparse in CWL (4618)
* cwl: use the latest commit from the proposed CWL v1.2.1 branch (4565)
* Upgrade cwltool to avoid broken galaxy-tool-util release. (4639)
* Implement a better config file system for CWL/WDL options (4666)
* Allow working with remote files in CWL and WDL workflows (4690)
* Make cwl mutually exclusive groups exist only when cwl is not suppressed (4725)
* Log more usefully for CWL workflows (4736)


WDL
* Simplify WDL Toil job graphs (4524)
* More WDL and Slurm documentation (4558)
* Improve WDL documentation (4732)
* Add String to File functionality into toil-wdl-runner (4589)
* Run WDL output through Toil export system to support URIs (4579)
* Allow the WDL output section to reference itself (4592)
* Ensure sibling files in toil-wdl-runner (4610)
* Make WDLOutputJob collect all task outputs (4602)
* Report errors in WDL using MiniWDL's error location printer (4637)
* Remove the WDL compiler. (4679)
* Implement a better config file system for CWL/WDL options (4666)
* Allow working with remote files in CWL and WDL workflows (4690)
* Strip leading whitespace from WDL commands (4720)


Misc
* Add config file support (4569)
* Support Python3.11 and drop Python 3.7 (4646)
* Move TES batch system to a plugin (4650)
* Turn batch system tests back on (4649)
* Separate out integration tests to run on a schedule (4612)
* Avoid concurrent modification in cluster scaler tests (4600)
* Remove old buckets from AWS (4588)
* Tests: only request a single core (4572)
* Reduce the number of assert statements (4590)
* take any nvidia-smi exception as not having gpu (4611)
* More resiliancy (4395)
* Remove useage of the deprecated pkg_resources (4701)
* Make sure cwltool always knows we have an outdir to fix 4698 (4699)
* AWS jobStoreTest: re-use delete_s3_bucket from toil.lib.aws (4700)
* Only count output file usage when using the file store (4692)
* Remove the parasol batch system. (4678)
* Move around reqs and move aws dev libraries to aws (4664)
* Make sure the `--batchLogsDir` exists if it is set (4635)
* Update EC2 instances and EC2 update script. (4745)
* remove extraneous dependency on old 'mock' (4739)
* Point CI at the new public URLs for stuff we host
* Add __init__.py to options folder (4723)


_Bug Fixes_
* Lower redirect log level to fix 4526 (4578)
* Fix mypy from being broken by new boto types (4577)
* Fix CI on local Gitlab runners (4571)
* Banish ghost jobs (4563)
* Stop deleting chained-to jobs which fail as orphaned jobs (4557)
* Fix pickling error when jobstate file doesnt exist and fix threading error when lock file exists then disappears (4575)
* Fix 3867 and try to explain but not crash when bad things happen to our mutex file (4656)
* Fix CI Appliance Builds (4655)
* Tolerate a failed AMI polling attempt (4727)* Add pure Python fallback for getDirSizeRecursively() (4753)
* Don't mark inputs (or outputs) executable for no reason (4728)
* Fix scheduled CI tests (4742)
* Fix --printJobInfo (4709)


Thank you to our contributors: stxue1 , w-gao, DailyDreaming , mr-c , adamnovak , glennhickey, misterbrandonwalker, and a-detiste !

WDL
* Virtualize filenames as in-container paths from point of view of WDL command (4527)
* Add WDL conformance tests to CI (4530)
* Use less memory in the Giraffe WDL test (4541)

Version Upgrades
* Upgrade to cwltool 3.1.20230601100705 (4500)
* Update mock requirement from <5,>=4.0.3 to >=4.0.3,<6 (4366)

Misc
* Anonymous access to Google Storage (4518)
* Reorder config so that default settings are applied first (4528)
* Add a way to forward accelerators to Docker containers (4492)

_Bug Fixes_
* Fix test failures without docker installed (4544)
* Prevent certain tests from being run twice in CI (4529)
* Drop external Docker builder (4523)
* Fix CI lint test (4533)
* Grab AWS group policies on top of user (4505)
* Grab accelerator set off the end of the list instead of by index (4506)
* Fix RtD build (4491)
* Include tests (4499)

Thank you to our contributors: stxue1 , DailyDreaming , mr-c , adamnovak , and tjni !

Breaking Changes
* Imported files will be symlinked by default, unless the user sets `--noLinkImports` or the workflow imports with `symlink=False`. (3949)

WDL
* Toil will now stop if it encounters an error polling a possible import URL for a WDL workflow input file. (4479)
* WDL workflows will be protected against imported files with no basenames. (4477)

Misc
* Toil batch system ID numbers for issued jobs now start at 1. (4482)
* Attempts to import files from URLs when the implementing job store is missing an extra are now better reported. (4479)
* Include tests in the source distribution that gets published to PyPI (4499)

_Bug Fixes_
* Toil should no longer crash when a delete wins a race against a load in `FileJobStore` (4484)
* Prevent local root jobs (such as WDLRootJob) from being run twice. (4482)
* Slurm and other grid batch system jobs will now have more informative names (4472)
* WDL workflows can no longer import `""` as a File. (4477)

Thank you to our contributors: stxue1, DailyDreaming, mr-c, adamnovak

Changelog

Highlighted Features Added
* Add a `--caching` option which explicitly states whether to use caching with a workflow. Uses a default value depending on whether or not we are using the file job store if not specified. (4218)
* New prototype WDL runner `python -m toil.wdl.wdltoil` using MiniWDL (3468)
* MiniWDL-based WDL implementation can now run the vg Giraffe WDL workflow ( 4353)
* Toil now tests against our own tiny set of WDL conformance tests (4351)
* Toil can run the HPRC assembly WDL workflows (4435)
* Toil can now use Mesos roles (4455)

Breaking Changes

* Replace "preemptable" with "preemptible", add example of using --defaultPreemptible flag to Preemptibility documentation (1951)

CWL
* CWL: run all ExpressionTools on the Leader node, instead of submitting separate jobs (4157)

Kubernetes
* Kubernetes batch system: Delete jobs individually when batch delete fails (3403)
* Documentation for running a Toil leader for a Kubernetes workflow outside Kubernetes now covers examples and common problems for running CWL workflows (document toil-cwl-runner + "Running the Leader Outside Kubernetes" 3422)
* Kubernetes batch system: support `--maxCores`, `--maxDisk`, and `--maxMemory` (2864)
* Add tutorial for Kubernetes launch cluster (3743)


Dependencies

* Require htcondor 10 exactly (4315)
* Toil jobs now have a `local` parameter which determines if they should run on the leader. (4388)

Misc

* The offline tests can now be run in parallel (3493)
* Code updated to be more idiomatic for Python3.7 (4295)
* Support for a `--network` for `toil launch-cluster` for Google cloud (4196)
* Support for a `--use_private_ip` for `toil launch-cluster` to dial nodes by private IP instead of public IP (4196)
* GPU scheduling should now be supported on Slurm (4308)
* Toil now supports a `--batchLogsDir` option and `TOIL_BATCH_LOGS_DIR` environment variable, to provide a directory other than the work dir where Toil will instruct HPC batch systems to save their captured job logs.
* `htcondor` batch system should now work again, and will retry connections
* Updated the --coalesceStatusCalls help documentation to reflect the current state of https://github.com/DataBiosphere/toil/issues/4431 (#4437)
* Toil no longer trusts XDG_RUNTIME_DIR under Slurm (fixes some of the issues behind 4395 when Slurm is configured not to follow the XDG spec) (4435)
* Toil now puts it lock files for Singularity cache directories for WDL in those directories (4435)
* Toil's WDL interpreter can now use local-to-the-leader jobs for evaluating WDL code that doesn't need appreciable resources (4388)
* Toil now tolerates more possible exceptions related to the panasas network file system (4440)
* Type hinting to functions in resource.py (938)
* Added return type to inVirtualEnv() in `__init__.py` (938)
* Added None checks to some function bodies (938)



_Bug Fixes_
* Stop crashing when predefined batch job exit reasons are used and need to go into the message bus log file (4321)
* Added `import subprocess` to restore the behavior of 588. (4429)
* Toil will no longer use the stored message bus path from an old execution of a workflow when deciding where to save the message bus log when restarting a workflow (4438)
* Fix --custom-net mutual exclusivity bug. (4458)


Thank you to our contributors: stxue1 , DailyDreaming , mr-c , adamnovak , jfennick , misterbrandonwalker , w-gao , stephanaime , glennhickey , Hexotical , manabuishii gmloose , boukn , and thiagogenez !