Toil

Caching is now turned on by default.

CWL v1.1 is also now fully supported and passing all conformance tests.

The following batch system names are now deprecated (3225) and replaced by:
- singleMachine -> single_machine
- gridEngine -> grid_engine
- LSF -> lsf
- Mesos -> mesos
- Slurm -> slurm
- Torque -> torque
- HTCondor -> htcondor
- Kubernetes -> kubernetes
- k8s -> kubernetes

Changelog

CWL
* Upgrade to a CWL v1.2.0 capable cwltool (3.0.20200709181526 -> 3.0.20200807132242). 3137
* Export to a bucket if cwl's CreateFile is specified. 3124
* CWL v1.1 Support: loadContents if True for StepValueFrom objects. 3266
* CWL: Fix workdir permissions. 3229
* Test CWL secondary file from s3. 3311
* CWL v1.1 Support: Inplace update has side effect on directory content. 3280
* More CWL v1.2 conformance tests. 3336
* CWL: Reformat w/ Black and better support for tmpdir_prefix with suffix. 3333

WDL
* Add ceil() and floor() WDL functions. 3168
* Create a stdout and stderr file for each WDL task. 3181
* Add WDL write functions and builtin tests. 3236
- write_lines
- write_tsv
- write_json
- write_map
* Add WDL read functions and builtin tests. 3244
- read_lines
- read_tsv
- read_json
- read_map
- read_int
- read_string
- read_float
- read_boolean
* Add transpose() WDL function. 3273
* Add length() WDL function. 3307
* Add sub() WDL function. 3252
* Add range() WDL function. 3277
* Add size() WDL function. 3255
* Add zip() WDL function. 3355
* Implement WDL Pair type. 3304
* Refactor WDL types 3335

Kubernetes
* Adding Labels to Pods/Jobs. 3233
* Make kubernetes & botocore packages fully optional. 3259 3261
* KubeWatch optimization. 3227
* Catch other kubernetes imports. 3261
* Automatically infer singularity for kubernetes. 3279

Misc

* Allow Caching By Default. 3111
* Update dependencies and loosen restrictions slightly. 3139
* Ensure that falsey (but non-null) values are considered. 3143
* Remove pathlib. 3169
* Move AWS job store uploads/downloads to boto3. 3153
* Update prom/node-exporter to use quay.io. 3186
* Add support for TOIL_CUSTOM_INIT_COMMAND. 3183
* Update docker module version to 4.3.1. 3222
* Add 'stream' and 'demux' options to apiDockerCall. 3224
* Use boto3 to create our leader node. 3145
* Prevent job kind filenames from getting too long. 3230
* Add a retry decorator. 3144
* Remove mesos offer messages. 3308
* Job Concept Unification. 3250
* Add reporting for accessed files of failed jobs. 3309
* Add coverage and refactor. 3314
* Expose --statePollingWait param through toil. 3321
* Record the maximum memory used by LSF jobs. 3327
* Remove custom math functions and replace with built ins. 3338
* Define a changelog process. 3316
* SLURM: getting job details from `scontrol show jobs` was non-functional and has now been fixed. This behavior in the case of `sacct` not being configured now works. 3346
* LSF: handle unicode configuration files 3354
* Replace abssympath w/ builtin. 3356

_Bug Fixes_
* Turn off watch code in Kubernetes. 3175
* Check against joining the current thread during CachingFileStore destructor. 3187
* Slurm.py error, undefined stdout - stdout should be stderr in code. 3194
* Fix restart flag error (issue 3094). 3142
* Batch System generates a key error instead of a usage message. 3225
* Remove Python 2 imports in WDL. 3228
* Move checkForDeadlocks() in the leader. 3234
* Fix boto3 migration bugs. 3243
* Fix None check when parsing WDL JSON file. 3272
* Fix broken stdout/stderr in torque job. 3306
* Add doubleMem for LSF jobs that die due to imposed memory limit. 3313
* Add 500 status to the list of retriable SDB BotoServerErrors. 3329
* Add a retry logic for BucketNotEmpty error. 3341
* Fix None check when parsing WDL JSON file. 3272
* Warn when trying to clean non-existent job store 3288
* Only rm a statefile that exists. 3348
* CWL: Remove unprocessed secondaryFiles from both Dict and List types.

Thank you to our contributors: mr-c, arostamianfar, adamnovak, diekhans, w-gao, jonathanxu18, jeffrey856, davidlougheed, altairwei, mberacochea, drkennetz, ionox0!

* Don't log in the inner scheduling loop. 3065
* Handle anticipated 404 errors without retrying. 3067
* Test and reimplement AMI finding via Flatcar JSON feed. 3061
* Bump Enlighten version to partly fix 3069. 3070
* Get wait duration is far too short of a rest for lsf. 3076
* Unify logging format. 3073
* Support CWL 1.1 Listing. 3058
* Enable CWL v1.2.0-dev3 (3.0.20200324120055 -> 3.0.20200530110633). 3092
* Enable CWL v1.2.0-dev4 (3.0.20200530110633 -> 3.0.20200709181526). 3105
* Add --nodeStorageOverrides option. 3096
* More robust CWL filepaths and secondary files. 3114
* Replace custom mkdir_p with the built in. 3123
* Move conditional execution test to before fill_in_defaults. 3117

_Bug Fixes_
* Stop CWL from trying to copy downloaded directories from somewhere local. 3053
* Exit the worker with 1 when jobs fail so the batch system can see. 3052
* Eliminate recursion in root finding to fix 3080. 3081
* Add a Cactus Kubernetes test to the integration tests on Gitlab. 3077
* Chaining doesn't make sense for CWL. 3091
* Fix failed job accounting. 3103
* Improve lsf batchsystem stability. 3101
* Deduplicate updated jobs. 3108
* Redundant virtualenv check. 3135
* Change Slurm command output processing. 3133
* Fix for log file being linked more than once. 3130
* Wrap Kubernetes watches to fix 3125. 3126
* Create dirs that do not exist when bind-mounting to docker. 3120
* Fix database state when file upload fails. 3122

Thank you to our contributors: mr-c, arostamianfar, adamnovak, diekhans, nikhil, drkennetz, jonathanxu18, tobiaszjarosiewicz, ionox0!

* Attempt to make batch system more robust and debuggable. 2959
* Kubernetes Shared Caching. 3012
* Test python3.8 and build a python3.8 appliance. 3028
* Use the original pymesos. 3036
* Detect more kinds of virtual environment and normalize prefixes. 3035
* Do More Hashing. Can't have enough. 3025
* Replace reissued jobs message with better progress indicators. 3044
* Detect insufficient resources and deadlocks more usefully. 3043

_Bug Fixes_
* Don't list specific CWL features, try them all (and fix some CWL running bugs). 3015
* Always honor TOIL_GRIDENGINE_PE and never assume a site-dependent default. 3041

Thank you to our contributors: mr-c, arostamianfar, adamnovak, diekhans!

This release moves to python3 only, dropping all python2.7 compatibility, and also deprecates the command "cwltoil" in favor of "toil-cwl-runner".

* Update cwltool version (==1.0.20190906054215 -> <=2.0.20200126090152). 2969
* Drop the long-deprecated cwltoil. 2843
* Dropped python2.7. 2973
* Port provisioner tests and only test py3.6. 2842
* CWL: Refactor of link merge + conditionals + pickValue. 2845
* Add import subprocess to executor.py. 2976
* Encode mtail stdin to utf-8 and flush. 2978
* Encode framework message as utf-8 as well. 2981
* Make the output files for grid engine batch systems not try and be in per-host directories. 2956
* Use file locks instead of PID polling to see if other processes sharing the cache are alive. 2982
* Size downloads during download. 2989
* Add timeouts to all SQLite connect statements. 2993
* Eliminate sleep time on --restart. 2990
* Add support for --awsEc2ExtraSecurityGroupId. 2997
* Add moveExports option. 2983
* Set worker threads as dameon to prevent hanging process on error. 2998
* Cut the Threads. 2999
* [Part 1] CWL v1.1 Support 2985
* [Part 2] CWL v1.1 Support 3000
* Update Sphinx and let its version float. 3011
* Add --enableUnlimitedPreemptableRetries option. 2896
* Add less and vim to docker. 3017
* Improve and Unify Log Dumping. 3008
* Make dependency on Python3.6+ machine-readable in setup.py. 3021
* Improve slurm update frequency. 3026
* Make Sphinx a make prepare component and not a dependency. 3023
* Use forked pymesos and http-parser dependencies. 3024

_Bug Fixes_
* Fix python3 string/bytes error when using --printLogs. 3005
* Fix connection between Kubernetes and Leader. 3004

Thank you to our contributors: mr-c, arostamianfar, adamnovak, jeffrey856, kaushik-work, glmxndr, johnbradley, cmarkello, diekhans!

This is the last version of Toil that supports python2.7 and allows the command word "cwltoil" (now "toil-cwl-runner").

* Update ssh and docker setup files. 2958
* Update EC2 instance types. 2962
* Source toil environment as a configuration file. 2960
* Scan for the latest caching database instead of using a hard link. 2948
* Specify py3.6 or py3.x instead of just py3 in appliance. 2966

_Bug Fixes_
* Fix database being locked when trying to write in the CachingFileStore. 2955
* Fix py3 spot market scaling. 2971

Thank you to our contributors: mr-c, arostamianfar, adamnovak, jeffrey856!

Functionally the same as Toil Version 3.22.0.