Toil

Changelog

CWL
* Fix cwl and wdl dependency bleed and add stand alone tests. 3582
* Use MpiConfig.load() to handle MPI config file 3574
* Add support for MPI with CWL. 3525

Misc
* Numerous Type Hints. 3571 3634 3626 3625 3616 3614 3601 3592 3590 3581
* Configurable Grafana port 3597
* Handle streaming reads from the cache when the data isn't written back yet 3595
* Balance over pools using AWS ASGs 3490
* Change how Kubernetes schedules and scales for hopefully better scale-down behavior 3587
* Add Owner tags to our AWS buckets when testing. 3577
* Allow workdir override 3586
* Additional info for "Permission denied" error 3579
* Consolidate memory functions 3529
* Pen children 3482
* Sniff raid better 3526
* Add a propagation policy to batch delete 3522
* Cleanup script for buckets, sdb domains, instance profiles, and roles. 3373
* Add decorator for flaky tests. 3510

_Bug Fixes_
* Enforce valid uuid from getNodeId() 3611
* Removed extra memory multiplication 3608
* Fix non integers lsf memory requested 3609
* Fixes jobCommand unpack 3610
* Fix CWL test 20 on Kubernetes 3572
* Fix 3579 breaking AWS docs 3596
* Update cactus test to fix broken bucket links. 3594
* Allow cleaning up a job whose overlargeID fell off 3584
* Only deploy a user script if we can deploy user scripts 3518
* Reduce path name lengths. 3438
* Insist on credentials for testing AMI finding 3524

Thank you to our contributors: mr-c, adamnovak, w-gao, jonathanxu18, thiagogenez, julian-klode, darafferty, nikhil, mhpopescu!

Changelog

CWL
* Run CWL conformance tests on Kubernetes. 3323
* CWL symlinking files into work directory 3445
* set resource reqs for all CWL_INTERNAL_JOBS 3442

WDL
* Add WDL flatten() function + tests. 3485
* Add WDL collect_by_key() function + tests. 3476
* Add WDL keys() function + tests. 3460
* Add WDL as_pairs() function + tests. 3364
* Add as_map() WDL function 3448
* Add basic WDL development support 3434
* Add basic WDL 1.0 support 3421
* Update dictionary structure in AnalyzeWDL. 3416

Misc
* Add extra context for disk usage warning. 3495
* Update developing.rst 3479
* Allow text/encode options for read and write functions in file store 3428
* Type hinting and mypy checking additions 3470 3458 3456
* Quality of life improvements for cactus. 3463
* Add checklists for reviewing and merging PRs 3432
* Ensure that python3 and not python (2) is used 3446
* Support Kubernetes clusters in toil launch-cluster 3357
* Additional check for predecessors. 3417
* Allow setOptions to accept an argparse group. 3426

_Bug Fixes_
* AWS: Handle socket timeout during AWS discovery 3503
* Use back-up flatcar AMI if not found. 3513
* Lsf command parser 3475
* Retry on s3 throttling. 3504
* Stop deleting all ASGs with tags 3474
* Only determine execute permissions on files we copy/move. 3437
* cope with an invalid HTTP_PROXY 3447
* Fix failing Google jobstore test 3420
* Update Google job store 3412

Thank you to our contributors: mr-c, adamnovak, w-gao, jonathanxu18, Arthur Rand, mberacochea, Jessime, thiagogenez, julian-klode!

Changelog

CWL
* Update to the latest cwltool (3.0.20201121085451 -> 3.0.20201203173111). 3375
* Add better handling for potentially mis-ordered CWL args. 3395
* Confirm CommandInputParameter expression can receive a File object. 3350

WDL
* Make AnalyzeWDL abstract to allow implementation for different WDL versions 3391

Misc
* Clean up memoize.py. 3374
* Refactor bioio library, simplify logging, and small adjustments to utils. 3351
* Use regular division for memory calculation on LSF. 3387
* Remove unused --nodeOptions arg. 3397
* Get boto S3 args for minio from environment variables. (3370)
* Update EC2 lists. 3376
* Add a retry to AWS destroy. 3379
* Regular division for memory calculation in LSF. 3387
* Add option to blank mem allocation for SLURM. 3399
* Preserve file permissions on imported/exported files 3322

_Bug Fixes_
* Preserve file permissions on imported/exported files. 3322
* Make building a flattened list of live jobs non-recursive. 3394
* Move AWS job store uploads/downloads to boto3 (reopened) 3400
* Use a Docker Hub mirror when pulling Docker images. 3411
* Corrected gridengine maxMEM check to use memorystring 3410

Thank you to our contributors: mr-c, adamnovak, w-gao, jonathanxu18, douglowe, stevekm , thiagogenez!

Changelog

WDL
* Add WDL cross function. 3360

_Bug Fixes_
* Better error message for insufficient disk. 3362
* Fix maxMEM comparison in LSF. 3369

Thank you to our contributors: mr-c , adamnovak , w-gao , jonathanxu18 !

Caching is now turned on by default.

CWL v1.1 is also now fully supported and passing all conformance tests.

The following batch system names are now deprecated (3225) and replaced by:
- singleMachine -> single_machine
- gridEngine -> grid_engine
- LSF -> lsf
- Mesos -> mesos
- Slurm -> slurm
- Torque -> torque
- HTCondor -> htcondor
- Kubernetes -> kubernetes
- k8s -> kubernetes

Changelog

CWL
* Upgrade to a CWL v1.2.0 capable cwltool (3.0.20200709181526 -> 3.0.20200807132242). 3137
* Export to a bucket if cwl's CreateFile is specified. 3124
* CWL v1.1 Support: loadContents if True for StepValueFrom objects. 3266
* CWL: Fix workdir permissions. 3229
* Test CWL secondary file from s3. 3311
* CWL v1.1 Support: Inplace update has side effect on directory content. 3280
* More CWL v1.2 conformance tests. 3336
* CWL: Reformat w/ Black and better support for tmpdir_prefix with suffix. 3333

WDL
* Add ceil() and floor() WDL functions. 3168
* Create a stdout and stderr file for each WDL task. 3181
* Add WDL write functions and builtin tests. 3236
- write_lines
- write_tsv
- write_json
- write_map
* Add WDL read functions and builtin tests. 3244
- read_lines
- read_tsv
- read_json
- read_map
- read_int
- read_string
- read_float
- read_boolean
* Add transpose() WDL function. 3273
* Add length() WDL function. 3307
* Add sub() WDL function. 3252
* Add range() WDL function. 3277
* Add size() WDL function. 3255
* Add zip() WDL function. 3355
* Implement WDL Pair type. 3304
* Refactor WDL types 3335

Kubernetes
* Adding Labels to Pods/Jobs. 3233
* Make kubernetes & botocore packages fully optional. 3259 3261
* KubeWatch optimization. 3227
* Catch other kubernetes imports. 3261
* Automatically infer singularity for kubernetes. 3279

Misc

* Allow Caching By Default. 3111
* Update dependencies and loosen restrictions slightly. 3139
* Ensure that falsey (but non-null) values are considered. 3143
* Remove pathlib. 3169
* Move AWS job store uploads/downloads to boto3. 3153
* Update prom/node-exporter to use quay.io. 3186
* Add support for TOIL_CUSTOM_INIT_COMMAND. 3183
* Update docker module version to 4.3.1. 3222
* Add 'stream' and 'demux' options to apiDockerCall. 3224
* Use boto3 to create our leader node. 3145
* Prevent job kind filenames from getting too long. 3230
* Add a retry decorator. 3144
* Remove mesos offer messages. 3308
* Job Concept Unification. 3250
* Add reporting for accessed files of failed jobs. 3309
* Add coverage and refactor. 3314
* Expose --statePollingWait param through toil. 3321
* Record the maximum memory used by LSF jobs. 3327
* Remove custom math functions and replace with built ins. 3338
* Define a changelog process. 3316
* SLURM: getting job details from `scontrol show jobs` was non-functional and has now been fixed. This behavior in the case of `sacct` not being configured now works. 3346
* LSF: handle unicode configuration files 3354
* Replace abssympath w/ builtin. 3356

_Bug Fixes_
* Turn off watch code in Kubernetes. 3175
* Check against joining the current thread during CachingFileStore destructor. 3187
* Slurm.py error, undefined stdout - stdout should be stderr in code. 3194
* Fix restart flag error (issue 3094). 3142
* Batch System generates a key error instead of a usage message. 3225
* Remove Python 2 imports in WDL. 3228
* Move checkForDeadlocks() in the leader. 3234
* Fix boto3 migration bugs. 3243
* Fix None check when parsing WDL JSON file. 3272
* Fix broken stdout/stderr in torque job. 3306
* Add doubleMem for LSF jobs that die due to imposed memory limit. 3313
* Add 500 status to the list of retriable SDB BotoServerErrors. 3329
* Add a retry logic for BucketNotEmpty error. 3341
* Fix None check when parsing WDL JSON file. 3272
* Warn when trying to clean non-existent job store 3288
* Only rm a statefile that exists. 3348
* CWL: Remove unprocessed secondaryFiles from both Dict and List types.

Thank you to our contributors: mr-c, arostamianfar, adamnovak, diekhans, w-gao, jonathanxu18, jeffrey856, davidlougheed, altairwei, mberacochea, drkennetz, ionox0!

* Don't log in the inner scheduling loop. 3065
* Handle anticipated 404 errors without retrying. 3067
* Test and reimplement AMI finding via Flatcar JSON feed. 3061
* Bump Enlighten version to partly fix 3069. 3070
* Get wait duration is far too short of a rest for lsf. 3076
* Unify logging format. 3073
* Support CWL 1.1 Listing. 3058
* Enable CWL v1.2.0-dev3 (3.0.20200324120055 -> 3.0.20200530110633). 3092
* Enable CWL v1.2.0-dev4 (3.0.20200530110633 -> 3.0.20200709181526). 3105
* Add --nodeStorageOverrides option. 3096
* More robust CWL filepaths and secondary files. 3114
* Replace custom mkdir_p with the built in. 3123
* Move conditional execution test to before fill_in_defaults. 3117

_Bug Fixes_
* Stop CWL from trying to copy downloaded directories from somewhere local. 3053
* Exit the worker with 1 when jobs fail so the batch system can see. 3052
* Eliminate recursion in root finding to fix 3080. 3081
* Add a Cactus Kubernetes test to the integration tests on Gitlab. 3077
* Chaining doesn't make sense for CWL. 3091
* Fix failed job accounting. 3103
* Improve lsf batchsystem stability. 3101
* Deduplicate updated jobs. 3108
* Redundant virtualenv check. 3135
* Change Slurm command output processing. 3133
* Fix for log file being linked more than once. 3130
* Wrap Kubernetes watches to fix 3125. 3126
* Create dirs that do not exist when bind-mounting to docker. 3120
* Fix database state when file upload fails. 3122

Thank you to our contributors: mr-c, arostamianfar, adamnovak, diekhans, nikhil, drkennetz, jonathanxu18, tobiaszjarosiewicz, ionox0!