Toil

This pre-release of Toil contains bug fixes and the addition of experimental, CGCloud-based auto-scaling.

https://github.com/BD2KGenomics/toil/milestone/17?closed=1

Due to the experimental nature of the auto-scaling feature, we decided to skip publishing a stable release for it. To install this release with pip, use `pip install toil==3.4.0a1.dev228`.

The next stable release of Toil will contain docker-based autoscaling for EC2.

https://pypi.python.org/pypi?:action=display&name=toil&version=3.3.4
- Fix: Toil fails completely when not installed with [aws] extra (1115)
- Prevent virtualenv from silently upgrading pip and setuptools, potentially breaking build (1270)

Core production is advised to skip this release.

- Fix `--restart` not working at all on distributed clusters (resolves 1127)
- Work around apparent eventual consistency with SDB (resolves 760)

This release is recommended to replace 3.3.1 and 3.3.0 for production use.

[CGCloud 1.5.5](https://github.com/BD2KGenomics/cgcloud/releases/tag/releases%2F1.5.5) was released to deploy this release of Toil on toil-box images/instances.

The 3.3.2 release was invalidated due a error late in the release process.

This hotfix release adds disk tracking to the single machine batch system (770). Previously, that batch system ignored any disk requirements associated with jobs. The fix was contributed by arkal.

This major release adds bug fixes and a minor feature, specifically:
- `toil clean` is more robust and can clean partially corrupt job stores (869 et al)
- `job.fileStore.writeGlobalFIle(…)` returns a magic string with a `.size` attribute reflecting the size of the file just having been written. In conjunction with promised requirements, this small change greatly simplifies dynamic computation of job requirements as a function of the size of another job's output (1014)
- Two race conditions in the caching layer are fixed (1046 and 1033)
- A show-stopping race condition with promised requirements was fixed (1039)
- Several other bug fixes and code quality improvements
- The undocumented `cache=` job requirement has been removed

This major release adds many new features, improvements and bug fixes. It is the new stable release and users are encouraged to upgrade to it. Support for the previous stable release, now considered legacy, will continue but be limited to security fixes.

The important new features are:
- Completely rewritten caching allows for sharing of files between worker processes running on the same worker node. Previously, only a chain of jobs running inside a single worker process would benefit from caching.
- Improved ability to run long-running jobs that provide _services_ to other, regular jobs in the workflow.
- Building on top of the improved service jobs feature, we added support for ephemeral Spark sub-clusters that are formed on-demand during a Toil workflow and can be used by jobs to perform ADAM or other Spark workloads.
- Support for hosting job stores in Google Cloud Storage was added. Install Toil with the `google` extra to enable this support. The Google job store should be considered experimental. The Toil team is working on gaining more experience on the consistency and atomicity guarantees provided by Google Cloud Storage to ready that job store implementation for production use.
- Support for hosting job stores in Azure Storage was demoted to experimental status due to atomicity and consistency issues. The Toil team will be working closely with Microsoft engineers to resolve these.
- Added ability to import and export files into a job store on both the leader prior/after to invoking a workflow and on the workers as part of a workflow. Importing files from `http://`, `https://` and `file://` URLs is supported out of the box. So is exporting to `file://` URLs. Installing Toil with any or all of the `azure`, `aws` or `google` extras extends the import/export facility to files hosted on Azure Blob Storage, S3 and Google Cloud Storage, respectively.
- Toil scripts can now have dependencies and those dependencies are deployed automatically to each worker node. The script and the dependencies must be installed in a virtualenv and must be pure-Python. Native dependencies must be installed by the user on each worker prior to running a workflow. The Toil team uses the CGCloud project to handle those native dependencies.
- CWL support has matured.