Snakemake

Added

- A new shadow mode (minimal) that only symlinks input files has been
added.

Changed

- The default shell is now bash on linux and macOS. If bash is not
installed, we fall back to sh. Previously, Snakemake used the
default shell of the user, which defeats the purpose of portability.
If the developer decides so, the shell can be always overwritten
using shell.executable().
- Snakemake now requires Singularity 2.4.1 at least (only when running
with --use-singularity).
- HTTP remote provider no longer automatically unpacks gzipped files.
- Fixed various smaller bugs.

Added

- List all conda environments with their location on disk via
--list-conda-envs.

Changed

- Do not clean up shadow on dry-run.
- Allow R wrappers.

Added

- GridFTP remote provider. This is a specialization of the GFAL remote
provider that uses globus-url-copy to download or upload files.
Changed
- Scheduling and execution mechanisms have undergone a major revision
that removes several potential (but rare) deadlocks.
- Several bugs and corner cases of the singularity support have been
fixed.
- Snakemake now requires singularity 2.4 at least.

Added

- Support for executing jobs in per-rule singularity images. This is
meant as an alternative to the conda directive (see docs), providing
even more guarantees for reproducibility.

Changed

- In cluster mode, jobs that are still running after Snakemake has
been killed are automatically resumed.
- Various fixes to GFAL remote provider.
- Fixed --summary and --list-code-changes.
- Many other small bug fixes.

Added

- Support for configuration profiles. Profiles allow to specify
default options, e.g., a cluster submission command. They can be
used via 'snakemake --profile myprofile'. See the docs for details.
- GFAL remote provider. This allows to use GridFTP, SRM and any other
protocol supported by GFAL for remote input and output files.
- Added --cluster-status flag that allows to specify a command that
returns jobs status. Changed
- The scheduler now tries to get rid of the largest temp files first.
- The Docker image used for kubernetes support can now be configured
at the command line.
- Rate-limiting for cluster interaction has been unified.
- S3 remote provider uses boto3.
- Resource functions can now use an additional `attempt` parameter,
that contains the number of times this job has already been tried.
- Various minor fixes.

Added

- Cloud computing support via Kubernetes. Snakemake workflows can be
executed transparently in the cloud, while storing input and output
files within the cloud storage (e.g. S3 or Google Storage). I.e.,
this feature does not need a shared filesystem between the cloud
notes, and thereby makes the setup really simple.
- WebDAV remote file support: Snakemake can now read and write from
WebDAV. Hence, it can now, e.g., interact with Nextcloud or
Owncloud.
- Support for default remote providers: define a remote provider to
implicitly use for all input and output files.
- Added an option to only create conda environments instead of
executing the workflow. Changed
- The number of files used for the metadata tracking of Snakemake
(e.g., code, params, input changes) in the .snakemake directory has
been reduced by a factor of 10, which should help with NFS and IO
bottlenecks. This is a breaking change in the sense that Snakemake
4.x won't see the metadata of workflows executed with Snakemake 3.x.
However, old metadata won't be overwritten, so that you can always
go back and check things by installing an older version of Snakemake
again.
- The google storage (GS) remote provider has been changed to use the
google SDK. This is a breaking change, since the remote provider
invocation has been simplified (see docs).
- Due to WebDAV support (which uses asyncio), Snakemake now requires
Python 3.5 at least.
- Various minor bug fixes (e.g. for dynamic output files).