Datalad

Major refactoring and deprecations

- the `files` argument of [save][] has been renamed to `path` to be uniform with
any other command
- all major commands now implement more uniform API semantics and result reporting.
Functionality for modification detection of dataset content has been completely replaced
with a more efficient implementation
- [publish][] now features a `--transfer-data` switch that allows for a
disambiguous specification of whether to publish data -- independent of
the selection which datasets to publish (which is done via their paths).
Moreover, [publish][] now transfers data before repository content is pushed.

Fixes

- [drop][] no longer errors when some subdatasets are not installed
- [install][] will no longer report nothing when a Dataset instance was
given as a source argument, but rather perform as expected
- [remove][] doesn't remove when some files of a dataset could not be dropped
- [publish][]
- no longer hides error during a repository push
- publish behaves "correctly" for `--since=` in considering only the
differences the last "pushed" state
- data transfer handling while publishing with dependencies, to github
- improved robustness with broken Git configuration
- [search][] should search for unicode strings correctly and not crash
- robustify git-annex special remotes protocol handling to allow for spaces in
the last argument
- UI credentials interface should now allow to Ctrl-C the entry
- should not fail while operating on submodules named with
numerics only or by bool (true/false) names
- crawl templates should not now override settings for `largefiles` if
specified in `.gitattributes`


Enhancements and new features

- **Exciting new feature** [run][] command to protocol execution of an external
command and rerun computation if desired.
See [screencast](http://datalad.org/features.html#reproducible-science)
- [save][] now uses Git for detecting with sundatasets need to be inspected for
potential changes, instead of performing a complete traversal of a dataset tree
- [add][] looks for changes relative to the last committed state of a dataset
to discover files to add more efficiently
- [diff][] can now report untracked files in addition to modified files
- [uninstall][] will check itself whether a subdataset is properly registered in a
superdataset, even when no superdataset is given in a call
- [subdatasets][] can now configure subdatasets for exclusion from recursive
installation (`datalad-recursiveinstall` submodule configuration property)
- precrafted pipelines of [crawl][] now will not override `annex.largefiles`
setting if any was set within `.gitattribues` (e.g. by `datalad create --text-no-annex`)
- framework for screencasts: `tools/cast*` tools and sample cast scripts under
`doc/casts` which are published at [datalad.org/features.html](http://datalad.org/features.html)
- new [project YouTube channel](https://www.youtube.com/channel/UCB8-Zf7D0DSzAsREoIt0Bvw)
- tests failing in direct and/or v6 modes marked explicitly

Bugfixes

Fixes

- Do not attempt to [update][] a not installed sub-dataset
- In case of too many files to be specified for [get][] or [copy_to][], we
will make multiple invocations of underlying git-annex command to not
overfill command line
- More robust handling of unicode output in terminals which might not support it

Enhancements and new features

- Ship a copy of numpy.testing to facilitate [test][] without requiring numpy
as dependency. Also allow to pass to command which test(s) to run
- In [get][] and [copy_to][] provide actual original requested paths, not the
ones we deduced need to be transferred, solely for knowing the total

A variety of fixes and enhancements

Fixes

- [publish][] would now push merged `git-annex` branch even if no other changes
were done
- [publish][] should be able to publish using relative path within SSH URI
(git hook would use relative paths)
- [publish][] should better tollerate publishing to pure git and `git-annex`
special remotes

Enhancements and new features

- [plugin][] mechanism came to replace [export][]. See [export_tarball][] for the
replacement of [export][]. Now it should be easy to extend datalad's interface
with custom functionality to be invoked along with other commands.
- Minimalistic coloring of the results rendering
- [publish][]/`copy_to` got progress bar report now and support of `--jobs`
- minor fixes and enhancements to crawler (e.g. support of recursive removes)

New features, refactorings, and bug fixes.

Major refactoring and deprecations

- [add-sibling][] has been fully replaced by the [siblings][] command
- [create-sibling][], and [unlock][] have been re-written to support the
same common API as most other commands

Enhancements and new features

- [siblings][] can now be used to query and configure a local repository by
using the sibling name ``here``
- [siblings][] can now query and set annex preferred content configuration. This
includes ``wanted`` (as previously supported in other commands), and now
also ``required``
- New [metadata][] command to interface with datasets/files [meta-data][]
- Documentation for all commands is now built in a uniform fashion
- Significant parts of the documentation of been updated
- Instantiate GitPython's Repo instances lazily

Fixes

- API documentation is now rendered properly as HTML, and is easier to browse by
having more compact pages
- Closed files left open on various occasions (Popen PIPEs, etc)
- Restored basic (consumer mode of operation) compatibility with Windows OS

This release includes a **huge** refactoring to make code base and functionality
more robust and flexible

- outputs from API commands could now be highly customized. See
`--output-format`, `--report-status`, `--report-type`, and `--report-type`
options for [datalad][] command.
- effort was made to refactor code base so that underlying functions behave as
generators where possible
- input paths/arguments analysis was redone for majority of the commands to provide
unified behavior

Major refactoring and deprecations

- `add-sibling` and `rewrite-urls` were refactored in favor of new [siblings][]
command which should be used for siblings manipulations
- 'datalad.api.alwaysrender' config setting/support is removed in favor of new
outputs processing

Fixes

- Do not flush manually git index in pre-commit to avoid "Death by the Lock" issue
- Deployed by [publish][] `post-update` hook script now should be more robust
(tolerate directory names with spaces, etc.)
- A variety of fixes, see
[list of pull requests and issues closed](https://github.com/datalad/datalad/milestone/41?closed=1)
for more information

Enhancements and new features

- new [annotate-paths][] plumbing command to inspect and annotate provided
paths. Use `--modified` to summarize changes between different points in
the history
- new [clone][] plumbing command to provide a subset (install a single dataset
from a URL) functionality of [install][]
- new [diff][] plumbing command
- new [siblings][] command to list or manipulate siblings
- new [subdatasets][] command to list subdatasets and their properties
- [drop][] and [remove][] commands were refactored
- `benchmarks/` collection of [Airspeed velocity](https://github.com/spacetelescope/asv/)
benchmarks initiated. See reports at http://datalad.github.io/datalad/
- crawler would try to download a new url multiple times increasing delay between
attempts. Helps to resolve problems with extended crawls of Amazon S3
- [CRCNS][] crawler pipeline now also fetches and aggregates meta-data for the
datasets from datacite
- overall optimisations to benefit from the aforementioned refactoring and
improve user-experience
- a few stub and not (yet) implemented commands (e.g. `move`) were removed from
the interface
- Web frontend got proper coloring for the breadcrumbs and some additional
caching to speed up interactions. See http://datasets.datalad.org
- Small improvements to the online documentation. See e.g.
[summary of differences between git/git-annex/datalad](http://docs.datalad.org/en/latest/related.html#git-git-annex-datalad)

A bugfix release

Fixes

- [add][] was forcing addition of files to annex regardless of settings
in `.gitattributes`. Now that decision is left to annex by default
- `tools/testing/run_doc_examples` used to run
doc examples as tests, fixed up to provide status per each example
and not fail at once
- `doc/examples`
- [3rdparty_analysis_workflow.sh](http://docs.datalad.org/en/latest/generated/examples/3rdparty_analysis_workflow.html)
was fixed up to reflect changes in the API of 0.5.0.
- progress bars
- should no longer crash **datalad** and report correct sizes and speeds
- should provide progress reports while using Python 3.x

Enhancements and new features

- `doc/examples`
- [nipype_workshop_dataset.sh](http://docs.datalad.org/en/latest/generated/examples/nipype_workshop_dataset.html)
new example to demonstrate how new super- and sub- datasets were established
as a part of our datasets collection