Luigi

Added

luigi:
- A button to show errors for disabled tasks in the visualizer (2266)
- `OptionalParameter` parameter class (2295)

luigi.contrib:
- Support for the Docker Python SDK (2158)

Changed

luigi:
- Change the logging status of prune messages from info to debug (2254) - reduce repetitiveness of logging
- Speed up the visualizer so that a refresh doesn’t take minutes when number of tasks gets into the millions (2239)
- Hide "re-enable" and "forgive failures" buttons on success (2281)
- Convert `date`s to `datetime`s for `DateHourParameter` (2285)
- Handle multiprocessing with request sessions (2290) - Fixes bug with "RPCError: Received null response from remote scheduler"
- Replaced `param_args` with dynamic property with deprecation message
- Check task equality using `param_kwargs` instead of `param_args`

Removed

luigi.contrib:
- Copying of Avro field documentation by `BigQueryLoadAvro` (2269) - said copying is done by BigQuery natively now

Fixed

luigi:
- Remove invalid entrypoint luigi.tools.migrate (2257)
- Preserve filter on server on reload (2273)
- Fix MRO on tasks using `util.requires` (2307)

luigi.contrib:
- Fix Python 3 TypeError in `contrib.hive.HiveTableTarget.exists` (2323)

Luigi 2.7.1 is mostly bug fixes and small feature additions.

* Standardize Redshift credential usage across Redshift tasks: https://github.com/spotify/luigi/pull/2068
* BigQueryLoadAvro now handles complex Avro types: https://github.com/spotify/luigi/pull/2224
* Support for user-specified number of paralleled scheduled processes:
https://github.com/spotify/luigi/pull/2205
* ECS support for non-default cluster: https://github.com/spotify/luigi/pull/2045
* BigQuery ExtractTask support: https://github.com/spotify/luigi/pull/2134
* Support for autocommitting queries within Redshift and Postgres, allowing VACUUM statement execution: https://github.com/spotify/luigi/pull/2242
* High Availability (HA) support with WebHdfsClient using multiple namenodes: https://github.com/spotify/luigi/pull/2230
* Column mapping support for Redshift S3CopyToTable using the column list: https://github.com/spotify/luigi/pull/2245

There have also been some other feature additions, bugfixes, and docfixes. See all commits [here](https://github.com/spotify/luigi/compare/2.7.0...2.7.1).

- _Removed deprecations!_ `luigi.{postgres, s3}` are removed, use `luigi.contrib.{..}` instead https://github.com/spotify/luigi/pull/2181
- As of the bullet above. We now believe all contrib and core modules are well seperated by either being under either `luigi.xyx` or `luigi.contrib.xyz`. This should simplify to move modules into seperate repositories, which is future work.
- Luigi finally has a progress bar! https://github.com/spotify/luigi/pull/2108
- Scheduling improvements / bug fixes: https://github.com/spotify/luigi/pull/2208 https://github.com/spotify/luigi/pull/2207
- Luigi can now run PySpark in *cluster mode* https://github.com/spotify/luigi/pull/2197
- The batch email notifier shouldn't stall any more on failures. https://github.com/spotify/luigi/pull/2177
- A global panic "stop scheduling button" https://github.com/spotify/luigi/pull/2145
- Fixed error emails from workers: https://github.com/spotify/luigi/pull/2191
- My personal favorite PR, as it removes many deprecation warnings when running the tests: https://github.com/spotify/luigi/pull/2130

There have been a lot of other bugfixes, docsfixes, cleanups and added testcases! See all commits
[here](https://github.com/spotify/luigi/compare/2.6.2...2.7.0).

Luigi 2.6.2 is also mainly about bug-fixes. Including many fixes in the scheduler web interface.

* One new feature is forgiving failures. https://github.com/spotify/luigi/pull/2091 (thanks FxIII!)
* The annoying "is not of type" warnings are now easier to fix/trace, as the var name is also included in the log message https://github.com/spotify/luigi/pull/2107 (thanks Lucas-C!)

* Fixed the installation issue that plagued 2.6.0 https://github.com/spotify/luigi/pull/2033 (thanks Aaronhrndz)
* Fixes to `temporary_path` decorator. Awesome! (https://github.com/spotify/luigi/pull/2057 and https://github.com/spotify/luigi/pull/2058, thanks adaitche!)

There actually is more! Read the whole commit log [here](https://github.com/spotify/luigi/compare/2.6.0...2.6.1)!

- _Removed deprecations!_ `luigi.{hadoop, hadoop_jar, hdfs, hive, scalding, webhdfs}` are removed, use `luigi.contrib.{..}` instead https://github.com/spotify/luigi/pull/1995
- _Deprecations!_ `luigi.{postgres, s3}` are now moved into `luigi.contrib`. https://github.com/spotify/luigi/pull/1997
- Multiple workers finally works for Windows again! https://github.com/spotify/luigi/pull/1992
- Server can now communicate with the clients! We started small and implemented so `--workers` can be set. https://github.com/spotify/luigi/pull/1993
- Make the visualizer put your search queries in the URLs hash. So links are now finally shareable! https://github.com/spotify/luigi/pull/1986 https://github.com/spotify/luigi/pull/2002
- A new recommended way to automatically set the task namespace! https://github.com/spotify/luigi/pull/2000 [(docs)](http://luigi.readthedocs.io/en/latest/api/luigi.task.html#luigi.task.auto_namespace)

There have been a lot of other bugfixes, docsfixes, cleanups and added testcases! See all commits
[here](https://github.com/spotify/luigi/compare/2.5.0...2.6.0).