Luigi

Added
luigi:
- Add a configuration parameter to force multiprocessing (2401)
- Add a configuration parameter to enable/disable the pause button (2399)
- Send messages from scheduler to tasks (via "Send message" UI button) (2426)
- Allow to inject a context manager around `TaskProcess.run` (via `task_process_context` configuration parameter) (2449)
luigi.contrib:
- S3: use Boto3 for the `S3Client` (2423, 2149)
- GCS: add method to push files using multiprocessing (2376)
- HDFS: add `get_merge` to snakebite client (2410)
- Redshift: add schema to DB if it doesn't exist (2439)
- Redshift: add table constraints support (2435)

Fixed
luigi:
- Allow long parameters in task history DB SQL result store (2404)
- Fix `MissingParameterException` when generating execution summary (2415)
- Fix luigid crash due to configuration file parsing (2394)
- Allow explicit parsing of `BoolParameters` (via `luigi.BoolParameter.parsing` variable) (2427)
- Make `ChoiceParameter` check if option is valid within `.normalize` (2454)
- ...and a good deal of documentation fixes and similar.
luigi.contrib:
- BigQuery: fix `bulk_complete` failing when argument is a generator (2441)
- Kubernetes: prevent `KeyError` in `KubernetesJobTask` (2433)
- Kubernetes: don't set `activeDeadlineSeconds` by default (2452)

Fixed
luigi
- append is vulnerable of xss 2391

Added
luigi:
- Add google-auth-httplib2 as dependency (2384)
- Add new company user (2388)
- Allow release of resources during task running. (2346)
luigi.contrib:
- Add parameterized backoff limit (2375)

Fixed
luigi:
- Typo in running luigi documentation (2387)
- Skip running coverage on unreasonable files
- Pass kwargs to `discovery.build()` when instantiating `GSCClient`. (2291)
- Removed message 'No Instance(s) Available.' in Windows when starting … 2294

Added
luigi:
- Added generated data files to `.gitignore` (2367)
luigi.contrib:
- Add possibility to specify `Redshift` column compression (2343)

Changed
luigi:
- Show status message button in worker tab when only progress is set (2344)
- Verbose worker error logging (2353)
luigi.contrib:
- Replace `oauth2client` by `google-auth` (2361)

Fixed
luigi:
- Fix unicode formatting (2339)
luigi.contrib:
- Fix `contrib.docker_runner` exit code check (2356)

Added

luigi:
- A button to show errors for disabled tasks in the visualizer (2266)
- `OptionalParameter` parameter class (2295)

luigi.contrib:
- Support for the Docker Python SDK (2158)

Changed

luigi:
- Change the logging status of prune messages from info to debug (2254) - reduce repetitiveness of logging
- Speed up the visualizer so that a refresh doesn’t take minutes when number of tasks gets into the millions (2239)
- Hide "re-enable" and "forgive failures" buttons on success (2281)
- Convert `date`s to `datetime`s for `DateHourParameter` (2285)
- Handle multiprocessing with request sessions (2290) - Fixes bug with "RPCError: Received null response from remote scheduler"
- Replaced `param_args` with dynamic property with deprecation message
- Check task equality using `param_kwargs` instead of `param_args`

Removed

luigi.contrib:
- Copying of Avro field documentation by `BigQueryLoadAvro` (2269) - said copying is done by BigQuery natively now

Fixed

luigi:
- Remove invalid entrypoint luigi.tools.migrate (2257)
- Preserve filter on server on reload (2273)
- Fix MRO on tasks using `util.requires` (2307)

luigi.contrib:
- Fix Python 3 TypeError in `contrib.hive.HiveTableTarget.exists` (2323)

Luigi 2.7.1 is mostly bug fixes and small feature additions.

* Standardize Redshift credential usage across Redshift tasks: https://github.com/spotify/luigi/pull/2068
* BigQueryLoadAvro now handles complex Avro types: https://github.com/spotify/luigi/pull/2224
* Support for user-specified number of paralleled scheduled processes:
https://github.com/spotify/luigi/pull/2205
* ECS support for non-default cluster: https://github.com/spotify/luigi/pull/2045
* BigQuery ExtractTask support: https://github.com/spotify/luigi/pull/2134
* Support for autocommitting queries within Redshift and Postgres, allowing VACUUM statement execution: https://github.com/spotify/luigi/pull/2242
* High Availability (HA) support with WebHdfsClient using multiple namenodes: https://github.com/spotify/luigi/pull/2230
* Column mapping support for Redshift S3CopyToTable using the column list: https://github.com/spotify/luigi/pull/2245

There have also been some other feature additions, bugfixes, and docfixes. See all commits [here](https://github.com/spotify/luigi/compare/2.7.0...2.7.1).