Luigi

Finally, a new PyPI release in a rather long while. Thanks Tarrasch and erikbern, and all the contributors!

Added

luigi:
- Notifications: more emails and proper coloring (1471), improved SMTP handling
- `EnumParameter` (1479), `DictParameter` (1574)
- Support for Python 3.5 (1494)
- Process locking on Alpine Linux (1530) and Windows (1557)
- Visualizer: resources tab (1566), GUI functionality to disable a worker (1564)

luigi.contrib:
- `ExternalBigqueryTask` (1434), `BigqueryCreateViewTask` (1465)
- Luigi tasks for Dataproc, Google's managed Hadoop MapReduce, Spark, Pig, and Hive service (1601)
- `ExternalProgramTask`, `ExternalPythonProgramTask` - commonalities for running any external application or script (1520)
- Support for SFTP (1585)
- Sped-up Hive client using Metastore (1533)
- `OpenerTarget`, a single Luigi target to open multiple file system types (1555)
- `Query` base task (giving rise also to `luigi.contrib.redshift.RedshiftQuery` and `luigi.postgres.PostgresQuery`) (1493)
- `RedshiftUnloadTask` (1527)
- `UploadToSalesforceTask` (1404)
- Support for S3 assumed role (1596)

Changed

luigi:
- Semi-opaque, hashed `task_id` (as opposed to `TaskName(param1=value1, param2=foo bar)`) (1444)
- More explicit way to handle timelike parameters (`date` vs `datetime`) (1473)
- Optimizations in scheduler algorithm

Removed

luigi:
- Old deprecated (2014) stuff around scheduler and its state (1592)

luigi.contrib:
- Deprecated classes `SparkJob`, `Spark1xBackwardCompat`, `Spark1xJob`, `PySpark1xJob` (1442)

Fixed

luigi.contrib:
- Ensure that FTP `RemoteTarget` successfully creates temporary files (meaning, in a directory relative to output) (1515)
- Remove superfluous `init_mapper()`/`init_reducer()` calls in `LocalJobRunner` (1475)
- Humanly format `HadoopJobError` (1528)
- Broken Redshift table creation (1453)
- Improved Salesforce reliability (1597, 1600)
- Missing call to `post_copy()` (1502)

...and a slew of other additions, fixes, improvements and documentation.

This version contains mostly bugfixes, in particular making Luigi 2.0 working on Windows

Thanks daveFNbuck econchick Tarrasch dlstadther DeaconDesperado jpiper nresare dhurlburtusa stephenpascoe mbruggmann and some more people I probably forgot!

Pull requests
- 1441 don't print nested stack traces
- 1439 spark: use list2command
- 1447 fix unix socket
- 1446 fix daemon
- 1448 support symlinks in hdfs
- 1431 1433 worker as a context manager
- 1429 keyboardinterrupt fix
- 1428 make it work without tornado
- 1425 positional parameter qualification
- 1422 fixed broken task trigger
- 1411 mention misspelled task name
- 1418 bigquery fix
- 1417 visualizer fix
- 1413 kwargs for s3
- 1345 retry get_work
- 1315 limit graph size
- 1391 amazon sns support
- 1392 pyspark parameter fix
- 1395 tornado fix
- 1397 spark package fix
- 1400 serialize TaskParameter
- 1402 kwargs for boto s3
- 1388 disable email from cfg
- 1341 get_work fix
- 1340 stop subprocesses on exception
- 1352 fix bug in scheduler
- 1357 fix issues in requires during scheduling
- 1367 fix visualizer w weird task names
- 1372 s3target.path
- 1314 redshift updates
- 1370 gmail authentication for notifications
- 1366 pig fix
- 1331 path warning fix
- 1338 ssh forwarding agent
- 1337 ssh multiplexing
- 1363 windows signal fix
- 1298 tracking fix
- 1346 fix dep graph on firefox

Highlights:
- Improved documentation.
- New visualiser page! https://github.com/spotify/luigi/pull/1086, thanks stephenpascoe
- Execution summaries at end of runs! https://github.com/spotify/luigi/pull/1091, thanks nicolehedblom
- Smarter layout algorithm for the SVG dependency graphs, thanks daveFNbuck
- Command line interface now has a useful --help, see https://github.com/spotify/luigi/pull/1276
- Luigi finally has semantic exit codes! https://github.com/spotify/luigi/pull/1264

Contrib:
- Stablization of GCP tools
- AWS EC2 support. https://github.com/spotify/luigi/pull/1317, thanks jfeala
- Sungrid support. https://github.com/spotify/luigi/pull/1124, thanks jfeala (again :))

Scheduling:
- UNIX sockets are now supported, hence also encryption. https://github.com/spotify/luigi/pull/1129, thanks graingert
- Smarter scheduling! https://github.com/spotify/luigi/pull/1272
- Speedup in scheduler. https://github.com/spotify/luigi/pull/1046
- A `--take-lock` option to smoother roll out code changes. https://github.com/spotify/luigi/pull/1137

Deprecations:
- There have been a lot of deprecations of features regarding Parameters. Before upgrading, please consider taking a look at the new [module documentation for parameters](http://luigi.readthedocs.org/en/stable/api/luigi.parameter.html).

Notable changes:
- `is_global` finally removed! (It still exists to not break old code, but it only sets `positional=False`)
- Initial support for Google Cloud Storage and Google BigQuery. Thanks mikekap! https://github.com/spotify/luigi/pull/999 https://github.com/spotify/luigi/pull/1002
- Massive luigid speedup for `get_work()` response times. https://github.com/spotify/luigi/pull/986
- Various bugfixes.
- Standardized FileSystem/Target interface. Again thanks mikekap! https://github.com/spotify/luigi/pull/997
- SQLAlchemy works for multiple engines. Thanks dylanjbarth! https://github.com/spotify/luigi/pull/991

Same as v1.2.1. Only that we can't have a initial "v" as otherwise readthedocs won't build the documentation for us. See https://github.com/spotify/luigi/issues/970#issuecomment-105623162

Notable changes:
- Dropped Python 2.6 support. https://github.com/spotify/luigi/pull/934
- New experimental ["assistant"](https://github.com/spotify/luigi/blob/ac530c6cb3ac4df81ad236817373bdac19effa6d/luigi/interface.py#L104) feature.
- External tasks are now also registered. This might cause breakage in settings like described [here](https://github.com/spotify/luigi/pull/880#issuecomment-94008171)
- Many configuration settings have been hoisted to the new configuration system. Particularly for hdfs-related parameters. The `--help` command will show what new options are available.
- While `is_global` is not removed yet. It'll get removed anytime soon. Please use the new `positional=False` setting as a replacement in your parameters. In fact, we suggest to always use `positional=False`.
- The central scheduler now supports searching within groups. https://github.com/spotify/luigi/pull/903
- New dependency graph rendering using D3. https://github.com/spotify/luigi/pull/920
- The built in map reduce framework can now use `ujson` instead of doing `eval`. It's still not clear to us how much faster it is. https://github.com/spotify/luigi/pull/939
- Various bug fixes and other features.