Joblib

--------------

- Make joblib work on Python 3 installation that do not ship with the lzma
package in their standard library.

--------------

- Drop support for Python 2 and Python 3.5. All objects in
``joblib.my_exceptions`` and ``joblib.format_stack`` are now deprecated and
will be removed in joblib 0.16. Note that no deprecation warning will be
raised for these objects Python < 3.7.
https://github.com/joblib/joblib/pull/1018

- Fix many bugs related to the temporary files and folder generated when
automatically memory mapping large numpy arrays for efficient inter-process
communication. In particular, this would cause `PermissionError` exceptions
to be raised under Windows and large leaked files in `/dev/shm` under Linux
in case of crash.
https://github.com/joblib/joblib/pull/966

- Make the dask backend collect results as soon as they complete
leading to a performance improvement:
https://github.com/joblib/joblib/pull/1025

- Fix the number of jobs reported by ``effective_n_jobs`` when ``n_jobs=None``
called in a parallel backend context.
https://github.com/joblib/joblib/pull/985

- Upgraded vendored cloupickle to 1.4.1 and loky to 2.8.0. This allows for
Parallel calls of dynamically defined functions with type annotations
in particular.

--------------

- Configure the loky workers' environment to mitigate oversubsription with
nested multi-threaded code in the following case:

- allow for a suitable number of threads for numba (``NUMBA_NUM_THREADS``);

- enable Interprocess Communication for scheduler coordination when the
nested code uses Threading Building Blocks (TBB) (``ENABLE_IPC=1``)

https://github.com/joblib/joblib/pull/951

- Fix a regression where the loky backend was not reusing previously
spawned workers.
https://github.com/joblib/joblib/pull/968

- Revert https://github.com/joblib/joblib/pull/847 to avoid using
`pkg_resources` that introduced a performance regression under Windows:
https://github.com/joblib/joblib/issues/965

--------------

- Improved the load balancing between workers to avoid stranglers caused by an
excessively large batch size when the task duration is varying significantly
(because of the combined use of ``joblib.Parallel`` and ``joblib.Memory``
with a partially warmed cache for instance).
https://github.com/joblib/joblib/pull/899

- Add official support for Python 3.8: fixed protocol number in `Hasher`
and updated tests.

- Fix a deadlock when using the dask backend (when scattering large numpy
arrays).
https://github.com/joblib/joblib/pull/914

- Warn users that they should never use `joblib.load` with files from
untrusted sources. Fix security related API change introduced in numpy
1.6.3 that would prevent using joblib with recent numpy versions.
https://github.com/joblib/joblib/pull/879

- Upgrade to cloudpickle 1.1.1 that add supports for the upcoming
Python 3.8 release among other things.
https://github.com/joblib/joblib/pull/878

- Fix semaphore availability checker to avoid spawning resource trackers
on module import.
https://github.com/joblib/joblib/pull/893

- Fix the oversubscription protection to only protect against nested
`Parallel` calls. This allows `joblib` to be run in background threads.
https://github.com/joblib/joblib/pull/934

- Fix `ValueError` (negative dimensions) when pickling large numpy arrays on
Windows.
https://github.com/joblib/joblib/pull/920

- Upgrade to loky 2.6.0 that add supports for the setting environment variables
in child before loading any module.
https://github.com/joblib/joblib/pull/940

- Fix the oversubscription protection for native libraries using threadpools
(OpenBLAS, MKL, Blis and OpenMP runtimes).
The maximal number of threads is can now be set in children using the
``inner_max_num_threads`` in ``parallel_backend``. It defaults to
``cpu_count() // n_jobs``.
https://github.com/joblib/joblib/pull/940

--------------

Pierre Glaser

Upgrade to cloudpickle 0.8.0

Add a non-regression test related to joblib issues 836 and 833, reporting
that cloudpickle versions between 0.5.4 and 0.7 introduced a bug where
global variables changes in a parent process between two calls to
joblib.Parallel would not be propagated into the workers

--------------

Pierre Glaser

Memory now accepts pathlib.Path objects as ``location`` parameter.
Also, a warning is raised if the returned backend is None while
``location`` is not None.

Olivier Grisel

Make ``Parallel`` raise an informative ``RuntimeError`` when the
active parallel backend has zero worker.

Make the ``DaskDistributedBackend`` wait for workers before trying to
schedule work. This is useful in particular when the workers are
provisionned dynamically but provisionning is not immediate (for
instance using Kubernetes, Yarn or an HPC job queue).