Mlflow

Bug fixes and improvements:

- Fixes `AttributeError` when pickling an instance of the Python `MlflowClient` class (2955, Polyphenolx)
- Fixes bug that prevented updating model-version descriptions in the model registry UI (2969, AnastasiaKol)
- Fixes bug where credentials were not properly propagated to artifact CLI commands when logging artifacts from Java to the DatabricksArtifactRepository (3001, dbczumar)
- Removes use of new Pandas API in new MLflow model-schema functionality, so that it can be used with older Pandas versions (2988, aarondav)

Small bug fixes and doc updates (2998, dbczumar; 2999, arjundc-db)

experimental APIs:

Breaking Changes:

- The `new_name` argument to `MlflowClient.update_registered_model`
has been removed. Call `MlflowClient.rename_registered_model` instead. (2946, mparkhe)
- The `stage` argument to `MlflowClient.update_model_version`
has been removed. Call `MlflowClient.transition_model_version_stage` instead. (2946, mparkhe)

Features (MLflow Models and Flavors)

- `log_model` and `save_model` APIs now support saving model signatures (the model's input and output schema)
and example input along with the model itself (2698, 2775, tomasatdatabricks). Model signatures are used
to reorder and validate input fields when scoring/serving models using the pyfunc flavor, `mlflow models`
CLI commands, or `mlflow.pyfunc.spark_udf` (2920, tomasatdatabricks and aarondav)
- Introduce fastai model persistence and autologging APIs under `mlflow.fastai` (2619, 2689 antoniomdk)
- Add pluggable `mlflow.deployments` API and CLI for deploying models to custom serving tools, e.g. RedisAI
(2327, hhsecond)
- Enables loading and scoring models whose conda environments include dependencies in conda-forge (2797, dbczumar)
- Add support for scoring ONNX-persisted models that return Python lists (2742, andychow-db)

Features (MLflow Projects)

- Add plugin interface for executing MLflow projects against custom backends (2566, jdlesage)
- Add ability to specify additional cluster-wide Python and Java libraries when executing
MLflow projects remotely on Databricks (2845, pogil)
- Allow running MLflow projects against remote artifacts stored in any location with a corresponding
ArtifactRepository implementation (Azure Blob Storage, GCS, etc) (2774, trangevi)
- Allow MLflow projects running on Kubernetes to specify a different tracking server to log to via the
`KUBE_MLFLOW_TRACKING_URI` for passing a different tracking server to the kubernetes job (2874, catapulta)

Features (UI)

- Significant performance and scalability improvements to metric comparison and scatter plots in
the UI (2447, mjlbach)
- The main MLflow experiment list UI now includes a link to the model registry UI (2805, zhidongqu-db),
- Enable viewing PDFs logged as artifacts from the runs UI (2859, ankmathur96)
- UI accessibility improvements: better color contrast (2872, Zangr), add child roles to DOM elements (2871, Zangr)

Features (Tracking Client and Server)

- Adds ability to pass client certs as part of REST API requests when using the tracking or model
registry APIs. (2843, PhilipMay)
- New community plugin: support for storing artifacts in Aliyun (Alibaba Cloud) (2917, SeaOfOcean)
- Infer and set content type and encoding of objects when logging models and artifacts to S3 (2881, hajapy)
- Adds support for logging artifacts to HDFS Federation ViewFs (2782, fhoering)
- Add healthcheck endpoint to the MLflow server at `/health` (2725, crflynn)
- Improves performance of default file-based tracking storage backend by using LibYAML (if installed)
to read experiment and run metadata (2707, Higgcz)

Bug fixes and documentation updates:

- Several UI fixes: remove margins around icon buttons (2827, harupy),
fix alignment issues in metric view (2811, zhidongqu-db), add handling of `NaN`
values in metrics plot (2773, dbczumar), truncate run ID in the run name when
comparing multiple runs (2508, harupy)
- Database engine URLs are no longer logged when running `mlflow db upgrade` (2849, hajapy)
- Updates `log_artifact`, `log_model` APIs to consistently use posix paths, rather than OS-dependent
paths, when computing artifact subpaths. (2784, mikeoconnor0308)
- Fix `ValueError` when scoring `tf.keras` 1.X models using `mlflow.pyfunc.predict` (2762, juntai-zheng)
- Fixes conda environment activation bug when running MLflow projects on Windows (2731, MynherVanKoek)
- `mlflow.end_run` will now clear the active run even if the run cannot be marked as
terminated (e.g. because it's been deleted), (2693, ahmed-shariff)
- Add missing documentation for `mlflow.spacy` APIs (2771, harupy)

Small bug fixes and doc updates (2919, willzhan-db; 2940, 2942, 2941, 2943, 2927, 2929, 2926, 2914, 2928, 2913, 2852, 2876, 2808, 2810, 2442, 2780, 2758, 2732, 2734, 2431, 2733, 2716, harupy; 2915, 2897, jwgwalton; 2856, jkthompson; 2962, hhsecond; 2873, 2829, 2582, dmatrix; 2908, 2865, 2880, 2866, 2833, 2785, 2723, smurching; 2906, dependabot[bot]; 2724, aarondav; 2896, ezeeetm; 2741, 2721, mlflow-automation; 2864, tallen94; 2726, crflynn; 2710, 2951 mparkhe; 2935, 2921, ankitmathur-db; 2963, 2739, dbczumar; 2853, stat4jason; 2709, 2792, juntai-zheng juntai-zheng; 2749, HiromuHota; 2957, 2911, 2718, arjundc-db; 2885, willzhan-db; 2803, 2761, pogil; 2392, jnmclarty; 2794, Zethson; 2766, 2916 shubham769)

Features:

- Added `mlflow.azureml.deploy` API for deploying MLflow models to AzureML (2375 csteegz, 2711, akshaya-a)
- Added support for case-sensitive LIKE and case-insensitive ILIKE queries (e.g. `'params.framework LIKE '%sklearn%'`) with the SearchRuns API & UI when running against a SQLite backend (2217, t-henri; 2708, mparkhe)
- Improved line smoothing in MLflow metrics UI using exponential moving averages (2620, Valentyn1997)
- Added `mlflow.spacy` module with support for logging and loading spaCy models (2242, arocketman)
- Parameter values that differ across runs are highlighted in run comparison UI (2565, gabrielbretschner)
- Added ability to compare source runs associated with model versions from the registered model UI (2537, juntai-zheng)
- Added support for alphanumerical experiment IDs in the UI. (2568, jonas)
- Added support for passing arguments to `docker run` when running docker-based MLflow projects (2608, ksanjeevan)
- Added Windows support for `mlflow sagemaker build-and-push-container` CLI & API (2500, AndreyBulezyuk)
- Improved performance of reading experiment data from local filesystem when LibYAML is installed (2707, Higgcz)
- Added a healthcheck endpoint to the REST API server at `/health` that always returns a 200 response status code, to be used to verify health of the server (2725, crflynn)
- MLflow metrics UI plots now scale to rendering thousands of points using scattergl (2447, mjlbach)

Bug fixes:

- Fixed CLI summary message in `mlflow azureml build_image` CLI (2712, dbczumar)
- Updated `examples/flower_classifier/score_images_rest.py` with multiple bug fixes (2647, tfurmston)
- Fixed pip not found error while packaging models via `mlflow models build-docker` (2699, HiromuHota)
- Fixed bug in `mlflow.tensorflow.autolog` causing erroneous deletion of TensorBoard logging directory (2670, dbczumar)
- Fixed a bug that truncated the description of the `mlflow gc` subcommand in `mlflow --help` (2679, dbczumar)
- Fixed bug where `mlflow models build-docker` was failing due to incorrect Miniconda download URL (2685, michaeltinsley)
- Fixed a bug in S3 artifact logging functionality where `MLFLOW_S3_ENDPOINT_URL` was ignored (2629, poppash)
- Fixed a bug where Sqlite in-memory was not working as a tracking backend store by modifying DB upgrade logic (2667, dbczumar)
- Fixed a bug to allow numerical parameters with values >= 1000 in R `mlflow::mlflow_run()` API (2665, lorenzwalthert)
- Fixed a bug where AWS creds was not found in the Windows platform due path differences (2634, AndreyBulezyuk)
- Fixed a bug to add pip when necessary in `_mlflow_conda_env` (2646, tfurmston)
- Fixed error code to be more meaningful if input to model version is incorrect (2625, andychow-db)
- Fixed multiple bugs in model registry (2638, aarondav)
- Fixed support for conda env dicts with `mlflow.pyfunc.log_model` (2618, dbczumar)
- Fixed a bug where hiding the start time column in the UI would also hide run selection checkboxes (2559, harupy)

Documentation updates:

- Added links to source code to mlflow.org (2627, harupy)
- Documented fix for pandas-records payload (2660, SaiKiranBurle)
- Fixed documentation bug in TensorFlow `load_model` utility (2666, pogil)
- Added the missing Model Registry description and link on the first page (2536, dmatrix)
- Added documentation for expected datatype for step argument in `log_metric` to match REST API (2654, mparkhe)
- Added usage of the model registry to the `log_model` function in `sklearn_elasticnet_wine/train.py` example (2609, netanel246)

Small bug fixes and doc updates (2594, Trollgeir; 2703,2709, juntai-zheng; 2538, 2632, keigohtr; 2656, 2553, lorenzwalthert; 2622, pingsutw; 2615, 2600, 2533, mlflow-automation; 1391, sueann; 2613, 2598, 2534, 2723, smurching; 2652, 2710, mparkhe; 2706, 2653, 2639, tomasatdatabricks; 2611, 9dogs; 2700, 2705, aarondav; 2675, 2540, mengxr; 2686, RensDimmendaal; 2694, 2695, 2532, dbczumar; 2733, 2716, harupy; 2726, crflynn; 2582, 2687, dmatrix)

- Pin alembic version to 1.4.1 or below to prevent pep517-related installation errors
(2612, smurching)

- Remove usage of Nonnull annotations and findbugs dependency in Java package (2583, mparkhe)
- Add version upper bound (<=1.3.13) to sqlalchemy dependency in Python package (2587, smurching)

Other bugfixes and doc updates (2595, mparkhe; 2567, jdlesage)

MLflow support for Python 2 is now deprecated and will be dropped in a future release. At that
point, existing Python 2 workflows that use MLflow will continue to work without modification, but
Python 2 users will no longer get access to the latest MLflow features and bugfixes. We recommend
that you upgrade to Python 3 - see https://docs.python.org/3/howto/pyporting.html for a migration
guide.

Breaking changes to Model Registry REST APIs:

Model Registry REST APIs have been updated to be more consistent with the other MLflow APIs. With
this release Model Registry APIs are intended to be stable until the next major version.

- Python and Java client APIs for Model Registry have been updated to use the new REST APIs. When using an MLflow client with a server using updated REST endpoints, you won't need to change any code but will need to upgrade to a new client version. The client APIs contain deprecated arguments, which for this release are backward compatible, but will be dropped in future releases. (2457, tomasatdatabricks; 2502, mparkhe).
- The Model Registry UI has been updated to use the new REST APIs (2476 aarondav; 2507, mparkhe)

Other Features:

- Ability to click through to individual runs from metrics plot (2295, harupy)
- Added `mlflow gc` CLI for permanent deletion of runs (2265, t-henri)
- Metric plot state is now captured in page URLs for easier link sharing (2393, 2408, 2498 smurching; 2459, harupy)
- Added experiment management to MLflow UI (create/rename/delete experiments) (2348, ggliem)
- Ability to search for experiments by name in the UI (2324, ggliem)
- MLflow UI page titles now reflect the content displayed on the page (2420, AveshCSingh)
- Added a new `LogModel` REST API endpoint for capturing model metadata, and call it from the Python and R clients (2369, 2430, 2468 tomasatdatabricks)
- Java Client API to download model artifacts from Model Registry (2308, andychow-db)

Bug fixes and documentation updates:

- Updated Model Registry documentation page with code snippets and examples (2493, dmatrix; 2517, harupy)
- Better error message for Model Registry, when using incompatible backend server (2456, aarondav)
- matplotlib is no longer required to use XGBoost and LightGBM autologging (2423, harupy)
- Fixed bug where matplotlib figures were not closed in XGBoost and LightGBM autologging (2386, harupy)
- Fixed parameter reading logic to support param values with newlines in FileStore (2376, dbczumar)
- Improve readability of run table column selector nodes (2388, dbczumar)
- Validate experiment name supplied to `UpdateExperiment` REST API endpoint (2357, ggliem)
- Fixed broken MLflow DB README link in CLI docs (2377, dbczumar)
- Change copyright year across docs to 2020 (2349, ParseThis)

Small bug fixes and doc updates (2378, 2449, 2402, 2397, 2391, 2387, 2523, 2527 harupy; 2314, juntai-zheng; 2404, andychow-db; 2343, pogil; 2366, 2370, 2364, 2356, AveshCSingh; 2373, 2365, 2363, smurching; 2358, jcuquemelle; 2490, RensDimmendaal; 2506, dbczumar; 2234 Zangr; 2359 lbernickm; 2525, mparkhe)