Mlflow

Features:

- [Tracking] Introduce a new fluent API `mlflow.last_active_run()` that provides the most recent fluent active run (5584, MarkYHZhang)
- [Tracking] Add `experiment_names` argument to the `mlflow.search_runs()` API to support searching runs by experiment names (5564, r3stl355)
- [Tracking] Add a `description` parameter to `mlflow.start_run()` (5534, dogeplusplus)
- [Tracking] Add `log_every_n_step` parameter to `mlflow.pytorch.autolog()` to control metric logging frequency (5516, adamreeve)
- [Tracking] Log `pyspark.ml.param.Params` values as MLflow parameters during PySpark autologging (5481, serena-ruan)
- [Tracking] Add support for `pyspark.ml.Transformer`s to PySpark autologging (5466, serena-ruan)
- [Tracking] Add input example and signature autologging for Keras models (5461, bali0019)
- [Models] Introduce `mlflow.diviner` flavor for large-scale [time series forecasting](https://databricks-diviner.readthedocs.io/en/latest/?badge=latest) (#5553, BenWilson2)
- [Models] Add `pyfunc.get_model_dependencies()` API to retrieve reproducible environment specifications for MLflow Models with the pyfunc flavor (5503, WeichenXu123)
- [Models] Add `code_paths` argument to all model flavors to support packaging custom module code with MLflow Models (5448, stevenchen-db)
- [Models] Support creating custom artifacts when evaluating models with `mlflow.evaluate()` (5405, 5476 MarkYHZhang)
- [Models] Add `mlflow_version` field to MLModel specification (5515, 5576, r3stl355)
- [Models] Add support for logging models to preexisting destination directories (5572, akshaya-a)
- [Scoring / Projects] Introduce `--env-manager` configuration for specifying environment restoration tools (e.g. `conda`) and deprecate `--no-conda` (5567, harupy)
- [Scoring] Support restoring model dependencies in `mlflow.pyfunc.spark_udf()` to ensure accurate predictions (5487, 5561, WeichenXu123)
- [Scoring] Add support for `numpy.ndarray` type inputs to the TensorFlow pyfunc `predict()` function (5545, WeichenXu123)
- [Scoring] Support deployment of MLflow Models to Sagemaker Serverless (5610, matthewmayo)
- [UI] Add MLflow version to header beneath logo (5504, adamreeve)
- [Artifacts] Introduce a `mlflow.artifacts.download_artifacts()` API mirroring the functionality of the `mlflow artifacts download` CLI (5585, dbczumar)
- [Artifacts] Introduce environment variables for controlling GCS artifact upload/download chunk size and timeouts (5438, 5483, mokrueger)

Bug fixes and documentation updates:

- [Tracking/SQLAlchemy] Create an index on `run_uuid` for PostgreSQL to improve query performance (5446, harupy)
- [Tracking] Remove client-side validation of metric, param, tag, and experiment fields (5593, BenWilson2)
- [Projects] Support setting the name of the MLflow Run when executing an MLflow Project (5187, bramrodenburg)
- [Scoring] Use pandas `split` orientation for DataFrame inputs to SageMaker deployment `predict()` API to preserve column ordering (5522, dbczumar)
- [Server-Infra] Fix runs search compatibility bugs with PostgreSQL, MySQL, and MSSQL (5540, harupy)
- [CLI] Fix a bug in the `mlflow-skinny` client that caused `mlflow --version` to fail (5573, BenWilson2)
- [Docs] Update guidance and examples for model deployment to AzureML to recommend using the `mlflow-azureml` package (5491, santiagxf)

Small bug fixes and doc updates (5591, 5629, 5597, 5592, 5562, 5477, BenWilson2; 5554, juntai-zheng; 5570, tahesse; 5605, guelate; 5633, 5632, 5625, 5623, 5615, 5608, 5600, 5603, 5602, 5596, 5587, 5586, 5580, 5577, 5568, 5290, 5556, 5560, 5557, 5548, 5547, 5538, 5513, 5505, 5464, 5495, 5488, 5485, 5468, 5455, 5453, 5454, 5452, 5445, 5431, harupy; 5640, nchittela; 5520, 5422, Ark-kun; 5639, 5604, nishipy; 5543, 5532, 5447, 5435, WeichenXu123; 5502, singankit; 5500, Sohamkayal4103; 5449, 5442, apurva-koti; 5552, vinijaiswal; 5511, adamreeve; 5428, jinzhang21; 5309, sunishsheth2009; 5581, 5559, Kr4is; 5626, 5618, 5529, sisp; 5652, 5624, 5622, 5613, 5509, 5459, 5437, dbczumar; 5616, liangz1)

Features:

- [Tracking] Support uploading, downloading, and listing artifacts through the MLflow server via `mlflow server --serve-artifacts` (5320, BenWilson2, harupy)
- [Tracking] Add the `registered_model_name` argument to `mlflow.autolog()` for automatic model registration during autologging (5395, WeichenXu123)
- [UI] Improve and restructure the Compare Runs page. Additions include "show diff only" toggles and scrollable tables (5306, WeichenXu123)
- [Models] Introduce `mlflow.pmdarima` flavor for pmdarima models (5373, BenWilson2)
- [Models] When loading an MLflow Model, print a warning if a mismatch is detected between the current environment and the Model's dependencies (5368, WeichenXu123)
- [Models] Support computing custom scalar metrics during model evaluation with `mlflow.evaluate()` (5389, MarkYHZhang)
- [Scoring] Add support for deploying and evaluating SageMaker models via the [`MLflow Deployments API`](https://mlflow.org/docs/latest/models.html#deployment-to-custom-targets) (4971, 5396, jamestran201)

Bug fixes and documentation updates:

- [Tracking / UI] Fix artifact listing and download failures that occurred when operating the MLflow server in `--serve-artifacts` mode (5409, dbczumar)
- [Tracking] Support environment-variable-based authentication when making artifact requests to the MLflow server in `--serve-artifacts` mode (5370, TimNooren)
- [Tracking] Fix bugs in hostname and path resolution when making artifacts requests to the MLflow server in `--serve-artifacts` mode (5384, 5385, mert-kirpici)
- [Tracking] Fix an import error that occurred when `mlflow.log_figure()` was used without `matplotlib.figure` imported (5406, WeichenXu123)
- [Tracking] Correctly log XGBoost metrics containing the `` symbol during autologging (5403, maxfriedrich)
- [Tracking] Fix a SQL Server database error that occurred during Runs search (5382, dianacarvalho1)
- [Tracking] When downloading artifacts from HDFS, store them in the user-specified destination directory (5210, DimaClaudiu)
- [Tracking / Model Registry] Improve performance of large artifact and model downloads (5359, mehtayogita)
- [Models] Fix fast.ai PyFunc inference behavior for models with 2D outputs (5411, santiagxf)
- [Models] Record Spark model information to the active run when `mlflow.spark.log_model()` is called (5355, szczeles)
- [Models] Restore onnxruntime execution providers when loading ONNX models with `mlflow.pyfunc.load_model()` (5317, ecm200)
- [Projects] Increase Docker image push timeout when using Projects with Docker (5363, zanitete)
- [Python] Fix a bug that prevented users from enabling DEBUG-level Python log outputs (5362, dbczumar)
- [Docs] Add a developer guide explaining how to build custom plugins for `mlflow.evaluate()` (5333, WeichenXu123)

Small bug fixes and doc updates (5298, wamartin-aml; 5399, 5321, 5313, 5307, 5305, 5268, 5284, harupy; 5329, Ark-kun; 5375, 5346, 5304, dbczumar; 5401, 5366, 5345, BenWilson2; 5326, 5315, WeichenXu123; 5236, singankit; 5302, timvink; 5357, maitre-matt; 5347, 5344, mehtayogita; 5367, apurva-koti; 5348, 5328, 5310, liangz1; 5267, sunishsheth2009)

- [Models] Fix a directory creation failure when loading PySpark ML models (5299, arjundc-db)
- [Model Registry] Revert to using case-insensitive validation logic for stage names in `models:/` URIs (5312, lichenran1234)
- [Projects] Fix a race condition during Project tar file creation (5303, dbczumar)

Features:

- [Models] Introduce an `mlflow.evaluate()` API for evaluating MLflow Models, providing performance and explainability insights. For an overview, see https://mlflow.org/docs/latest/models.html#model-evaluation (5069, 5092, 5256, WeichenXu123)
- [Models] `log_model()` APIs now return information about the logged MLflow Model, including artifact location, flavors, and schema (5230, liangz1)
- [Models] Introduce an `mlflow.models.Model.load_input_example()` Python API for loading MLflow Model input examples (5212, maitre-matt)
- [Models] Add a UUID field to the MLflow Model specification. MLflow Models now have a unique identifier (5149, 5167, WeichenXu123)
- [Models] Support passing SciPy CSC and CSR matrices as MLflow Model input examples (5016, WeichenXu123)
- [Model Registry] Support specifying `latest` in model URI to get the latest version of a model regardless of the stage (5027, lichenran1234)
- [Tracking] Add support for LightGBM scikit-learn models to `mlflow.lightgbm.autolog()` (5130, 5200, 5271 jwyyy)
- [Tracking] Improve S3 artifact download speed by caching boto clients (4695, Samreay)
- [UI] Automatically update metric plots for in-progress runs (5017, cedkoffeto, harupy)

Bug fixes and documentation updates:

- [Models] Fix a bug in MLflow Model schema enforcement where strings were incorrectly cast to Pandas objects (5134, stevenchen-db)
- [Models] Fix a bug where keyword arguments passed to `mlflow.pytorch.load_model()` were not applied for scripted models (5163, schmidt-jake)
- [Model Registry/R] Fix bug in R client `mlflow_create_model_version()` API that caused model `source` to be set incorrectly (5185, bramrodenburg)
- [Projects] Fix parsing behavior for Project URIs containing quotes (5117, dinaldoap)
- [Scoring] Use the correct 400-level error code for malformed MLflow Model Server requests (5003, abatomunkuev)
- [Tracking] Fix a bug where `mlflow.start_run()` modified user-supplied tags dictionary (5191, matheusMoreno)
- [UI] Fix a bug causing redundant scroll bars to be displayed on the Experiment Page (5159, sunishsheth2009)

Small bug fixes and doc updates (5275, 5264, 5244, 5249, 5255, 5248, 5243, 5240, 5239, 5232, 5234, 5235, 5082, 5220, 5219, 5226, 5217, 5194, 5188, 5132, 5182, 5183, 5180, 5177, 5165, 5164, 5162, 5015, 5136, 5065, 5125, 5106, 5127, 5120, harupy; 5045, BenWilson2; 5156, pbezglasny; 5202, jwyyy; 3863, JoshuaAnickat; 5205, abhiramr; 4604, OSobky; 4256, einsmein; 5140, AveshCSingh; 5273, 5186, 5176, WeichenXu123; 5260, 5229, 5206, 5174, 5160, liangz1)

Features:

- [UI] Add a share button to the Experiment page (4936, marijncv)
- [UI] Improve readability of column sorting dropdown on Experiment page (5022, WeichenXu123; 5018, NieuweNils, coder-freestyle)
- [Tracking] Mark all autologging integrations as stable by removing `experimental` decorators (5028, liangz1)
- [Tracking] Add optional `experiment_id` parameter to `mlflow.set_experiment()` (5012, dbczumar)
- [Tracking] Add support for XGBoost scikit-learn models to `mlflow.xgboost.autolog()` (5078, jwyyy)
- [Tracking] Improve statsmodels autologging performance by removing unnecessary metrics (4942, WeichenXu123)
- [Tracking] Update R client to tag nested runs with parent run ID (4197, yitao-li)
- [Models] Support saving and loading all XGBoost model types (4954, jwyyy)
- [Scoring] Support specifying AWS account and role when deploying models to SageMaker (4923, andresionek91)
- [Scoring] Support serving MLflow models with MLServer (4963, adriangonz)

Bug fixes and documentation updates:

- [UI] Fix bug causing Metric Plot page to crash when metric values are too large (4947, ianshan0915)
- [UI] Fix bug causing parallel coordinate curves to vanish (5087, harupy)
- [UI] Remove `Creator` field from Model Version page if user information is absent (5089, jinzhang21)
- [UI] Fix model loading instructions for non-pyfunc models in Artifact Viewer (5006, harupy)
- [Models] Fix a bug that added `mlflow` to `conda.yaml` even if a hashed version was already present (5058, maitre-matt)
- [Docs] Add Python documentation for metric, parameter, and tag key / value length limits (4991, westford14)
- [Examples] Update Python version used in Prophet example to fix installation errors (5101, BenWilson2)
- [Examples] Fix Kubernetes `resources` specification in MLflow Projects + Kubernetes example (4948, jianyuan)

Small bug fixes and doc updates (5119, 5107, 5105, 5103, 5085, 5088, 5051, 5081, 5039, 5073, 5072, 5066, 5064, 5063, 5060, 4718, 5053, 5052, 5041, 5043, 5047, 5036, 5037, 5029, 5031, 5032, 5030, 5007, 5019, 5014, 5008, 4998, 4985, 4984, 4970, 4966, 4980, 4967, 4978, 4979, 4968, 4976, 4975, 4934, 4956, 4938, 4950, 4946, 4939, 4913, 4940, 4935, harupy; 5095, 5070, 5002, 4958, 4945, BenWilson2; 5099, chaosddp; 5005, you-n-g; 5042, 4952, shrinath-suresh; 4962, 4995, WeichenXu123; 5010, lichenran1234; 5000, wentinghu; 5111, alexott; 5102, 5024, 5011, 4959, dbczumar; 5075, 5044, 5026, 4997, 4964, 4989, liangz1; 4999, stevenchen-db)

Features:

- [UI] Add a diff-only toggle to the runs table for filtering out columns with constant values (4862, marijncv)
- [UI] Add a duration column to the runs table (4840, marijncv)
- [UI] Display the default column sorting order in the runs table (4847, marijncv)
- [UI] Add `start_time` and `duration` information to exported runs CSV (4851, marijncv)
- [UI] Add lifecycle stage information to the run page (4848, marijncv)
- [UI] Collapse run page sections by default for space efficiency, limit artifact previews to 50MB (4917, dbczumar)
- [Tracking] Introduce autologging capabilities for PaddlePaddle model training (4751, jinminhao)
- [Tracking] Add an optional tags field to the CreateExperiment API (4788, dbczumar; 4795, apurva-koti)
- [Tracking] Add support for deleting artifacts from SFTP stores via the `mlflow gc` CLI (4670, afaul)
- [Tracking] Support AzureDefaultCredential for authenticating with Azure artifact storage backends (4002, marijncv)
- [Models] Upgrade the fastai model flavor to support fastai V2 (`>=2.4.1`) (4715, jinzhang21)
- [Models] Introduce an `mlflow.prophet` model flavor for Prophet time series models (4773, BenWilson2)
- [Models] Introduce a CLI for publishing MLflow Models to the SageMaker Model Registry (4669, jinnig)
- [Models] Print a warning when inferred model dependencies are not available on PyPI (4891, dbczumar)
- [Models, Projects] Add `MLFLOW_CONDA_CREATE_ENV_CMD` for customizing Conda environment creation (4746, giacomov)

Bug fixes and documentation updates:

- [UI] Fix an issue where column selections made in the runs table were persisted across experiments (4926, sunishsheth2009)
- [UI] Fix an issue where the text `null` was displayed in the runs table column ordering dropdown (4924, harupy)
- [UI] Fix a bug causing the metric plot view to display NaN values upon click (4858, arpitjasa-db)
- [Tracking] Fix a model load failure for paths containing spaces or special characters on UNIX systems (4890, BenWilson2)
- [Tracking] Correct a migration issue that impacted usage of MLflow Tracking with SQL Server (4880, marijncv)
- [Tracking] Spark datasource autologging tags now respect the maximum allowable size for MLflow Tracking (4809, dbczumar)
- [Model Registry] Add previously-missing certificate sources for Model Registry REST API requests (4731, ericgosno91)
- [Model Registry] Throw an exception when users supply invalid Model Registry URIs for Databricks (4877, yunpark93)
- [Scoring] Fix a schema enforcement error that incorrectly cast date-like strings to datetime objects (4902, wentinghu)
- [Docs] Expand the documentation for the MLflow Skinny Client (4113, eedeleon)

Small bug fixes and doc updates (4928, 4919, 4927, 4922, 4914, 4899, 4893, 4894, 4884, 4864, 4823, 4841, 4817, 4796, 4797, 4767, 4768, 4757, harupy; 4863, 4838, marijncv; 4834, ksaur; 4772, louisguitton; 4801, twsl; 4929, 4887, 4856, 4843, 4789, 4780, WeichenXu123; 4769, Ark-kun; 4898, 4756, apurva-koti; 4784, lakshikaparihar; 4855, ianshan0915; 4790, eedeleon; 4931, 4857, 4846, 4777, 4748, dbczumar)