Mlflow

Bug fixes:

- [Security] Fix critical LFI attack vulnerability by disabling the ability to provide relative paths in registered model sources (8281, BenWilson2)
- **If you are using `mlflow server` or `mlflow ui`, we recommend upgrading to MLflow 2.3.1 as soon as possible.** For more details, see https://github.com/mlflow/mlflow/security/advisories/GHSA-xg73-94fp-g449.
- [Tracking] Fix an issue causing file and model uploads to hang on Databricks (8348, harupy)
- [Tracking / Model Registry] Fix an issue causing file and model downloads to hang on Databricks (8350, dbczumar)
- [Scoring] Fix regression in schema enforcement for model serving when using the `inputs` format for inference (8326, BenWilson2)
- [Model Registry] Fix regression in model naming parsing where special characters were not accepted in model names (8322, arpitjasa-db)
- [Recipes] Fix card rendering with the pandas profiler to handle columns containing all null values (8263, sunishsheth2009)

Documentation updates:

- [Docs] Add an H2O pyfunc usage example to the models documentation (8292, ericvincent18)
- [Examples] Add a TensorFlow Core 2.x API usage example (8235, dheerajnbhat)

Small bug fixes and documentation updates:

8324, 8325, smurching; 8313, dipanjank; 8323, liangz1; 8331, 8328, 8319, 8316, 8308, 8293, 8289, 8283, 8284, 8285, 8282, 8241, 8270, 8272, 8271, 8268, harupy; 8312, 8294, 8295, 8279, 8267, BenWilson2; 8290, jinzhang21; 8257, WeichenXu123; 8307, arpitjasa-db

Features:

- [Models] Introduce a new `transformers` named flavor (8236, 8181, 8086, BenWilson2)
- [Models] Introduce a new `openai` named flavor (8191, 8155, harupy)
- [Models] Introduce a new `langchain` named flavor (8251, 8197, liangz1, sunishsheth2009)
- [Models] Add support for `Pytorch` and `Lightning` 2.0 (8072, shrinath-suresh)
- [Tracking] Add support for logging LLM input, output, and prompt artifacts (8234, 8204, sunishsheth2009)
- [Tracking] Add support for HTTP Basic Auth in the MLflow tracking server (8130, gabrielfu)
- [Tracking] Add `search_model_versions` to the fluent API (8223, mariusschlegel)
- [Artifacts] Add support for parallelized artifact downloads (8116, apurva-koti)
- [Artifacts] Add support for parallelized artifact uploads for AWS (8003, harupy)
- [Artifacts] Add content type headers to artifact upload requests for the `HttpArtifactRepository` (8048, WillEngler)
- [Model Registry] Add alias support for logged models within Model Registry (8164, 8094, 8055 arpitjasa-db)
- [UI] Add support for custom domain git providers (7933, gusghrlrl101)
- [Scoring] Add plugin support for customization of MLflow serving endpoints (7757, jmahlik)
- [Scoring] Add support to MLflow serving that allows configuration of multiple inference workers (8035, M4nouel)
- [Sagemaker] Add support for asynchronous inference configuration on Sagemaker (8009, thomasbell1985)
- [Build] Remove `shap` as a core dependency of MLflow (8199, jmahlik)

Bug fixes:

- [Models] Fix a bug with `tensorflow` autologging for models with multiple inputs (8097, jaume-ferrarons)
- [Recipes] Fix a bug with `Pandas` 2.0 updates for profiler rendering of datetime types (7925, sunishsheth2009)
- [Tracking] Prevent exceptions from being raised if a parameter is logged with an existing key whose value is identical to the logged parameter (8038, AdamStelmaszczyk)
- [Tracking] Fix an issue with deleting experiments in the FileStore backend (8178, mariusschlegel)
- [Tracking] Fix a UI bug where the "Source Run" field in the Model Version page points to an incorrect set of artifacts (8156, WeichenXu123)
- [Tracking] Fix a bug wherein renaming a run reverts its current lifecycle status to `UNFINISHED` (8154, WeichenXu123)
- [Tracking] Fix a bug where a file URI could be used as a model version source (8126, harupy)
- [Projects] Fix an issue with MLflow projects that have submodules contained within a project (8050, kota-iizuka)
- [Examples] Fix `lightning` hyperparameter tuning examples (8039, BenWilson2)
- [Server-infra] Fix bug with Cache-Control headers for static server files (8016, jmahlik)

Documentation updates:

- [Examples] Add a new and thorough example for the creation of custom model flavors (7867, benjaminbluhm)

Small bug fixes and documentation updates:

8262, 8252, 8250, 8228, 8221, 8203, 8134, 8040, 7994, 7934, BenWilson2; 8258, 8255, 8253, 8248, 8247, 8245, 8243, 8246, 8244, 8242, 8240, 8229, 8198, 8192, 8112, 8165, 8158, 8152, 8148, 8144, 8143, 8120, 8107, 8105, 8102, 8088, 8089, 8096, 8075, 8073, 8076, 8063, 8064, 8033, 8024, 8023, 8021, 8015, 8005, 7982, 8002, 7987, 7981, 7968, 7931, 7930, 7929, 7917, 7918, 7916, 7914, 7913, harupy; 7955, arjundc-db; 8219, 8110, 8093, 8087, 8091, 8092, 8029, 8028, 8031, jerrylian-db; 8187, apurva-koti; 8210, 8001, 8000, arpitjasa-db; 8161, 8127, 8095, 8090, 8068, 8043, 7940, 7924, 7923, dbczumar; 8147, morelen17; 8106, WeichenXu123; 8117, eltociear; 8100, laerciop; 8080, elado; 8070, grofte; 8066, yukimori; 8027, 7998, liangz1; 7999, martlaf; 7964, viditjain99; 7928, alekseyolg; 7909, 7901, 7844, smurching; 7971, n30111; 8012, mingyu89; 8137, lobrien; 7992, robmarkcole; 8263, sunishsheth2009

- [Model Registry] Allow `source` to be a local path within a run's artifact directory if a `run_id` is specified (7993, harupy)
- [Model Registry] Fix a bug where a windows UNC path is considered a local path (7988, WeichenXu123)
- [Model Registry] Disallow `name` to be a file path in `FileStore.get_registered_model` (7965, harupy)

- [Model Registry] Fix a bug that caused too many results to be requested by default when calling `MlflowClient.search_model_versions()` (7935, dbczumar)
- [Model Registry] Patch for GHSA-xg73-94fp-g449 (7908, harupy)
- [Model Registry] Patch for GHSA-wp72-7hj9-5265 (7965, harupy)

Features:

- [Recipes] Add support for score calibration to the classification recipe (7744, sunishsheth2009)
- [Recipes] Add automatic label encoding to the classification recipe (7711, sunishsheth2009)
- [Recipes] Support custom data splitting logic in the classification and regression recipes (7815, 7588, sunishsheth2009)
- [Recipes] Introduce customizable MLflow Run name prefixes to the classification and regression recipes (7746, kamalesh0406; 7763, sunishsheth2009)
- [UI] Add a new Chart View to the MLflow Experiment Page for model performance insights (7864, hubertzub-db, apurva-koti, prithvikannan, ridhimag11, sunishseth2009, dbczumar)
- [UI] Modernize and improve parallel coordinates chart for model tuning (7864, hubertzub-db, apurva-koti, prithvikannan, ridhimag11, sunishseth2009, dbczumar)
- [UI] Add typeahead suggestions to the MLflow Experiment Page search bar (7864, hubertzub-db, apurva-koti, prithvikannan, ridhimag11, sunishseth2009, dbczumar)
- [UI] Improve performance of Experiments Sidebar for large numbers of experiments (7804, jmahlik)
- [Tracking] Introduce autologging support for native PyTorch models (7627, temporaer)
- [Tracking] Allow specifying `model_format` when autologging XGBoost models (7781, guyrosin)
- [Tracking] Add `MLFLOW_ARTIFACT_UPLOAD_DOWNLOAD_TIMEOUT` environment variable to configure artifact operation timeouts (7783, wamartin-aml)
- [Artifacts] Include `Content-Type` response headers for artifacts downloaded from `mlflow server` (7827, bali0019)
- [Model Registry] Introduce the `searchModelVersions()` API to the Java client (7880, gabrielfu)
- [Model Registry] Introduce `max_results`, `order_by` and `page_token` arguments to `MlflowClient.search_model_versions()` (7623, serena-ruan)
- [Models] Support logging large ONNX models by using external data (7808, dogeplusplus)
- [Models] Add support for logging Diviner models fit in Spark (7800, BenWilson2)
- [Models] Introduce `MLFLOW_DEFAULT_PREDICTION_DEVICE` environment variable to set the device for pyfunc model inference (7922, ankit-db)
- [Scoring] Publish official Docker images for the MLflow Model scoring server at github.com/mlflow/mlflow/pkgs (7759, dbczumar)

Bug fixes:

- [Recipes] Fix dataset format validation in the ingest step for custom dataset sources (7638, sunishsheth2009)
- [Recipes] Fix bug in identification of worst performing examples during training (7658, sunishsheth2009)
- [Recipes] Ensure consistent rendering of the recipe graph when `inspect()` is called (7852, sunishsheth2009)
- [Recipes] Correctly respect `positive_class` configuration in the transform step (7626, sunishsheth2009)
- [Recipes] Make logged metric names consistent with `mlflow.evaluate()` (7613, sunishsheth2009)
- [Recipes] Add `run_id` and `artifact_path` keys to logged MLmodel files (7651, sunishsheth2009)
- [UI] Fix bugs in UI validation of experiment names, model names, and tag keys (7818, subramaniam02)
- [Tracking] Resolve artifact locations to absolute paths when creating experiments (7670, bali0019)
- [Tracking] Exclude Delta checkpoints from Spark datasource autologging (7902, harupy)
- [Tracking] Consistently return an empty list from GetMetricHistory when a metric does not exist (7589, bali0019; 7659, harupy)
- [Artifacts] Fix support for artifact operations on Windows paths in UNC format (7750, bali0019)
- [Artifacts] Fix bug in HDFS artifact listing (7581, pwnywiz)
- [Model Registry] Disallow creation of model versions with local filesystem sources in `mlflow server` (7908, harupy)
- [Model Registry] Fix handling of deleted model versions in FileStore (7716, harupy)
- [Model Registry] Correctly initialize Model Registry SQL tables independently of MLflow Tracking (7704, harupy)
- [Models] Correctly move PyTorch model outputs from GPUs to CPUs during inference with pyfunc (7885, ankit-db)
- [Build] Fix compatiblility issues with Python installations compiled using `PYTHONOPTIMIZE=2` (7791, dbczumar)
- [Build] Fix compatibility issues with the upcoming pandas 2.0 release (7899, harupy; 7910, dbczumar)

Documentation updates:

- [Docs] Add an example of saving and loading Spark MLlib models with MLflow (7706, dipanjank)
- [Docs] Add usage examples for `mlflow.lightgbm` APIs (7565, canerturkseven)
- [Docs] Add an example of custom model flavor creation with `sktime` (7624, benjaminbluhm)
- [Docs] Clarify `precision_recall_auc` metric calculation in `mlflow.evaluate()` (7701, BenWilson2)
- [Docs] Remove outdated example links (7587, asloan7)

Small bug fixes and documentation updates:

7866, 7751, 7724, 7699, 7697, 7666, alekseyolg; 7896, 7861, 7858, 7862, 7872, 7859, 7863, 7767, 7766, 7765, 7741, smurching; 7895, 7877, viditjain99; 7898, midhun1998; 7891, 7892, 7886, 7882, 7883, 7875, 7874, 7871, 7868, 7854, 7847, 7845, 7838, 7830, 7837, 7836, 7834, 7831, 7828, 7825, 7826, 7824, 7823, 7778, 7780, 7776, 7775, 7773, 7772, 7769, 7756, 7768, 7764, 7685, 7726, 7722, 7720, 7423, 7712, 7710, 7713, 7688, 7663, 7674, 7673, 7672, 7662, 7653, 7646, 7615, 7614, 7586, 7601, 7598, 7602, 7599, 7577, 7585, 7583, 7584, harupy; 7865, 7803, 7753, 7719, dipanjank; 7796, serena-ruan; 7849, turbotimon; 7822, 7600, WeichenXu123; 7811, guyrosin; 7812, 7788, 7787, 7748, 7730, 7616, 7593, dbczumar; 7793, Joel-hanson; 7792, 7694, 7643, BenWilson2; 7771, 7657, 7644, nsenno-dbr; 7738, wkrt7; 7740, Ark-kun; 7739, 7733, bali0019; 7723, andrehp; 7691, 7582, agoyot; 7721, Eseeldur; 7709, srowen; 7693, ry3s; 7649, funkypenguin; 7665, benjaminbluhm; 7668, eltociear; 7550, danielhstahl; 7920, arjundc-db

Features:

- [Recipes] Introduce support for multi-class classification (7458, mshtelma)
- [Recipes] Extend the pyfunc representation of classification models to output scores in addition to labels (7474, sunishsheth2009)
- [UI] Add user ID and lifecycle stage quick search links to the Runs page (7462, jaeday)
- [Tracking] Paginate the GetMetricHistory API (7523, 7415, BenWilson2)
- [Tracking] Add Runs search aliases for Run name and start time that correspond to UI column names (7492, apurva-koti)
- [Tracking] Add a `/version` endpoint to `mlflow server` for querying the server's MLflow version (7273, joncarter1)
- [Model Registry] Add FileStore support for the Model Registry (6605, serena-ruan)
- [Model Registry] Introduce an `mlflow.search_registered_models()` fluent API (7428, TSienki)
- [Model Registry / Java] Add a `getRegisteredModel()` method to the Java client (6602) (7511, drod331)
- [Model Registry / R] Add an `mlflow_set_model_version_tag()` method to the R client (7401, leeweijie)
- [Models] Introduce a `metadata` field to the MLmodel specification and `log_model()` methods (7237, jdonzallaz)
- [Models] Extend `Model.load()` to support loading MLmodel specifications from remote locations (7517, dbczumar)
- [Models] Pin the major version of MLflow in Models' `requirements.txt` and `conda.yaml` files (7364, BenWilson2)
- [Scoring] Extend `mlflow.pyfunc.spark_udf()` to support StructType results (7527, WeichenXu123)
- [Scoring] Extend TensorFlow and Keras Models to support multi-dimensional inputs with `mlflow.pyfunc.spark_udf()`(7531, 7291, WeichenXu123)
- [Scoring] Support specifying deployment environment variables and tags when deploying models to SageMaker (7433, jhallard)

Bug fixes:

- [Recipes] Fix a bug that prevented use of custom `early_stop` functions during model tuning (7538, sunishsheth2009)
- [Recipes] Fix a bug in the logic used to create a Spark session during data ingestion (7307, WeichenXu123)
- [Tracking] Make the metric names produced by `mlflow.autolog()` consistent with `mlflow.evaluate()` (7418, wenfeiy-db)
- [Tracking] Fix an autologging bug that caused nested, redundant information to be logged for XGBoost and LightGBM models (7404, WeichenXu123)
- [Tracking] Correctly classify SQLAlchemy OperationalErrors as retryable HTTP errors (7240, barrywhart)
- [Artifacts] Correctly handle special characters in credentials when using FTP artifact storage (7479, HCTsai)
- [Models] Address an issue that prevented MLeap models from being saved on Windows (6966, dbczumar)
- [Scoring] Fix a permissions issue encountered when using NFS during model scoring with `mlflow.pyfunc.spark_udf()` (7427, WeichenXu123)

Documentation updates:

- [Docs] Add more examples to the Runs search documentation page (7487, apurva-koti)
- [Docs] Add documentation for Model flavors developed by the community (7425, mmerce)
- [Docs] Add an example for logging and scoring ONNX Models (7398, Rusteam)
- [Docs] Fix a typo in the model scoring REST API example for inputs with the `dataframe_split` format (7540, zhouyangyu)
- [Docs] Fix a typo in the model scoring REST API example for inputs with the `dataframe_records` format (7361, dbczumar)

Small bug fixes and documentation updates:

7571, 7543, 7529, 7435, 7399, WeichenXu123; 7568, xiaoye-hua; 7549, 7557, 7509, 7498, 7499, 7485, 7486, 7484, 7391, 7388, 7390, 7381, 7366, 7348, 7346, 7334, 7340, 7323, BenWilson2; 7561, 7562, 7560, 7553, 7546, 7539, 7544, 7542, 7541, 7533, 7507, 7470, 7469, 7467, 7466, 7464, 7453, 7449, 7450, 7440, 7430, 7436, 7429, 7426, 7410, 7406, 7409, 7407, 7405, 7396, 7393, 7395, 7384, 7376, 7379, 7375, 7354, 7353, 7351, 7352, 7350, 7345, 6493, 7343, 7344, harupy; 7494, dependabot[bot]; 7526, tobycheese; 7489, liangz1; 7534, Jingnan-Jia; 7496, danielhstahl; 7504, 7503, 7459, 7454, 7447, tsugumi-sys; 7461, wkrt7; 7451, 7414, 7372, 7289, sunishsheth2009; 7441, ikrizanic; 7432, Pochingto; 7386, jhallard; 7370, 7373, 7371, 7336, 7341, 7342, dbczumar; 7335, prithvikannan