MLflow and Azure Machine Learning

APPLIES TO: Azure CLI ml extension v2 (current) Python SDK azure-ai-ml v2 (current)

This article describes the capabilities of MLflow, an open-source framework designed to manage the complete machine learning lifecycle. MLflow uses a consistent set of tools to train and serve models on different platforms. You can use MLflow whether your experiments are running locally or on a remote compute target, virtual machine, or Azure Machine Learning compute instance.

Azure Machine Learning workspaces are MLflow-compatible, which means that you can use an Azure Machine Learning workspace the same way you use an MLflow server. This compatibility has the following advantages:

• Azure Machine Learning doesn't host MLflow server instances, but can use the MLflow APIs directly.
• You can use an Azure Machine Learning workspace as your tracking server for any MLflow code, whether or not it runs in Azure Machine Learning. You only need to configure MLflow to point to the workspace where the tracking should occur.
• You can run any training routine that uses MLflow in Azure Machine Learning without making any changes.

What is tracking

When you work with jobs, Azure Machine Learning automatically tracks some information about experiments, such as code, environment, and input and output data. However, models, parameters, and metrics are specific to the scenario, so model builders must configure their tracking.

The saved tracking metadata varies by experiment, and can include:

• Code
• Environment details such as OS version and Python packages
• Input data
• Parameter configurations
• Models
• Evaluation metrics
• Evaluation visualizations such as confusion matrices and importance plots
• Evaluation results, including some evaluation predictions

Benefits of tracking experiments

Whether you train models with jobs in Azure Machine Learning or interactively in notebooks, experiment tracking helps you:

• Organize all of your machine learning experiments in a single place. You can then search and filter experiments and drill down to see details about previous experiments.
• Easily compare experiments, analyze results, and debug model training.
• Reproduce or rerun experiments to validate results.
• Improve collaboration, because you can see what other teammates are doing, share experiment results, and access experiment data programmatically.

Tracking with MLflow

Azure Machine Learning workspaces are MLflow-compatible. This compatibility means you can use MLflow to track runs, metrics, parameters, and artifacts in workspaces without needing to change your training routines or inject any cloud-specific syntax. To learn how to use MLflow for tracking experiments and runs in Azure Machine Learning workspaces, see Track experiments and models with MLflow.

Azure Machine Learning uses MLflow tracking to log metrics and store artifacts for your experiments. When you're connected to Azure Machine Learning, all MLflow tracking materializes in the workspace you're working in.

To learn how to enable logging to monitor real-time run metrics with MLflow, see Log metrics, parameters, and files with MLflow. You can also query and compare experiments and runs with MLflow.

MLflow in Azure Machine Learning provides a way to centralize tracking. You can connect MLflow to Azure Machine Learning workspaces even when you're working locally or in a different cloud. The Azure Machine Learning workspace provides a centralized, secure, and scalable location to store training metrics and models.

MLflow in Azure Machine Learning can:

• Track machine learning experiments and models running locally or in the cloud.
• Track Azure Databricks machine learning experiments.

Tracking with MLflow in R

MLflow support in R has the following limitations:

• MLflow tracking is limited to tracking experiment metrics, parameters, and models on Azure Machine Learning jobs.
• Interactive training on RStudio, Posit (formerly RStudio Workbench), or Jupyter notebooks with R kernels isn't supported.
• Model management and registration aren't supported. Use the Azure Machine Learning CLI or Azure Machine Learning studio for model registration and management.

For examples of using the MLflow tracking client with R models in Azure Machine Learning, see Train R models using the Azure Machine Learning CLI (v2).

Tracking with MLflow in Java

MLflow support in Java has the following limitations:

• MLflow tracking is limited to tracking experiment metrics and parameters on Azure Machine Learning jobs.
• Artifacts and models can't be tracked. Instead, use the mlflow.save_model method with the outputs folder in jobs to save models or artifacts that you want to capture.

For a Java example that uses the MLflow tracking client with the Azure Machine Learning tracking server, see azuremlflow-java.

Example notebooks for MLflow tracking

• Training and tracking an XGBoost classifier with MLflow demonstrates how to use MLflow to track experiments, log models, and combine multiple flavors into pipelines.
• Training and tracking an XGBoost classifier with MLflow using service principal authentication demonstrates how to use MLflow to track experiments from a compute that's running outside Azure Machine Learning. The example shows how to authenticate against Azure Machine Learning services by using a service principal.
• Hyperparameter optimization using HyperOpt and nested runs in MLflow demonstrates how to use child runs to do hyperparameter optimization for models by using the popular HyperOpt library. The example shows how to transfer metrics, parameters, and artifacts from child runs to parent runs.
• Logging models with MLflow demonstrates how to use the concept of models instead of artifacts with MLflow. The example also shows how to construct custom models.
• Manage runs and experiments with MLflow demonstrates how to use MLflow to query experiments, runs, metrics, parameters, and artifacts from Azure Machine Learning.

Model registration with MLflow

Azure Machine Learning supports MLflow for model management. This support is a convenient way for users who are familiar with the MLflow client to manage the entire model lifecycle. For more information about how to manage models by using the MLflow API in Azure Machine Learning, see Manage model registries in Azure Machine Learning with MLflow.

Example notebook for MLflow model registration

Model management with MLflow demonstrates how to manage models in registries.

Model deployment with MLflow

You can deploy MLflow models to Azure Machine Learning to take advantage of an improved experience. Azure Machine Learning supports deployment of MLflow models to both real-time and batch endpoints without having to specify an environment or a scoring script.

The MLflow SDK, Azure Machine Learning CLI, Azure Machine Learning SDK for Python, and Azure Machine Learning studio all support MLflow model deployment. For more information about deploying MLflow models to Azure Machine Learning for both real-time and batch inferencing, see Guidelines for deploying MLflow models.

Example notebooks for MLflow model deployment

• Deploy MLflow to online endpoints demonstrates how to deploy MLflow models to online endpoints using the MLflow SDK.
• Progressive rollout of MLflow deployments demonstrates how to deploy MLflow models to online endpoints using the MLflow SDK with progressive model rollout. The example also shows deployment of multiple versions of a model to the same endpoint.
• Deploy MLflow models to legacy web services demonstrates how to deploy MLflow models to legacy web services (Azure Container Instances or Azure Kubernetes Service v1) using the MLflow SDK.
• Train models in Azure Databricks and deploy them on Azure Machine Learning demonstrates how to train models in Azure Databricks and deploy them in Azure Machine Learning. The example also covers tracking experiments with the MLflow instance in Azure Databricks.

Training with MLflow Projects (preview)

You can submit training jobs to Azure Machine Learning by using MLflow Projects. You can submit jobs locally with Azure Machine Learning tracking or migrate your jobs to the cloud via Azure Machine Learning compute.

To learn how to submit training jobs that use MLflow Projects to Azure Machine Learning workspaces for tracking, see Train with MLflow Projects in Azure Machine Learning (preview).

Example notebooks for MLflow Projects

• Train with MLflow Projects on local compute.
• Train with MLflow Projects on Azure Machine Learning compute.

MLflow vs Azure Machine Learning client tools capabilities

The following table shows machine learning lifecycle operations that are possible with the MLflow SDK and the Azure Machine Learning client tools.

Feature	MLflow SDK	Azure Machine Learning CLI/SDK v2	Azure Machine Learning studio
Track and log metrics, parameters, and models	✓
Retrieve metrics, parameters, and models	✓	Only artifacts and models can be downloaded.	✓
Submit training jobs	Possible by using MLflow Projects (preview).	✓	✓
Submit training jobs with Azure Machine Learning data assets		✓	✓
Submit training jobs with machine learning pipelines		✓	✓
Manage experiments and runs	✓	✓	✓
Manage MLflow models	Some operations might not be supported.1	✓	✓
Manage non-MLflow models		✓	✓
Deploy MLflow models to Azure Machine Learning (online and batch)	Deploying MLflow models for batch inference isn't currently supported.2	✓	✓
Deploy non-MLflow models to Azure Machine Learning		✓	✓

¹ For more information, see Manage model registries in Azure Machine Learning with MLflow.

² For an alternative, see Deploy and run MLflow models in Spark jobs.

Related content

• Configure MLflow for Azure Machine Learning
• Track ML experiments and models with MLflow
• Guidelines for deploying MLflow models