Log metrics, parameters, and files with MLflow

APPLIES TO: Python SDK azure-ai-ml v2 (current)

Azure Machine Learning supports logging and tracking experiments using MLflow Tracking. You can log models, metrics, parameters, and artifacts with MLflow, either locally on your computer or in a cloud environment.

Logs can help you diagnose errors and warnings, or track performance metrics like parameters and model performance. This article explains how to enable logging in the following scenarios:

Prerequisites

• You must have an Azure Machine Learning workspace. If you don't have one, see Create workspace resources.

• You must have the mlflow and azureml-mlflow packages installed. If you don't, use the following command to install them in your development environment:

  pip install mlflow azureml-mlflow
  

  Note

  For asynchronous logging of metrics, you need to have MLflow version 2.8.0+ and azureml-mlflow version 1.55+.

• If you're doing remote tracking (tracking experiments that run outside Azure Machine Learning), configure MLflow to track experiments. For more information, see Configure MLflow for Azure Machine Learning.

• To log metrics, parameters, artifacts, and models in your experiments in Azure Machine Learning using MLflow, just import MLflow into your script:

  import mlflow
  

Configure experiments

MLflow organizes the information in experiments and runs (in Azure Machine Learning, runs are called jobs). There are some differences in how to configure them depending on how you run your code:

import mlflow
# Set the experiment
mlflow.set_experiment("mlflow-experiment")

# Start the run
mlflow_run = mlflow.start_run()
# Log metrics or other information
mlflow.log_metric('mymetric', 1)
# End run 
mlflow.end_run()

import mlflow
mlflow.set_experiment("mlflow-experiment")

# Start the run, log metrics, end the run
with mlflow.start_run() as run:
    # Run started when context manager is entered, and ended when context manager exits
    mlflow.log_metric('mymetric', 1)
    mlflow.log_metric('anothermetric',1)
    pass

with mlflow.start_run(run_name="iris-classifier-random-forest") as run:
    mlflow.log_metric('mymetric', 1)
    mlflow.log_metric('anothermetric',1)

import mlflow

mlflow.autolog()

mlflow.log_metric('mymetric', 1)
mlflow.log_metric('anothermetric',1)

Log parameters

MLflow supports the logging parameters used by your experiments. Parameters can be of any type, and can be logged using the following syntax:

mlflow.log_param("num_epochs", 20)

MLflow also offers a convenient way to log multiple parameters by indicating all of them using a dictionary. Several frameworks can also pass parameters to models using dictionaries and hence this is a convenient way to log them in the experiment.

params = {
    "num_epochs": 20,
    "dropout_rate": .6,
    "objective": "binary_crossentropy"
}

mlflow.log_params(params)

Log metrics

Metrics, as opposite to parameters, are always numeric, and they can be logged either synchronously or asynchronously. When metrics are logged, they are immediately available for consumption upon call return. The following table describes how to log specific numeric types:

Log metrics asynchronously

MLflow also allows logging of metrics in an asynchronous way. Asynchronous metric logging is particularly useful in cases where large training jobs with tens of compute nodes might be running and trying to log metrics concurrently. It's also useful when a small number of nodes is trying to log a high number of metrics.

Asynchronous metric logging allows you to log metrics immediately by avoiding waiting for them to materialize in the backend service. This approach scales to large training routines that log hundreds of thousands of metric values and it's the recommended approach.

MLflow logs metrics synchronously by default, however, you can change this behavior at any time:

import mlflow

mlflow.config.enable_async_logging()

The same property can be set, using an environment variable:

export MLFLOW_ENABLE_ASYNC_LOGGING=True

To log specific metrics asynchronously, use the MLflow logging API as you typically would, but add the extra parameter synchronous=False.

import mlflow

with mlflow.start_run():
    # (...)
    mlflow.log_metric("metric1", 9.42, synchronous=False)
    # (...)

When you use log_metric(synchronous=False), control is automatically returned to the caller once the operation is accepted; however, the value is not available for reading inmediately. Asynchronous logging of metrics does guarantee order and they are persisted with the timestamp of when they were logged.

If you need to wait for a particular value to be persisted in the backend, then you can use the metric operation returned to wait on it, as shown in the following example:

import mlflow

with mlflow.start_run():
    # (...)
    run_operation = mlflow.log_metric("metric1", 9.42, synchronous=False)
    # (...)
    run_operation.wait()
    # (...)

You can asynchronously log one metric at a time or log a batch of metrics, as shown in the following example:

import mlflow
import time
from mlflow.entities import Metric

with mlflow.start_run() as current_run:
    mlflow_client = mlflow.tracking.MlflowClient()

    metrics = {"metric-0": 3.14, "metric-1": 6.28}
    timestamp = int(time.time() * 1000)
    metrics_arr = [Metric(key, value, timestamp, 0) for key, value in metrics.items()]

    run_operation = mlflow_client.log_batch(
        run_id=current_run.info.run_id,
        metrics=metrics_arr,
        synchronous=False,
    )

The wait() operation is also available when logging a batch of metrics:

run_operation.wait()

You don't have to call wait() on your routines if you don't need immediate access to the metric values. Azure Machine Learning automatically waits when the job is about to finish, to see if there is any pending metric to be persisted. By the time a job is completed in Azure Machine Learning, all metrics are guaranteed to be persisted.

Log curves or list of values

Curves (or a list of numeric values) can be logged with MLflow by logging the same metric multiple times. The following example shows how to do it:

list_to_log = [1, 2, 3, 2, 1, 2, 3, 2, 1]
from mlflow.entities import Metric
from mlflow.tracking import MlflowClient
import time

client = MlflowClient()
client.log_batch(mlflow.active_run().info.run_id, 
                 metrics=[Metric(key="sample_list", value=val, timestamp=int(time.time() * 1000), step=0) for val in list_to_log])

Log images

MLflow supports two ways of logging images. Both ways persist the given image as an artifact inside of the run.

Log files

In general, files in MLflow are called artifacts. You can log artifacts in multiple ways in Mlflow:

Log models

MLflow introduces the concept of models as a way to package all the artifacts required for a given model to function. Models in MLflow are always a folder with an arbitrary number of files, depending on the framework used to generate the model. Logging models has the advantage of tracking all the elements of the model as a single entity that can be registered and then deployed. On top of that, MLflow models enjoy the benefit of no-code deployment and can be used with the Responsible AI dashboard in studio. For more information, see From artifacts to models in MLflow.

To save the model from a training run, use the log_model() API for the framework you're working with. For example, mlflow.sklearn.log_model(). For more information, see Logging MLflow models. For migrating existing models to MLflow, see Convert custom models to MLflow.

Automatic logging

With Azure Machine Learning and MLflow, users can log metrics, model parameters, and model artifacts automatically when training a model. Each framework decides what to track automatically for you. A variety of popular machine learning libraries are supported. Learn more about Automatic logging with MLflow.

To enable automatic logging, insert the following code before your training code:

mlflow.autolog()

View information about jobs or runs with MLflow

You can view the logged information using MLflow through the MLflow.entities.Run object:

import mlflow

run = mlflow.get_run(run_id="<RUN_ID>")

You can view the metrics, parameters, and tags for the run in the data field of the run object.

metrics = run.data.metrics
params = run.data.params
tags = run.data.tags

MLflow can query any artifact logged by a run. Artifacts can't be accessed using the run object itself and the MLflow client should be used instead:

client = mlflow.tracking.MlflowClient()
client.list_artifacts("<RUN_ID>")

This method lists all the artifacts logged in the run, but they remain stored in the artifacts store (Azure Machine Learning storage). To download any of them, use the method download_artifact:

file_path = client.download_artifacts("<RUN_ID>", path="feature_importance_weight.png")

For more information, please refer to Getting metrics, parameters, artifacts and models.

View information about jobs or runs in the studio

You can browse completed job records, including logged metrics, in the Azure Machine Learning studio.

Navigate to the Jobs tab. To view all your jobs in your Workspace across Experiments, select the All jobs tab. You can drill down on jobs for specific experiments by applying the Experiment filter in the top menu bar. Select the job of interest to enter the details view, and then select the Metrics tab.

Select the logged metrics to render charts on the right side. You can customize the charts by applying smoothing, changing the color, or plotting multiple metrics on a single graph. You can also resize and rearrange the layout as you wish. After you create your desired view, you can save it for future use and share it with your teammates using a direct link.

View and download diagnostic logs

Log files are an essential resource for debugging the Azure Machine Learning workloads. After submitting a training job, drill down to a specific run to view its logs and outputs:

1. Navigate to the Jobs tab.
2. Select the runID for a specific run.
3. Select Outputs and logs at the top of the page.
4. Select Download all to download all your logs into a zip folder.
5. You can also download individual log files by choosing the log file and selecting Download

user_logs folder

This folder contains information about the user generated logs. This folder is open by default, and the std_log.txt log is selected. The std_log.txt is where your code's logs (for example, print statements) show up. This file contains stdout log and stderr logs from your control script and training script, one per process. In most cases, you monitor the logs here.

system_logs folder

This folder contains the logs generated by Azure Machine Learning and it's closed by default. The logs generated by the system are grouped into different folders, based on the stage of the job in the runtime.

Other folders

For jobs training on multi-compute clusters, logs are present for each IP node. The structure for each node is the same as single node jobs. There's one more logs folder for overall execution, stderr, and stdout logs.

Azure Machine Learning logs information from various sources during training, such as AutoML or the Docker container that runs the training job. Many of these logs aren't documented. If you encounter problems and contact Azure support, they might be able to use these logs during troubleshooting.

Related content

• Train ML models with MLflow and Azure Machine Learning
• Migrate from SDK v1 logging to MLflow tracking