Author scoring scripts for batch deployments

APPLIES TO: Azure CLI ml extension v2 (current) Python SDK azure-ai-ml v2 (current)

Batch endpoints allow you to deploy models that perform long-running inference at scale. When deploying models, you must create and specify a scoring script (also known as a batch driver script) to indicate how to use it over the input data to create predictions. In this article, you'll learn how to use scoring scripts in model deployments for different scenarios. You'll also learn about best practices for batch endpoints.

Understanding the scoring script

The scoring script is a Python file (.py) that specifies how to run the model, and read the input data that the batch deployment executor submits. Each model deployment provides the scoring script (along with all other required dependencies) at creation time. The scoring script usually looks like this:

code_configuration:
  code: code
  scoring_script: batch_driver.py

deployment = ModelBatchDeployment(
    ...
    code_configuration=CodeConfiguration(
        code="src",
        scoring_script="batch_driver.py"
    ),
    ...
)

The scoring script must contain two methods:

The init method

Use the init() method for any costly or common preparation. For example, use it to load the model into memory. The start of the entire batch job calls this function one time. The files of your model are available in a path determined by the environment variable AZUREML_MODEL_DIR. Depending on how your model was registered, its files might be contained in a folder. In the next example, the model has several files in a folder named model. For more information, visit how you can determine the folder that your model uses.

def init():
    global model

    # AZUREML_MODEL_DIR is an environment variable created during deployment
    # The path "model" is the name of the registered model's folder
    model_path = os.path.join(os.environ["AZUREML_MODEL_DIR"], "model")

    # load the model
    model = load_model(model_path)

In this example, we place the model in global variable model. To make available the assets required to perform inference on your scoring function, use global variables.

The run method

Use the run(mini_batch: List[str]) -> Union[List[Any], pandas.DataFrame] method to handle the scoring of each mini-batch that the batch deployment generates. This method is called once for each mini_batch generated for your input data. Batch deployments read data in batches according to how the deployment configuration.

import pandas as pd
from typing import List, Any, Union

def run(mini_batch: List[str]) -> Union[List[Any], pd.DataFrame]:
    results = []

    for file in mini_batch:
        (...)

    return pd.DataFrame(results)

The method receives a list of file paths as a parameter (mini_batch). You can use this list to iterate over and individually process each file, or to read the entire batch and process it all at once. The best option depends on your compute memory and the throughput you need to achieve. For an example that describes how to read entire batches of data at once, visit High throughput deployments.

The run() method should return a Pandas DataFrame or an array/list. Each returned output element indicates one successful run of an input element in the input mini_batch. For file or folder data assets, each returned row/element represents a single file processed. For a tabular data asset, each returned row/element represents a row in a processed file.

The resulting DataFrame or array is appended to the indicated output file. There's no requirement about the cardinality of the results. One file can generate 1 or many rows/elements in the output. All elements in the result DataFrame or array are written to the output file as-is (considering the output_action isn't summary_only).

Python packages for scoring

You must indicate any library that your scoring script requires to run in the environment where your batch deployment runs. For scoring scripts, environments are indicated per deployment. Usually, you indicate your requirements using a conda.yml dependencies file, which might look like this:

mnist/environment/conda.yaml

name: mnist-env
channels:
  - conda-forge
dependencies:
  - python=3.8.5
  - pip<22.0
  - pip:
    - torch==1.13.0
    - torchvision==0.14.0
    - pytorch-lightning
    - pandas
    - azureml-core
    - azureml-dataset-runtime[fuse]

Writing predictions in a different way

By default, the batch deployment writes the model's predictions in a single file as indicated in the deployment. However, in some cases, you must write the predictions in multiple files. For instance, for partitioned input data, you would likely want to generate partitioned output as well. In those cases, you can Customize outputs in batch deployments to indicate:

Visit Customize outputs in batch deployments for more information about how to achieve it.

Source control of scoring scripts

It's highly advisable to place scoring scripts under source control.

Best practices for writing scoring scripts

When writing scoring scripts that handle large amounts of data, you must take into account several factors, including

• The size of each file
• The amount of data on each file
• The amount of memory required to read each file
• The amount of memory required to read an entire batch of files
• The memory footprint of the model
• The model memory footprint, when running over the input data
• The available memory in your compute

Batch deployments distribute work at the file level. This means that a folder that contains 100 files, in mini-batches of 10 files, generates 10 batches of 10 files each (regardless of the size of the files involved). For files too large to process in large mini-batches, we suggest that you split the files into smaller files, to achieve a higher level of parallelism, or that you decrease the number of files per mini-batch. At this time, batch deployment can't account for skews in the file's size distribution.

Relationship between the degree of parallelism and the scoring script

Your deployment configuration controls both the size of each mini-batch and the number of workers on each node. This becomes important when you decide whether or not to read the entire mini-batch to perform inference, to run inference file by file, or run the inference row by row (for tabular). Visit Running inference at the mini-batch, file or the row level for more information.

When running multiple workers on the same instance, you should account for the fact that memory is shared across all the workers. An increase in the number of workers per node should generally accompany a decrease in the mini-batch size, or by a change in the scoring strategy if data size and compute SKU remains the same.

Running inference at the mini-batch, file or the row level

Batch endpoints call the run() function in a scoring script once per mini-batch. However, you can decide if you want to run the inference over the entire batch, over one file at a time, or over one row at a time for tabular data.

Mini-batch level

You'll typically want to run inference over the batch all at once, to achieve high throughput in your batch scoring process. This happens if you run inference over a GPU, where you want to achieve saturation of the inference device. You might also rely on a data loader that can handle the batching itself if data doesn't fit on memory, like TensorFlow or PyTorch data loaders. In these cases, you might want to run inference on the entire batch.

Visit High throughput deployments to learn how to achieve this. This example processes an entire batch of files at a time.

File level

One of the easiest ways to perform inference is iteration over all the files in the mini-batch, and then run the model over it. In some cases, for example image processing, this might be a good idea. For tabular data, you might need to make a good estimation about the number of rows in each file. This estimate can show whether or not your model can handle the memory requirements to both load the entire data into memory and to perform inference over it. Some models (especially those models based on recurrent neural networks) unfold and present a memory footprint with a potentially nonlinear row count. For a model with high memory expense, consider running inference at the row level.

Visit Image processing with batch deployments to learn how to do this. That example processes a file at a time.

Row level (tabular)

For models that present challenges with their input sizes, you might want to run inference at the row level. Your batch deployment still provides your scoring script with a mini-batch of files. However, you'll read one file, one row at a time. This might seem inefficient, but for some deep learning models it might be the only way to perform inference without scaling up your hardware resources.

Visit Text processing with batch deployments to learn how to do this. That example processes a row at a time.

Using models that are folders

The AZUREML_MODEL_DIR environment variable contains the path to the selected model location, and the init() function typically uses it to load the model into memory. However, some models might contain their files in a folder, and you might need to account for that when loading them. You can identify the folder structure of your model as shown here:

1. Go to Azure Machine Learning portal.

2. Go to the section Models.

3. Select the model you want to deploy, and select the Artifacts tab.

4. Note the displayed folder. This folder was indicated when the model was registered.

   

Use this path to load the model:

def init():
    global model

    # AZUREML_MODEL_DIR is an environment variable created during deployment
    # The path "model" is the name of the registered model's folder
    model_path = os.path.join(os.environ["AZUREML_MODEL_DIR"], "model")

    model = load_model(model_path)

Next steps

• Troubleshooting batch endpoints
• Use MLflow models in batch deployments
• Image processing with batch deployments