Schedule data import jobs (preview)

• Install the Azure CLI and the ml extension. Follow the installation steps in Install, set up, and use the CLI (v2).

• Create an Azure Machine Learning workspace if you don't have one. For workspace creation, see Install, set up, and use the CLI (v2).

• Create an Azure Machine Learning workspace if you don't have one.
• The Azure Machine Learning SDK v2 for Python.

• An Azure Machine Learning workspace. See Create workspace resources.
• Understanding of Azure Machine Learning pipelines. See what are machine learning pipelines, and how to create pipeline job in CLI v2 or SDK v2.

APPLIES TO: Azure CLI ml extension v2 (current)

YAML: Schedule for data import with recurrence pattern

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: simple_recurrence_import_schedule
display_name: Simple recurrence import schedule
description: a simple hourly recurrence import schedule

trigger:
  type: recurrence
  frequency: day #can be minute, hour, day, week, month
  interval: 1 #every day
  schedule:
    hours: [4,5,10,11,12]
    minutes: [0,30]
  start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time
  time_zone: "Pacific Standard Time" # optional - default will be UTC

import_data: ./my-snowflake-import-data.yaml

YAML: Schedule for data import definition inline with recurrence pattern on managed datastore

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: inline_recurrence_import_schedule
display_name: Inline recurrence import schedule
description: an inline hourly recurrence import schedule

trigger:
  type: recurrence
  frequency: day #can be minute, hour, day, week, month
  interval: 1 #every day
  schedule:
    hours: [4,5,10,11,12]
    minutes: [0,30]
  start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time
  time_zone: "Pacific Standard Time" # optional - default will be UTC

import_data:
  type: mltable
  name: my_snowflake_ds
  path: azureml://datastores/workspacemanagedstore
  source:
    type: database
    query: select * from TPCH_SF1.REGION
    connection: azureml:my_snowflake_connection

A trigger contains these properties:

• (Required) type specifies the schedule type, either recurrence or cron. The following section has more information.

Next, run this command in the CLI:

> az ml schedule create -f <file-name>.yml

APPLIES TO: Python SDK azure-ai-ml v2 (current)

from azure.ai.ml.data_transfer import Database
from azure.ai.ml.constants import TimeZone
from azure.ai.ml.entities import (
    ImportDataSchedule,
    RecurrenceTrigger,
    RecurrencePattern,
)
from datetime import datetime

source = Database(connection="azureml:my_sf_connection", query="select * from my_table")

path = "azureml://datastores/workspaceblobstore/paths/snowflake/schedule/${{name}}"


my_data = DataImport(
    type="mltable", source=source, path=path, name="my_schedule_sfds_test"
)

schedule_name = "my_simple_sdk_create_schedule_recurrence"

schedule_start_time = datetime.utcnow()

recurrence_trigger = RecurrenceTrigger(
    frequency="day",
    interval=1,
    schedule=RecurrencePattern(hours=1, minutes=[0, 1]),
    start_time=schedule_start_time,
    time_zone=TimeZone.UTC,
)

import_schedule = ImportDataSchedule(
    name=schedule_name, trigger=recurrence_trigger, import_data=my_data
)

ml_client.schedules.begin_create_or_update(import_schedule).result()

RecurrenceTrigger contains following properties:

• (Required) For a better coding experience, use RecurrenceTrigger for the recurrence schedule.

When your data import has satisfactory performance and outputs, you can set up a schedule to automatically trigger that import.

1. Navigate to Azure Machine Learning studio

2. Under Assets in the left navigation, select Data. At the Data import tab, select the imported data asset to which you want to attach a schedule. The Import jobs history page should appear, as shown in this screenshot:

   

3. At the Import jobs history page, select the latest Import job name hyperlink URL, to open the pipelines job details page as shown in this screenshot:

   

4. At the pipeline job details page of any data import, select Schedule -> Create new schedule to open the schedule creation wizard, as shown in this screenshot:

   

5. The Basic settings of the schedule creation wizard have the properties shown in this screenshot:

   

• Name: the unique identifier of the schedule within the workspace.

• Description: the schedule description.

• Trigger: the recurrence pattern of the schedule, which includes these properties:

  • Time zone: the trigger time calculation is based on this time zone; (UTC) Coordinated Universal Time by default.
  • Recurrence or Cron expression: select recurrence to specify the recurring pattern. Under Recurrence, you can specify the recurrence frequency - by minutes, hours, days, weeks, or months.
  • Start: the schedule first becomes active on this date. By default, the creation date of this schedule.
  • End: the schedule will become inactive after this date. By default, it's NONE, which means that the schedule remains active until you manually disable it.
  • Tags: the selected schedule tags.

  After you configure the basic settings, you can select Review + Create, and the schedule will automatically submit the data import based on the recurrence pattern you specified. You can also select Next, and navigate through the wizard to select or update the data import parameters.

YAML: Schedule for a data import with cron expression

APPLIES TO: Azure CLI ml extension v2 (current)

YAML: Schedule for data import with cron expression (preview)

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: simple_cron_import_schedule
display_name: Simple cron import schedule
description: a simple hourly cron import schedule

trigger:
  type: cron
  expression: "0 * * * *"
  start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time
  time_zone: "Pacific Standard Time" # optional - default will be UTC

import_data: ./my-snowflake-import-data.yaml

YAML: Schedule for data import definition inline with cron expression (preview)

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: inline_cron_import_schedule
display_name: Inline cron import schedule
description: an inline hourly cron import schedule

trigger:
  type: cron
  expression: "0 * * * *"
  start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time
  time_zone: "Pacific Standard Time" # optional - default will be UTC

import_data:
  type: mltable
  name: my_snowflake_ds
  path: azureml://datastores/workspaceblobstore/paths/snowflake/${{name}}
  source:
    type: database
    query: select * from TPCH_SF1.REGION
    connection: azureml:my_snowflake_connection

The trigger section defines the schedule details and contains these properties:

• (Required) type specifies the cron schedule type.

> az ml schedule create -f <file-name>.yml

The list continues here:

APPLIES TO: Python SDK azure-ai-ml v2 (current)

from azure.ai.ml.data_transfer import Database
from azure.ai.ml.constants import TimeZone
from azure.ai.ml.entities import CronTrigger, ImportDataSchedule

source = Database(connection="azureml:my_sf_connection", query="select * from my_table")

path = "azureml://datastores/workspaceblobstore/paths/snowflake/schedule/${{name}}"


my_data = DataImport(
    type="mltable", source=source, path=path, name="my_schedule_sfds_test"
)

schedule_name = "my_simple_sdk_create_schedule_cron"

cron_trigger = CronTrigger(
    expression="15 10 * * 1",
    start_time=datetime.utcnow(),
    end_time="2023-12-03T18:40:00",
)
import_schedule = ImportDataSchedule(
    name=schedule_name, trigger=cron_trigger, import_data=my_data
)
ml_client.schedules.begin_create_or_update(import_schedule).result()

The CronTrigger section defines the schedule details and contains these properties:

• (Required) For a better coding experience, use CronTrigger for the recurrence schedule.

The list continues here:

When your data import has satisfactory performance and outputs, you can set up a schedule to automatically trigger that import.

1. Navigate to Azure Machine Learning studio

2. Under Assets in the left navigation, select Data. On the Data import tab, select the imported data asset to which you want to attach a schedule. The Import jobs history page should appear, as shown in this screenshot:

   

3. At the Import jobs history page, select the latest Import job name hyperlink URL, to open the pipelines job details page as shown in this screenshot:

   

4. At the pipeline job details page of any data import, select Schedule -> Create new schedule to open the schedule creation wizard, as shown in this screenshot:

   

5. The Basic settings of the schedule creation wizard have the properties shown in this screenshot:

   

• Name: the unique identifier of the schedule within the workspace.

• Description: the schedule description.

• Trigger: the recurrence pattern of the schedule, which includes these properties:

  • Time zone: the trigger time calculation is based on this time zone; (UTC) Coordinated Universal Time by default.
  • Recurrence or Cron expression: select recurrence to specify the recurring pattern. With Cron expression, you can specify a more flexible and customized recurrence pattern.
  • Start: the schedule first becomes active on this date. By default, the creation date of this schedule.
  • End: the schedule will become inactive after this date. By default, it's NONE, which means that the schedule remains active until you manually disable it.
  • Tags: the selected schedule tags.

  After you configure the basic settings, you can select Review + Create, and the schedule will automatically submit the data import based on the recurrence pattern you specified. You can also select Next, and navigate through the wizard to select or update the data import parameters.

APPLIES TO: Azure CLI ml extension v2 (current)

az ml schedule list

APPLIES TO: Python SDK azure-ai-ml v2 (current)

schedules = ml_client.schedules.list()
[s.name for s in schedules]

In the studio portal, under the Jobs extension, select the All schedules tab. That tab shows all your job schedules created by the SDK/cli/UI, in a single list. In the schedule list, you have an overview of all schedules in this workspace, as shown in this screenshot:

APPLIES TO: Azure CLI ml extension v2 (current)

az ml schedule show -n simple_cron_data_import_schedule

APPLIES TO: Python SDK azure-ai-ml v2 (current)

created_schedule = ml_client.schedules.get(name=schedule_name)
[created_schedule.name]

You can select a schedule name to show the schedule details page. The schedule details page contains the following tabs, as shown in this screenshot:

• Overview: basic information for the specified schedule.

  

• Job definition: defines the job that the specified schedule triggers, as shown in this screenshot:

  

APPLIES TO: Azure CLI ml extension v2 (current)

az ml schedule update -n simple_cron_data_import_schedule  --set description="new description" --no-wait

APPLIES TO: Python SDK azure-ai-ml v2 (current)

job_schedule = ml_client.schedules.begin_create_or_update(
    schedule=job_schedule
).result()
print(job_schedule)

Update a data import definition to existing schedule

To change the import frequency, or to create a new association for the data import job, you can update the import definition of an existing schedule.

1. Navigate to Azure Machine Learning studio

2. Under Assets in the left navigation, select Data. On the Data import tab, select the imported data asset to which you want to attach a schedule. The Import jobs history page should appear, as shown in this screenshot:

   

3. At the Import jobs history page, select the latest Import job name link, to open the pipelines job details page as shown in this screenshot:

   

4. At the pipeline job details page of any data import, select Schedule -> Updated to existing schedule, to open the Select schedule wizard, as shown in this screenshot:

   

5. Select an existing schedule from the list, as shown in this screenshot:

   

   Important

   Make sure you select the correct schedule to update. Once you finish the update, the schedule will trigger different data imports.

6. You can also modify the source, query and change the destination path, for future data imports that the schedule triggers.

7. Select Review + Update to finish the update process. The completed update will send a notification.

8. You can view the new data import definition in the schedule details page when the update is completed.

Update in schedule detail page

In the schedule details page, you can select Update settings to update both the basic settings and advanced settings, including the job input/output and runtime settings of the schedule, as shown in this screenshot:

APPLIES TO: Azure CLI ml extension v2 (current)

az ml schedule disable -n simple_cron_data_import_schedule --no-wait

job_schedule = ml_client.schedules.begin_disable(name=schedule_name).result()
job_schedule.is_enabled

You can disable the current schedule at the schedule details page. You can also disable schedules at the All schedules tab.

APPLIES TO: Azure CLI ml extension v2 (current)

az ml schedule enable -n simple_cron_data_import_schedule --no-wait

APPLIES TO: Python SDK azure-ai-ml v2 (current)

# Update trigger expression
job_schedule.trigger.expression = "10 10 * * 1"
job_schedule = ml_client.schedules.begin_create_or_update(
    schedule=job_schedule
).result()
print(job_schedule)

At the schedule details page, you can enable the current schedule. You can also enable schedules at the All schedules tab.

APPLIES TO: Azure CLI ml extension v2 (current)

az ml schedule delete -n simple_cron_data_import_schedule

APPLIES TO: Python SDK azure-ai-ml v2 (current)

# Only disabled schedules can be deleted
ml_client.schedules.begin_disable(name=schedule_name).result()
ml_client.schedules.begin_delete(name=schedule_name).result()

You can delete a schedule from the schedule details page or the all schedules tab.