Run jobs continuously

  • Article

Use continuous mode to schedule workloads to run continuously. Databricks recommends using continuous mode for always-on streaming workloads.

Continuous mode replaces legacy recommendations for Structured Streaming workloads to configure jobs with an unlimited retry policy and a maximum of one concurrent run.

Configure job to run in continuous mode

To configure a job to run in continuous mode, do the following:

  1. In the sidebar, click Workflows.
  2. Click the job name in the Name column on the Jobs tab.
  3. Click Add trigger in the Job details panel, select Continuous in Trigger type, and click Save.

To stop a continuous job, click the Pause button. Click Resume to restart the job in continuous mode.

Note

  • There can be only one running instance of a continuous job.
  • A delay exists between a run finishing and a new run starting. This delay should be less than 60 seconds.
  • You cannot use task dependencies with a continuous job.
  • You cannot use retry policies with a continuous job. Instead, continuous jobs use exponential backoff to manage job run failures.
  • Select Run now to trigger a new job run on a paused continuous job.
  • To have your continuous job pick up a new configuration, cancel the existing run. A new run automatically starts. You can also click Restart run to restart the job run with the updated configuration.

How are failures handled for continuous jobs?

Azure Databricks uses an exponential backoff scheme to manage continuous jobs with multiple consecutive failures. Exponential backoff allows continuous jobs to run without pausing and return to a healthy state when recoverable failures occur.

When a continuous job exceeds the allowable threshold for consecutive failures, the following describes how subsequent job runs are managed:

  1. The job is restarted after a retry period set by the system.
  2. If the next job run fails, the retry period is increased, and the job is restarted after this new retry period.
    1. For each subsequent job run failure, the retry period is increased up to a maximum retry period set by the system. After reaching the maximum retry period, the job continues to be retried using the maximum retry period. There is no limit on the number of retries for a continuous job.
    2. If the job run completes successfully and starts a new run, or if the run exceeds a threshold without failure, the job is considered healthy, and the backoff sequence resets.

You can restart a continuous job in the exponential backoff state in the Jobs UI or by passing the job ID to the POST /api/2.1/jobs/run-now request in the Jobs 2.1 API or the POST /api/2.0/jobs/run-now request in the Jobs 2.0 API.