Databricks Terraform provider

Article
10/21/2024

HashiCorp Terraform is a popular open source tool for creating safe and predictable cloud infrastructure across several cloud providers. You can use the Databricks Terraform provider to manage your Azure Databricks workspaces and the associated cloud infrastructure using a flexible, powerful tool. The goal of the Databricks Terraform provider is to support all Databricks REST APIs, supporting automation of the most complicated aspects of deploying and managing your data platforms. Databricks customers are using the Databricks Terraform provider to deploy and manage clusters and jobs and to configure data access. You use the Azure Provider to provision Azure Databricks workspaces.

Getting started

In this section, you install and configure requirements to use Terraform and the Databricks Terraform provider on your local development machine. You then configure Terraform authentication. Following this section, this article provides a sample configuration that you can experiment with to provision an Azure Databricks notebook, cluster, and a job to run the notebook on the cluster in an existing Azure Databricks workspace.

Requirements

You must have the Terraform CLI. See Download Terraform on the Terraform website.
You must have a Terraform project. In your terminal, create an empty directory and then switch to it. (Each separate set of Terraform configuration files must be in its own directory, which is called a Terraform project.) For example: mkdir terraform_demo && cd terraform_demo.
```
export ARM_ENVIRONMENT=china
mkdir terraform_demo && cd terraform_demo
```
Include Terraform configurations for your project in one or more configuration files in your Terraform project. For information about the configuration file syntax, see Terraform Language Documentation on the Terraform website.
You must add to your Terraform project a dependency for the Databricks Terraform provider. Add the following to one of the configuration files in your Terraform project:
```
terraform {
  required_providers {
    databricks = {
      source = "databricks/databricks"
    }
  }
}
```
You must configure authentication for your Terraform project. See Authentication in the Databricks Terraform provider documentation.

Sample configuration

This section provides a sample configuration that you can experiment with to provision an Azure Databricks notebook, a cluster, and a job to run the notebook on the cluster, in an existing Azure Databricks workspace. It assumes that you have already set up the requirements, as well as created a Terraform project and configured the project with Terraform authentication as described in the previous section.

Create a file named me.tf in your Terraform project, and add the following code. This file gets information about the current user (you):
```
# Retrieve information about the current user.
data "databricks_current_user" "me" {}
```

Create another file named notebook.tf, and add the following code. This file represents the notebook.

variable "notebook_subdirectory" {
  description = "A name for the subdirectory to store the notebook."
  type        = string
  default     = "Terraform"
}

variable "notebook_filename" {
  description = "The notebook's filename."
  type        = string
}

variable "notebook_language" {
  description = "The language of the notebook."
  type        = string
}

resource "databricks_notebook" "this" {
  path     = "${data.databricks_current_user.me.home}/${var.notebook_subdirectory}/${var.notebook_filename}"
  language = var.notebook_language
  source   = "./${var.notebook_filename}"
}

output "notebook_url" {
 value = databricks_notebook.this.url
}

Create another file named notebook.auto.tfvars, and add the following code. This file specifies the notebook's properties.

notebook_subdirectory = "Terraform"
notebook_filename     = "notebook-getting-started.py"
notebook_language     = "PYTHON"

Create another file named notebook-getting-started.py, and add the following code. This file represents the notebook's contents.
```
display(spark.range(10))
```

Create another file named cluster.tf, and add the following code. This file represents the cluster.

variable "cluster_name" {
  description = "A name for the cluster."
  type        = string
  default     = "My Cluster"
}

variable "cluster_autotermination_minutes" {
  description = "How many minutes before automatically terminating due to inactivity."
  type        = number
  default     = 60
}

variable "cluster_num_workers" {
  description = "The number of workers."
  type        = number
  default     = 1
}

# Create the cluster with the "smallest" amount
# of resources allowed.
data "databricks_node_type" "smallest" {
  local_disk = true
}

# Use the latest Databricks Runtime
# Long Term Support (LTS) version.
data "databricks_spark_version" "latest_lts" {
  long_term_support = true
}

resource "databricks_cluster" "this" {
  cluster_name            = var.cluster_name
  node_type_id            = data.databricks_node_type.smallest.id
  spark_version           = data.databricks_spark_version.latest_lts.id
  autotermination_minutes = var.cluster_autotermination_minutes
  num_workers             = var.cluster_num_workers
}

output "cluster_url" {
 value = databricks_cluster.this.url
}

Create another file named cluster.auto.tfvars, and add the following code. This file specifies the cluster's properties.

cluster_name                    = "My Cluster"
cluster_autotermination_minutes = 60
cluster_num_workers             = 1

Create another file named job.tf, and add the following code. This file represents the job that runs the notebook on the cluster.

variable "job_name" {
  description = "A name for the job."
  type        = string
  default     = "My Job"
}

variable "task_key" {
  description = "A name for the task."
  type        = string
  default     = "my_task"
}

resource "databricks_job" "this" {
  name = var.job_name
  task {
    task_key = var.task_key
    existing_cluster_id = databricks_cluster.this.cluster_id
    notebook_task {
      notebook_path = databricks_notebook.this.path
    }
  }
  email_notifications {
    on_success = [ data.databricks_current_user.me.user_name ]
    on_failure = [ data.databricks_current_user.me.user_name ]
  }
}

output "job_url" {
  value = databricks_job.this.url
}

Create another file named job.auto.tfvars, and add the following code. This file specifies the jobs's properties.
```
job_name = "My Job"
task_key = "my_task"
```
Run terraform plan. If there are any errors, fix them, and then run the command again.
Run terraform apply.
Verify that the notebook, cluster, and job were created: in the output of the terraform apply command, find the URLs for notebook_url, cluster_url, and job_url, and go to them.
Run the job: on the Jobs page, click Run now. After the job finishes, check your email inbox.
When you are done with this sample, delete the notebook, cluster, and job from the Azure Databricks workspace by running terraform destroy.

Note

For more information about the terraform plan, terraform apply, and terraform destroy commands, see Terraform CLI Documentation in the Terraform documentation.
Verify that the notebook, cluster, and job were deleted: refresh the notebook, cluster, and Jobs pages to each display a message that the resource cannot be found.

Testing

Test your Terraform configurations before or after you deploy them. You can run tests analogous to unit testing before deploying resources. You can also run tests analogous to integration testing after resources are deployed. See Tests in the Terraform documentation.

Run tests analogous to integration tests against this article's sample configuration by following this process:

Create a file named cluster.tftest.hcl, and add the following code. This file tests whether the deployed cluster has the expected cluster name.

# Filename: cluster.tftest.hcl

run "cluster_name_test" {
  command = apply

  assert {
    condition     = databricks_cluster.this.cluster_name == var.cluster_name
    error_message = "Cluster name did not match expected name"
  }
}

Create a file named job.tftest.hcl, and add the following code. This file tests whether the deployed job has the expected job name.

run "job_name_test" {
  command = apply

  assert {
    condition     = databricks_job.this.name == var.job_name
    error_message = "Job name did not match expected name"
  }
}

Create a file named notebook.tftest.hcl, and add the following code. This file tests whether the deployed notebook has the expected workspace path.

run "notebook_path_test" {
  command = apply

  assert {
    condition     = databricks_notebook.this.path == "${data.databricks_current_user.me.home}/${var.notebook_subdirectory}/${var.notebook_filename}"
    error_message = "Notebook path did not match expected path"
  }
}

Run terraform test. Terraform deploys each resource to the Azure Databricks workspace, runs each related test and reports its test result, and then tears down the deployed resource.

Run tests analogous to unit tests against this article's sample configuration with the following process:

Change the line command = apply in each of the preceding tests to command = plan, and then run terraform test. Terraform runs each related test and reports its test result but does not deploy any resources.
Mock the Databricks Terraform provider, which enables you to run terraform test without deploying resources and also without requiring any authentication credentials. See Mocks in the Terraform documentation. To run mock tests, one approach is to add the line mock_provider "databricks" {} to your tests and to remove the line command = apply or command = plan, for example:

# Filename: cluster.tftest.hcl

mock_provider "databricks" {}

run "cluster_mock_name_test" {
  assert {
    condition     = databricks_cluster.this.cluster_name == var.cluster_name
    error_message = "Cluster name did not match expected name"
  }
}

# Filename: job.tftest.hcl

mock_provider "databricks" {}

run "job_mock_name_test" {
  assert {
    condition     = databricks_job.this.name == var.job_name
    error_message = "Job name did not match expected name"
  }
}

# Filename: notebook.tftest.hcl

mock_provider "databricks" {}

run "notebook_mock_path_test" {
  assert {
    condition     = databricks_notebook.this.path == "${data.databricks_current_user.me.home}/${var.notebook_subdirectory}/${var.notebook_filename}"
    error_message = "Notebook path did not match expected path"
  }
}

Next steps

Create an Azure Databricks workspace.
Manage workspace resources for an Azure Databricks workspace.

Troubleshooting

Note

For Terraform-specific support, see the Latest Terraform topics on the HashiCorp Discuss website. For issues specific to the Databricks Terraform Provider, see Issues in the databrickslabs/terraform-provider-databricks GitHub repository.

Error: Failed to install provider

Issue: If you did not check in a terraform.lock.hcl file to your version control system, and you run the terraform init command, the following message appears: Failed to install provider. Additional output may include a message similar to the following:

Error while installing databrickslabs/databricks: v1.0.0: checksum list has no SHA-256 hash for "https://github.com/databricks/terraform-provider-databricks/releases/download/v1.0.0/terraform-provider-databricks_1.0.0_darwin_amd64.zip"

Cause: Your Terraform configurations reference outdated Databricks Terraform providers.

Solution:

Replace databrickslabs/databricks with databricks/databricks in all of your .tf files.

To automate these replacements, run the following Python command from the parent folder that contains the .tf files to update:
```
python3 -c "$(curl -Ls https://dbricks.co/updtfns)"
```
Run the following Terraform command and then approve the changes when prompted:
```
terraform state replace-provider databrickslabs/databricks databricks/databricks
```
For information about this command, see Command: state replace-provider in the Terraform documentation.
Verify the changes by running the following Terraform command:
```
terraform init
```

Error: Failed to query available provider packages

Cause: Your Terraform configurations reference outdated Databricks Terraform providers.

Solution: Follow the solution instructions in Error: Failed to install provider.

Enable logging

The Databricks Terraform provider outputs logs that you can enable by setting the TF_LOG environment variable to DEBUG or any other log level that Terraform supports.

By default, logs are sent to stderr. To send logs to a file, set the TF_LOG_PATH environment variable to the target file path.

For example, you can run the following command to enable logging at the debug level, and to output logs in monochrome format to a file named tf.log relative to the current working directory, while the terraform apply command runs:

TF_LOG=DEBUG TF_LOG_PATH=tf.log terraform apply -no-color

For more information about Terraform logging, see Debugging Terraform.

Additional examples

Additional resources

Databricks Provider Documentation on the Terraform Registry website
Terraform Documentation on the Terraform website
The terraform-databricks-examples repository in GitHub