May 2023

  • Article

These features and Azure Databricks platform improvements were released in May 2023.

Note

The release date and content listed below only corresponds to actual deployment of the Azure Public Cloud in most case.

It provide the evolution history of Azure Databricks service on Azure Public Cloud for your reference that may not be suitable for Azure operated by 21Vianet.

Note

Releases are staged. Your Azure Databricks account might not be updated until a week or more after the initial release date.

Create or modify table UI supports Avro, Parquet, and text file uploads

May 29, 2023

The Create or modify table UI now supports the following file types:

  • Avro (.avro)
  • Parquet (.parquet)
  • Text (.txt)

For more information, see Create or modify a table using file upload.

Use the add data UI to load data using Unity Catalog external locations (Public Preview)

May 29, 2023

You can now use the add data UI to load data from a cloud object storage path that's defined as a Unity Catalog external location. For more information, see Load data using a Unity Catalog external location.

Run Databricks notebooks on SQL warehouses (Public Preview)

May 31, 2023

You can now run Databricks notebooks on SQL warehouses. SQL warehouses are optimized for SQL analytics and deliver better price performance for SQL execution compared to all-purpose compute. For more information, see Use a notebook with a SQL warehouse.

Prevent Enter key from accepting autocomplete suggestion

May 26, 2023

You can now prevent the Enter key from accepting autocomplete suggestions in the SQL editor. Select Disable enter to accept autocomplete from the kebab menu in the SQL editor.

New region: China East 3

May 25, 2023

Azure Databricks is now available in Azure region China East 3. See Azure Databricks regions.

Support for Azure confidential computing VM types (Public Preview)

May 25, 2023

Azure Databricks now supports using Azure confidential computing VM types when creating clusters. Azure confidential computing helps protect data in use, preventing the cloud provider from having access to sensitive data. See Azure confidential computing VMs.

Databricks Terraform provider updated to version 1.17.0

May 22, 2023

Databricks Terraform provider 1.17.0 removes support for releasing 32-bit binaries, adds more information on the impact of using a cluster policy in the databricks_cluster resource, and more. For more details, see the changelog for version 1.17.0.

Upload data UI supports new column data types

May 19, 2023

The upload data UI now supports the following column data types:

  • STRUCT
  • ARRAY
  • DECIMAL(P,S)

For more information, see Create or modify a table using file upload.

Bind Unity Catalog catalogs to specific workspaces

May 17, 2023

You can now limit catalog access to specific workspaces in your account, also known as workspace-catalog binding. This feature is especially helpful if you use workspaces to isolate user data access, for example if you have separate production and development workspaces or a dedicated workspace for handling sensitive data.

For more information, see Limit catalog access to specific workspaces.

All users can connect to Fivetran using Partner Connect (Public Preview)

May 17, 2023

The workspace admin user role is no longer required to connect to Fivetran using Partner Connect. For more information, see Connect to Fivetran using Partner Connect.

Databricks Terraform provider updated to version 1.16.1

May 16, 2023

Databricks Terraform provider 1.16.1 adds documentation for the databricks_service_principal_secret resource and Unity Catalog integration in the databricks_pipeline resource, updates documentation for the databricks_secret_scope resource, and more. For more details, see the changelog for version 1.16.1.

Databricks JDBC driver 2.6.33

May 15, 2023

We have released version 2.6.33 of the Databricks JDBC driver (download). This release adds Primary Key and Foreign Key APIs when connecting to a server that has Unity Catalog enabled.

Partner Connect supports Alation

May 15-22, 2023

You can now connect your Azure Databricks workspace to Alation using Partner Connect. For more information, see Connect Azure Databricks to Alation.

New default theme for editor

May 12, 2023

Based on customer feedback, the default color scheme in the editor was updated for better readability.

Databricks Terraform provider updated to version 1.16.0

May 12, 2023

Databricks Terraform provider 1.16.0 adds a databricks_workspace_file resource, adds a notification_settings block to the databricks_job resource, and more. For more details, see the changelog for version 1.16.0.

Unity Catalog support for Azure managed identities is now GA

May 11, 2023

The ability to use Azure managed identities to access storage containers on behalf of Unity Catalog users is now generally available. Azure managed identities provide significant advantages over service principals for this use case. See Use Azure managed identities in Unity Catalog to access storage.

Databricks Runtime 13.1 (Beta)

May 10, 2023

Databricks Runtime 13.1 and Databricks Runtime 13.1 ML are now available as Beta releases.

See Databricks Runtime 13.1 (EoS) and Databricks Runtime 13.1 for Machine Learning (EoS).

Run file-based SQL queries in an Azure Databricks workflow

May 9, 2023

You can now retrieve SQL queries from a Git provider when you add a SQL task to an Azure Databricks job. See Use Git with jobs.

Databricks Terraform provider updated to version 1.15.0

May 5, 2023

Version 1.15.0 adds an experimental databricks_sql_table resource, adds support for running databricks_cluster init scripts from workspace files, adds missing permissions to databricks_grants, and more. For more details, see the changelog for version 1.15.0.

Azure operated by 21Vianet regions now support Unity Catalog

May 5, 2023

China East 2, China North 2, and China North 3 regions now support Unity Catalog.

Account nicknames now available in the account console

May 5, 2023

Account admins can now set account nicknames for their Azure Databricks accounts. After you give your account a nickname, the name is displayed at the top of the account console and next to the account ID. For more information, see Add an account nickname.

Share notebooks using Delta Sharing

May 4, 2023

You can now use Delta Sharing to share notebook files securely using the Databricks-to-Databricks sharing flow.

Sharing notebooks empowers users to collaborate across metastores and accounts, and enables providers to demonstrate use cases and visualizations for the data that they share.

See Add notebook files to a share (for providers) and Read shared notebooks (for recipients).

Azure Databricks no longer creates a serverless starter SQL warehouse

May 3-12, 2023

Azure Databricks no longer creates a serverless starter SQL warehouse. To create your own, see Create a SQL warehouse.

In SQL Warehouses API, enabling serverless compute now must be explicit

May 3-12, 2023

There was a minor change in the SQL Warehouses API when creating a SQL warehouse.

For the field enable_serverless_compute, previously the default was true if serverless SQL endpoints were enabled for the workspace. The default is now false for most workspaces. However, if this workspace used the SQL Warehouses API to create a warehouse between November 1, 2022 and May 19, 2023, and fits the requirements for serverless SQL warehouse including its region, the default remains set to true.

To avoid ambiguity, especially for organizations with many workspaces, Databricks strongly recommends that you always explicitly set the enable_serverless_compute field.

Changes for workspace settings for serverless SQL warehouses

May 3-12, 2023

For accounts that have serverless compute enabled, workspaces are now enabled for serverless features by default. As part of this change, the workspace-level setting to enable or disable serverless SQL warehouses was removed from the workspace settings UI.

Changes for serverless compute settings for accounts and workspaces

May 3-12, 2023

All workspaces are now enabled for serverless SQL warehouses if the region supports it.

Azure Databricks notifies every workspace admin once in the Databricks SQL UI for each workspace.

For newly-enabled workspaces to support serverless SQL warehouses, each workspace must meet the requirements and might require an update to each workspace's Azure Storage firewall .

Databricks SQL Serverless is GA

May 3-12, 2023

Databricks SQL Serverless is now generally available in supported regions. Serverless SQL warehouses provide instant compute, minimal management, and cost optimization for SQL queries. Create serverless SQL warehouses, or convert pro or classic SQL warehouses to serverless.

May 3, 2023

You can enable secure cluster connectivity (SCC) on an existing workspace to make your VNet have no open ports and Databricks Runtime cluster nodes have no public IP addresses. This feature is configured in Azure templates using the name No Public IP (enableNoPublicIp).

You can enable or disable Azure Private Link on an existing workspace for private connectivity between users and their Databricks workspaces, and also between compute resources and the control plane of Azure Databricks infrastructure.

Both updates require the workspace to use VNet injection and the Premium pricing tier.

Deprecation of cluster-scoped init scripts on DBFS

May 2, 2023

Cluster-scoped init scripts on DBFS are deprecated. They should be stored as workspace files instead. See Store init scripts in workspace files.

Unified navigation (Public Preview)

May 1, 2023

The new unified navigation experience is in Public Preview. Unified navigation reduces clicks and context switches required to complete tasks across product areas. For more information, see Navigate the workspace.