May 2022

  • Article

These features and Azure Databricks platform improvements were released in May 2022.

Note

The release date and content listed below only corresponds to actual deployment of the Azure Public Cloud in most case.

It provide the evolution history of Azure Databricks service on Azure Public Cloud for your reference that may not be suitable for Azure operated by 21Vianet.

Note

Releases are staged. Your Azure Databricks account may not be updated until a week or more after the initial release date.

Copy and paste notebook cells between tabs and windows

May 31 - June 6, 2022: Version 3.73

You can now copy and paste cells between notebooks in different browser tabs or windows. See Cut, copy, and paste cells.

Additional data type support for Databricks Feature Store automatic feature lookup

May 31 - June 6, 2022: Version 3.73

Databricks Feature Store now supports additional data types for automatic feature lookup: DecimalType, ArrayType, MapType.

Databricks Runtime 11.0 (Beta)

May 24, 2022

Databricks Runtime 11.0, 11.0 Photon, and 11.0 ML are now available as Beta releases.

See the full release notes at Databricks Runtime 11.0 (EoS) and Databricks Runtime 11.0 for Machine Learning (EoS).

Improved workspace search (Public Preview)

May 16-23, 2022: Version 3.72

You can now search for notebooks, libraries, folders, files, and repos by name. You can also search for content within a notebook and see a preview of the matching content. Search results can be filtered by type. See Search for workspace objects.

Explore SQL cell results in Python notebooks natively using Python

May 16-23, 2022: Version 3.72

In Python notebooks, the results of SQL cells are now available as a PySpark DataFrame _sqldf for easy exploration in Python. See Explore SQL cell results in Python notebooks using Python.

Note

This feature was delayed and will be rolled out over Azure Databricks platform releases 3.74 through 3.76.

Databricks Repos: Support for more files in a repo

May 16-23, 2022: Version 3.72

Databricks increased the number of files supported in a repo:

  • The notebook limit: 5,000.
  • The limit for the total of all files including notebooks: 10,000.

Previously, you were limited to 5,000 total files in a repo.

For details, see File and repo limits.

Databricks Repos: Fix to issue with MLflow experiment data loss

May 16-23, 2022: Version 3.72

Previously, in some situations, switching branches in Databricks Repos could cause the MLflow experiment associated with a notebook to be deleted. This issue has been fixed for most scenarios.

New Azure region: West Central US

May 16, 2022

Azure Databricks is now available in the West Central US region.

Upgrade wizard makes it easier to copy databases and multiple tables to Unity Catalog (Public Preview)

May 13, 2022

If you use Unity Catalog for data governance, you can now use an upgrade wizard to copy complete schemas (databases) and multiple tables from your default Hive metastore to the Unity Catalog metastore. The wizard also streamlines permission assignment for the upgraded schemas and tables. Only upgrade of external tables is supported with this release. See Upgrade a schema or multiple tables from the Hive metastore to Unity Catalog external tables using the upgrade wizard.

Power BI Desktop system-wide HTTP proxy support

May 11, 2022

The Power BI connector now supports automatic detection of the system HTTP proxy configuration in Power BI Desktop. See Automated HTTP proxy detection.

Audit logs can now record when a notebook command is run

May 10, 2022

You can now configure Azure diagnostic logs (also known as audit logs) to record when a notebook command is run. To do so, use the workspace configuration setting verbose audit logs. See Enable verbose audit logs.

Databricks Runtime 10.5 and 10.5 ML are GA; 10.5 Photon is Public Preview

May 4, 2022

Databricks Runtime 10.5 and Databricks Runtime 10.5 ML are now generally available. Databricks Runtime 10.5 Photon is in Public Preview.

See Databricks Runtime 10.5 (EoS) and Databricks Runtime 10.5 for Machine Learning (EoS).

Databricks JDBC driver 2.6.25

May 3, 2022

We have released the Databricks JDBC driver 2.6.25 (download). The driver is also available as an Apache Maven artifact. This release optimizes the default configuration used to connect to Azure Databricks. See also Databricks ODBC Driver Downloads.

See the user a pipeline runs as in the Delta Live Tables UI

May 2-9, 2022: Version 3.71

The Run as user value is added to the Pipeline details panel in the Delta Live Tables user interface. The Run as user is the pipeline owner, and pipeline updates run with this user's permissions.