July 2022

  • Article

These features and Azure Databricks platform improvements were released in July 2022.

Note

The release date and content listed below only corresponds to actual deployment of the Azure Public Cloud in most case.

It provide the evolution history of Azure Databricks service on Azure Public Cloud for your reference that may not be suitable for Azure operated by 21Vianet.

Note

Releases are staged. Your Azure Databricks account may not be updated until a week or more after the initial release date.

Databricks Runtime 11.1 and 11.1 ML are GA

July 27, 2022

Databricks Runtime 11.1 and Databricks Runtime 11.1 ML are now generally available.

See Databricks Runtime 11.1 (EoS) and Databricks Runtime 11.1 for Machine Learning (EoS).

Photon is GA

July 27, 2022

Photon is now generally available, beginning with Databricks Runtime 11.1.

Photon is the native vectorized query engine on Azure Databricks, written to be directly compatible with Apache Spark APIs so it works with your existing code. Photon is developed in C++ to take advantage of modern hardware, and uses the latest techniques in vectorized query processing to capitalize on data- and instruction-level parallelism in CPUs, enhancing performance on real-world data and applications—all natively on your data lake.

Photon is part of a high-performance runtime that runs your existing SQL and DataFrame API calls faster and reduces your total cost per workload. Photon is used by default in Databricks SQL warehouses.

See Photon is GA in the Databricks Runtime 11.1 release notes.

Notification upon notebook completion

July 27 - August 15, 2022

If you navigate away from a notebook while it's running, a notification now appears when it finishes. You can enable or disable these notifications in your browser.

Increased limit for the number of jobs in your Azure Databricks workspaces

July 19-22, 2022

You can now create up to 10,000 jobs in an Azure Databricks workspace when you enable the Increased number of jobs toggle in the Azure Databricks admin console. This feature also enables a paginated Jobs list when you view jobs in the Jobs user interface, reducing the time required to load the Jobs list view.

Databricks recommends Jobs API 2.1 if you enable the increased jobs limit. Using the Jobs 2.0 API results in degraded performance when returning large numbers of jobs.

Verbose audit logs now record when Databricks SQL queries are run

July 14, 2022

You can now configure audit logs to record when Databricks SQL queries are run. To do so, use the workspace configuration setting verbose audit logs. See Enable verbose audit logs. There is one audit log record for submitting a query and another record for query completion (including cancellation).

If verbose audit logs were already enabled for your workspace to log notebook commands, your verbose audit logs automatically now include Databricks SQL queries for the workspace.

Databricks JDBC driver 2.6.27

July 11, 2022

We have released version 2.6.27 of the Databricks JDBC driver (download). This release upgrades the third-party libraries such as Apache Arrow 7.0.0. This release also includes updates to the license file and configuration installation guide.

In addition, this release resolves the following issues:

  • In some cases, when an HTTP connection has been active for a long period of time, the connector returns a NoHttpResponseException error.
  • In some cases, unshaded Log4j classes in the connector JAR file conflict with files deployed in the end user's environment. The unshaded classes have been removed from the connector JAR file.
  • When using a DataSource class, the subprotocol names are incorrect.

Databricks ODBC driver 2.6.25

July 11, 2022

We have released version 2.6.25 of the Databricks ODBC driver (download). This release adds support to configure the connector to mitigate straggling result file download operations.

Databricks Runtime 11.1 (Beta)

July 8, 2022

Databricks Runtime 11.1, 11.1 Photon, and 11.1 ML are now available as Beta releases.

See the full release notes at Databricks Runtime 11.1 (EoS) and Databricks Runtime 11.1 for Machine Learning (EoS).

Improved notebook visualizations

July 5-11, 2022: Version 3.75

The built-in support for charts and visualizations in notebooks now supports multiple charts and data profiles. When you run a SQL cell in a notebook or use the display function on a DataFrame, the notebook shows tabbed results, similar to the results display in Databricks SQL. You can now create the same rich visualizations in a notebook that you can create in Databricks SQL. For details, see Visualizations in Databricks notebooks.