February 2020

  • Article

These features and Azure Databricks platform improvements were released in February 2020.

Note

Releases are staged. Your Azure Databricks account may not be updated until up to a week after the initial release date.

Databricks Runtime 6.4 for Genomics GA

February 26, 2020

Databricks Runtime 6.4 for Genomics is built on top of Databricks Runtime 6.4. It includes many improvements and upgrades from Databricks Runtime 6.3 for Genomics.

The key features are:

  • You can now customize DNASeq Pipeline users can selectively disable any legitimate combination of the read alignment, variant calling, and variant annotation stages. Users can also perform single-end read alignment.
  • The version of Glow included in Databricks Runtime 6.4 for Genomics now provides Python and Scala APIs for functions previously exposed only via SQL expressions. These functions are available for DataFrame operations, providing improved compile-time safety.

Databricks Runtime 6.4 ML GA

February 26, 2020

Databricks Runtime 6.4 ML GA brings library upgrades, including:

  • PyTorch: 1.3.1 to 1.4.0
  • Horovod: 0.18.2 to 1.19.0

For details, see the complete Databricks Runtime 6.4 for ML (EoS) release notes.

Databricks Runtime 6.4 GA

February 26, 2020

Databricks Runtime 6.4 GA brings new features, improvements, and many bug fixes.

  • Process new data files incrementally with Auto Loader (Public Preview). Auto Loader gives you a more efficient way to process new data files incrementally as they arrive on a cloud blob store during ETL. This is an improvement over file-based structured streaming, which identifies new files by repeatedly listing the cloud directory and tracking the files that have been seen, and can be very inefficient as the directory grows.
  • Load data into Delta Lake with idempotent retries (Public Preview). The COPY INTO SQL command lets you load data into Delta Lake with idempotent retries (Public Preview). To load data into Delta Lake today you have to use Apache Spark DataFrame APIs. If there are failures during loads, you have to handle them effectively.
  • Operation metrics for all writes, updates, and deletes on a Delta table now shown in table history.
  • Inline Matplotlib figures now enabled by default in Azure Databricks notebooks (Public Preview).

For details, see the complete Databricks Runtime 6.4 (EoS) release notes.

New interactive charts offer rich client-side interactions

Feb 25 - March 3, 2019: Version 3.14

This release introduces two new interactive chart types that replace the bar chart and line chart implementations. In addition to existing chart functionality, the line chart has a few new custom plot options: setting a Y-axis range, showing or hiding markers, and applying log scale to the Y-axis. Both charts have a built-in toolbar that supports a rich set of client-side interactions.

Chart toolbar

If you want to use the existing chart implementations, you can select them from the Legacy Charts drop-down menu. Existing charts will continue to use the previously available implementations.

Legacy chart types

New data ingestion network adds partner integrations with Delta Lake (Public Preview)

February 24, 2020

Now you can easily populate your "lakehouse"—your data lake empowered by the kinds of data structures and data management features you typically get with a data warehouse—from hundreds of data sources into Delta Lake. At the heart of this network is the new Partner Integrations gallery, accessible from your workspace and providing access to a huge network of data sources via our partners Fivetran, Qlik, Infoworks, StreamSets, and Syncsort.

Partner integrations portal

For an overview, see our blog. For details, see Technology partners.

Workspace creator automatically added as an Azure Databricks admin

February 24, 2020

Before February 24, 2020, the user who created an Azure Databricks workspace would only be added as an admin user for the workspace if she also clicked the Launch Workspace button in the Azure Portal or was added as an admin by a user who was already an admin user in the workspace (any Azure Contributor for the subscription who clicked the Launch Workspace button would be created as an admin user in the workspace). Now the user who creates the workspace will be added automatically as a workspace admin.

For details about creating and launching workspaces, see Manage your subscription

Flags to manage workspace security and notebook features now available

February 4-11, 2020: Version 3.12

This release introduces new flags for managing the security headers that are sent to prevent attacks on your workspace, as well as access to notebook results downloads and Git versioning. All of these administrative options are enabled by default.