Databricks Connect release notes

This page lists releases and maintenance updates issued for Databricks Connect.

The Databricks Runtime version of the cluster must be greater than or equal to the Databricks Connect version. Databricks recommends using the latest version to receive any bug fixes and security updates. For system environment information for Databricks Runtime releases, see Databricks Runtime release notes versions and compatibility.

Databricks Connect for Databricks Runtime 15.4 LTS

Databricks Connect 15.4.0 (Scala)

September 3, 2024

This release of Databricks Connect for Scala supports Databricks Runtime 15.4 LTS.

Databricks Connect 15.4.2 (Python)

September 13, 2024

Bug fixes:

• Fixed a bug where error messages were not complete. Error messages now include the stack trace from Spark if it is relevant.

Databricks Connect 15.4.1 (Python)

September 2, 2024

Features and improvements:

• validateSession() and the databricks-connect test command now fail when the Databricks Connect version is incompatible with the Databricks serverless compute version. For information about Databricks Connect serverless support, see Install Databricks Connect for Python.

Databricks Connect 15.4.0 (Python)

August 27, 2024

This release of Databricks Connect for Python supports Databricks Runtime 15.4 LTS.

Databricks Connect for Databricks Runtime 15.3

Databricks Connect 15.3.1 (Scala)

September 3, 2024

Minor internal fixes and improvements.

Databricks Connect 15.3.0 (Scala)

July 22, 2024

Features and improvements:

• Supported Dataset.observe for getting stats of Columns of a DataFrame. See SPARK-47545.
• Added a new function timestamp_add for adding the specified number of units to the given timestamp. See SPARK-48369.
• Added a new function timestamp_diff for getting the difference between the timestamps in the specified units by truncating the fraction. See SPARK-48272.
• Added a new function try_mod for computing the remainder of the division of two numbers, which results in null on division by zero. See SPARK-48954.
• Added a new function try_parse_json, which attempts to parse a JSON string and construct a Variant value. If the attempt fails, the result is null. See SPARK-47922.
• Changed the default value for spark.sql.legacy.timeParserPolicy from EXCEPTION to CORRECTED. Instead of raising an INCONSISTENT_BEHAVIOR_CROSS_VERSION error, CANNOT_PARSE_TIMESTAMP will be raised if ANSI mode is enabled. NULL will be returned if ANSI mode is disabled. See SPARK-47719.

Databricks Connect 15.3.4 (Python)

September 13, 2024

Bug fixes:

• Fixed a bug where error messages were not complete. Error messages now include the stack trace from Spark if it is relevant.

Databricks Connect 15.3.3 (Python)

September 2, 2024

Features and improvements:

• validateSession() and the databricks-connect test command now fail when the Databricks Connect version is incompatible with the Databricks serverless compute version. For information about Databricks Connect serverless support, see Install Databricks Connect for Python.

Databricks Connect 15.3.2 (Python)

August 27, 2024

Features and improvements:

• A warning now occurs when the Databricks Connect version is incompatible with the serverless version .
• Improved error message when the local Python version does not match the serverless version .

Databricks Connect 15.3.1 (Python)

July 18, 2024

Bug fixes:

• Serverless Spark Connect sessions expire after 10 minutes of inactivity. Databricks Connect now automatically closes expired sessions on the client side. Subsequent calls to DatabricksSession.builder.getOrCreate() will return a new active connect session.

Databricks Connect 15.3.0 (Python)

July 15, 2024

Features and improvements:

• Added a new function timestamp_add for adding the specified number of units to the given timestamp. See SPARK-48369.
• Added a new function timestamp_diff for getting the difference between the timestamps in the specified units by truncating the fraction. See SPARK-48272.
• Added a new function try_parse_json, which attempts to parse a JSON string and construct a Variant value. If the attempt fails, the result is null. See SPARK-47922.
• Changed the default value for spark.sql.legacy.timeParserPolicy from EXCEPTION to CORRECTED. Instead of raising an INCONSISTENT_BEHAVIOR_CROSS_VERSION error, CANNOT_PARSE_TIMESTAMP will be raised if ANSI mode is enabled. NULL will be returned if ANSI mode is disabled. See SPARK-47719.

Databricks Connect for Databricks Runtime 15.2

Databricks Connect 15.2.1 (Scala)

September 3, 2024

Minor internal fixes and improvements.

Databricks Connect 15.2.0 (Scala)

July 22, 2024

Features and improvements:

• The split function now supports column type. See SPARK-47845.
• Added new functions is_variant_null, variant_get, try_variant_get, schema_of_variant, and schema_of_variant_agg to improve the usability of the Variant type. See SPARK-47890.
• Added new functions collate and collation to access collation capabilities. See SPARK-47395.
• Extensions are now allowed to log extended information in the explain plan. See SPARK-47289.

Databricks Connect 15.2.4 (Python)

September 13, 2024

Bug fixes:

• Fixed a bug where error messages were not complete. Error messages now include the stack trace from Spark if it is relevant.

Databricks Connect 15.2.3 (Python)

September 2, 2024

Features and improvements:

• validateSession() and the databricks-connect test command now fail when the Databricks Connect version is incompatible with the Databricks serverless compute version. For information about Databricks Connect serverless support, see Install Databricks Connect for Python.

Databricks Connect 15.2.2 (Python)

August 27, 2024

Features and improvements:

• A warning now occurs when the Databricks Connect version is incompatible with the serverless version .

• Improved error message when the local Python version does not match the serverless version .

Databricks Connect 15.2.1 (Python)

July 18, 2024

Bug fixes:

• Serverless spark connect sessions expire after 10 minutes of inactivity. Databricks Connect now automatically closes expired sessions on the client side. Subsequent calls to DatabricksSession.builder.getOrCreate() will return a new active connect session.

Databricks Connect 15.2.0 (Python)

July 15, 2024

Features and improvements:

• The split function now supports column type. See SPARK-47845.
• Added new functions is_variant_null, variant_get, try_variant_get, schema_of_variant, and schema_of_variant_agg to improve the usability of the Variant type. See SPARK-47890.
• Added new functions collate and collation to access collation capabilities. See SPARK-47395.
• Extensions are now allowed to log extended information in the explain plan. See SPARK-47289.

Databricks Connect for Databricks Runtime 15.1

Databricks Connect 15.1.1 (Scala)

September 3, 2024

Minor internal fixes and improvements.

Databricks Connect 15.1.0 (Scala)

July 22, 2024

Features and improvements:

• Added a new function try_cast, which attempts to cast to a different data type. If the attempt fails, the result is null. See SPARK-47669.
• Added a new function monthname that returns the three-letter abbreviated month name from a given date or timestamp. See SPARK-46515.

Bug fixes:

• Session stop() API is idempotent now and will not fail if the session is already closed by the server. See SPARK-48810.
• The bloomFilter function now throws an AnalysisException for invalid input. See SPARK-46787.

Databricks Connect 15.1.3 (Python)

September 13, 2024

Bug fixes:

• Fixed a bug where error messages were not complete. Error messages now include the stack trace from Spark if it is relevant.

Databricks Connect 15.1.2 (Python)

September 2, 2024

Features and improvements:

• Improved error message when the local Python version does not match the Python version of the serverless release .

Databricks Connect 15.1.1 (Python)

July 18, 2024

Bug fixes:

• Serverless spark connect sessions expire after 10 minutes of inactivity. Databricks Connect now automatically closes expired sessions on the client side. Subsequent calls to DatabricksSession.builder.getOrCreate() will return a new active connect session.

• Supported Python versions are now limited to 3.10 and 3.11, because the distutils dependency has been removed in Python 3.12.

Databricks Connect 15.1.0 (Python)

July 15, 2024

Features and improvements:

• Databricks Connect for Python now supports connecting to serverless compute. This feature is available in Public Preview. See Configure a connection to serverless compute.
• Added a new function try_cast, which attempts to cast to a different data type. If the attempt fails, the result is null. See SPARK-47669.
• Added a new function monthname that returns the three-letter abbreviated month name from a given date or timestamp. See SPARK-46515.

Bug fixes:

• Session stop() API is idempotent now and will not fail if the session is already closed by the server. See SPARK-48810.
• Supported numpy versions are now limited to 1.15<=numpy<2.0 because PySpark 3.5.0 is incompatible with numpy 2.0.

Databricks Connect for Databricks Runtime 14.3 LTS

Databricks Connect 14.3.4 (Scala)

June 24, 2024

Features and improvements:

• A number of internal dependencies that were vendored as part of the Databricks Connect Maven package were not shaded. In this release, all internal dependencies that are not part of the Databricks Connect public API surface are shaded.

Bug fixes:

• Fixed the issue where a serial version UID mismatch error occurred when executing queries with UDFs.
• json4s and log4j have been declared as dependencies instead of being vendored. This was required as part of the fix for serial version UID.

Databricks Connect 14.3.3 (Python)

September 11, 2024

Bug fixes:

• Calling spark.stop() on an already stopped or expired session will no longer fail.
• Fixed a bug where error messages were not complete. Error messages now include the stack trace from Spark if it is relevant.

Databricks Connect 14.3.2 (Python)

May 8, 2024

Bug fixes:

• Automatically re-try plan execution in some cases of SESSION_NOT_FOUND error from the service.
• Subsequent calls to SparkSession.builder.getOrCreate() after DatabricksSession.builder.getOrCreate() will return the active default connect session.

Databricks Connect 14.3.1 (Python)

February 22, 2024

Minor fixes and internal improvements.

Databricks Connect 14.3.0 (Python)

February 1, 2024

Features and improvements:

• Databricks Connect now ships with the databricks-connect binary. Users can now validate their Python environment and the connection to the Azure Databricks cluster by running: databricks-connect test.

Bug fixes:

• Limit the number of retries when Spark Connect requests fails to the service.
• spark.table() now accepts Delta table time travel syntax.
• Aggregate functions run on top of df.melt() now work. Seehttps://issues.apache.org/jira/browse/SPARK-46484.
• DataFrame joins using queries that contain common table expressions (CTE) now work. Seehttps://issues.apache.org/jira/browse/SPARK-46337.
• df.withColumnsRenamed() now retains original dictionary ordering. Seehttps://issues.apache.org/jira/browse/SPARK-46261.
• df.withColumn() now accepts an argument of Column type. Seehttps://issues.apache.org/jira/browse/SPARK-46028.
• The behavior of df.column() in Spark Connect is now more aligned with the behavior in classic Apache Spark. Seehttps://issues.apache.org/jira/browse/SPARK-45509.
• Referencing the join column in df.drop() resolves correctly, instead of generating an "ambiguous column" error code. Seehttps://issues.apache.org/jira/browse/SPARK-45770.
• An object of Struct type can now be converted to XML using the to_xml() method. Seehttps://issues.apache.org/jira/browse/SPARK-44790.
• Improve the usability of exceptions raised by Spark Connect (for example, SparkConnectGrpcException). Seehttps://issues.apache.org/jira/browse/SPARK-45808.

Databricks Connect 14.3.0 (Scala GA)

February 1, 2024

Beginning with Databricks Runtime 14.3 LTS, Databricks Connect is now also generally available for Scala.

Features and improvements:

• The dependencies on Apache Spark packages has been removed and replaced by bundling these packages and their transitive dependencies into the Databricks Connect JAR, improving the alignment of Databricks Connect releases with the releases of the Databricks Runtime.

Databricks Connect for Databricks Runtime 14.2

Databricks Connect 14.2.1 (Python)

January 12, 2024

Bug fixes:

• Previously missed pyspark module (pyspark.sql.connect.protobuf) is now included. The to_protobuf() method is now available.

Databricks Connect 14.2.0 (Python)

December 7, 2023

Features and improvements:

• The SparkSession stop() method now closes the session on the server, interrupting release running operations and releasing resources.
• Improved error handling for addArtifact API and for cluster start up.

Databricks Connect 14.2.0 (Scala)

December 6, 2023

Features and improvements:

• The SparkSession stop() method now closes the session on the server, interrupting release running operations and releasing resources.
• Improved error handling: Previous generic GRPC and Spark Connect server exceptions now come with more user-friendly messages.
• Improved retry logic when waiting for cluster start up.

Databricks Connect for Databricks Runtime 14.1

Databricks Connect 14.1.1 (Python)

January 12, 2024

Bug fixes:

• Previously missed pyspark module (pyspark.sql.connect.protobuf) is now included. The to_protobuf() method is now available.

Databricks Connect 14.1.0 (Python)

October 23, 2023

Minor fixes and improvements.

Databricks Connect 14.1.0 (Scala)

November 7, 2023

Features and improvements:

• User-defined functions (UDFs) can now be called by using the new addCompiledArtifacts() API. See User-defined functions in Databricks Connect for Scala.
• Logging improvements.

Databricks Connect for Databricks Runtime 14.0

Databricks Connect 14.0.2 (Python)

January 12, 2024

Bug fixes:

• Previously missed pyspark module (pyspark.sql.connect.protobuf) is now included. The to_protobuf() method is now available.

Databricks Connect 14.0.1 (Python)

September 29, 2023

Bug fixes:

• Skip starting PyTorch distribution server.

Improvements:

• The AddArtifact API upload speed has been improved.

Databricks Connect 14.0.1 (Scala)

September 22, 2023

Improvements:

• The Databricks license file is now included in the JAR.

Databricks Connect 14.0.0

September 20, 2023

Features and improvements:

• Queries can now run for longer than 1 hour.
• Databricks Connect can now reconnect and resume from network interruptions while waiting for query results.
• Ability to interrupt running queries.
• Support for all Structured Streaming APIs, including foreachBatch.

Databricks Connect for Databricks Runtime 13.3 LTS

Databricks Connect 13.3.3

October 12, 2023

Bug fixes:

• Improved retry logic for network problems.

Databricks Connect 13.3.2

September 21, 2023

Bug fixes:

• We fixed a bug where createDataFrame() failed when a large dataset was used.

Databricks Connect 13.3.1

September 18, 2023

Features and improvements:

• This release improves support for mypy type hints. mypy provides optional static typing improving the IDE experience and error checking when working on Python code.

Databricks Connect 13.3.0

August 28, 2023

Features and improvements:

• The Spark Connect Service can now be disabled by setting the Spark conf on the cluster spark.databricks.service.server.enabled false.
• The DatabricksSession class is now available and usable in a Databricks notebook. Calling DatabricksSession.builder.getOrCreate() is equivalent to using the default spark object.
• It is now possible to upload local data larger than 128 MB via createDataFrame().

Bug fixes:

• The DatabricksSession.getOrCreate() API did not work correctly when configuring multiple sessions.

Databricks Connect for Databricks Runtime 13.2

Databricks Connect 13.2.1

August 22, 2023

Bug fixes:

• Fix Databricks Connect client not waiting for clusters to start up.

Databricks Connect 13.2.0

July 6, 2023

Features and improvements:

• You can now pass in custom headers as part of the Spark Connect request.
• An Azure Databricks SDK configuration profile can now be directly specified in DatabricksSession by using DatabricksSession.profile().
• The built-in PySpark REPL now recognizes connection parameters (such as the host and cluster ID) from environment variables and the .databrickscfg file. The behavior is similar to the default behaviorwhen a session is created from DatabricksSession.

Bug fixes:

• DatabricksSession was able to keep only one connection at any time. Now you can use DatabricksSession to set up multiple connections to different workspaces or the same workspace but with different credentials.
• A bug that was affecting auto-completion in Visual Studio Code has been fixed.

Databricks Connect for Databricks Runtime 13.1

Databricks Connect for Databricks Runtime 13.1.0 Python GA

June 2, 2023

Databricks Connect "v2" is now generally available for Python.

• Databricks SDK integration which enables authentication with OAuth and Azure Active Director (in Public Preview in the Python SDK).
• Support for all pandas API on Spark.
• Streaming support except for a few items in structured streaming, such as foreachBatch and streaming event listener.
• Full Delta support.

Databricks Connect for Databricks Runtime 13.0

Databricks Connect for Databricks Runtime 13.0.0 Public Preview

Beginning with Databricks Runtime 13.0, Databricks Connect builds upon Spark Connect, enabling developers to use the power of Databricks and their IDE of choice to interactively develop and debug their code in any Databricks workspace.

From this version, Databricks Connect can connect to Unity Catalog's single user and shared cluster access modes.

Databricks Connect for Databricks Runtime 12.2 LTS

Databricks Connect 12.2.33

August 20, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.30

May 30, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.27

April 25, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.26

April 8, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.24

March 14, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.23

February 27, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.22

February 14, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.21

February 2, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.20

January 24, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.19

December 13, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.18

December 4, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.17

November 7, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.16

October 30, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.15

October 16, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.14

September 28, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.13

September 11, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.12

August 28, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.11

August 18, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.10

August 7, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.9

July 19, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.8

July 5, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.7

June 15, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.6

June 12, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

Databricks Connect 12.2.0b99 Beta

May 16, 2023

• Initial Databricks Connect client release to support Databricks Runtime 12.2 LTS.
• The Python and Java integrations are functioning correctly.

Databricks Connect for Databricks Runtime 11.3 LTS

Databricks Connect 11.3.40

August 20, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.37

May 30, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.34

April 25, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.33

April 8, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.31

March 14, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.30

February 27, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.29

February 14, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.28

February 2, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.27

January 24, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.26

December 13, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.25

December 4, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.24

November 7, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.23

October 30, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.22

October 16, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.21

September 28, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.20

September 11, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.19

August 28, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.18

August 18, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.17

August 7, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.16

July 19, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.15

July 5, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.14

June 15, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.13

June 12, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.12

May 16, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.11

May 1, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.10

April 11, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

Databricks Connect 11.3.7

March 8, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release of March 8, 2023.
• [ES-589567] Fixes java.lang.NoSuchMethodError error when enabling spark.databricks.io.cache.enabled

Databricks Connect 11.3.6

February 20, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release of February 20, 2023.
• [ES-573062] Fixes java.lang.ClassCastException when assigning fields to org.apache.spark.sql.catalyst.json.JSONOptions

Databricks Connect 11.3.0b0

December 8, 2022

• Initial Databricks Connect client release to support Databricks Runtime 11.3 LTS.
• There is a known issue in the R integration (some Spark plans will fail to execute).

Databricks Connect for Databricks Runtime 10.4 LTS

Databricks Connect 10.4.55

August 20, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.52

May 30, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.49

April 25, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.48

April 8, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.46

March 14, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.45

February 27, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.44

February 14, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.43

February 2, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.42

January 24, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.41

December 13, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.40

December 4, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.39

November 7, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.38

October 30, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.37

October 16, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.36

September 28, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.35

September 11, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.34

August 28, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.33

August 18, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.32

August 7, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.31

July 19, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.30

July 5, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.29

June 15, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.28

June 12, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.27

May 16, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.
• [ES-573304] Fix issues relating to resolving tables from file paths (i.e. parquet."/some/path")

Databricks Connect 10.4.26

May 1, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.25

April 11, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

Databricks Connect 10.4.22

March 8, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release of March 8, 2023.

Databricks Connect 10.4.21

February 20, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release of February 20, 2023.
• [ES-573062] Fixes java.lang.ClassCastException when assigning fields to org.apache.spark.sql.catalyst.json.JSONOptions

Databricks Connect 10.4.18

December 1, 2022

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release of December 1, 2022.

Databricks Connect 10.4.12

September 12, 2022

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release of September 6, 2022.

Databricks Connect 10.4.11

August 25, 2022

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release of August 25, 2022.

Databricks Connect 10.4.10

August 12, 2022

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release of August 12, 2022.

Databricks Connect 10.4.9

July 29, 2022

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release of July 29, 2022.

Databricks Connect 10.4.8

July 21, 2022

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release of July 19, 2022.

Databricks Connect 10.4.6

June 16, 2022

Databricks Connect 10.4 is now GA.

Databricks Connect 10.4.0b0

March 29, 2022

• Initial Databricks Connect client release to support Databricks Runtime 10.4 LTS.
• The Python and Java integrations are functioning correctly
• There is a known issue in the R integration (some Spark plans will fail to execute)

Databricks Connect for Databricks Runtime 9.1 LTS

Databricks Connect 9.1.67

August 20, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.64

May 30, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.61

April 25, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.60

April 8, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.58

March 14, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.57

February 27, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.56

February 14, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.55

February 2, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.54

January 24, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.53

December 13, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.52

December 4, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.51

November 7, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.50

October 30, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.49

October 16, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.48

September 28, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.47

September 11, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.46

August 28, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.45

August 18, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.44

August 7, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.43

July 19, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.42

July 5, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.41

June 15, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.40

June 12, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.38

May 16, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

May 1, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.37

April 11, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

Databricks Connect 9.1.34

March 8, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of March 8, 2023.

Databricks Connect 9.1.33

February 20, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of February 20, 2023.
• [ES-573062] Fixes java.lang.ClassCastException when assigning fields to org.apache.spark.sql.catalyst.json.JSONOptions

Databricks Connect 9.1.30

December 1, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of December 1, 2022.

Databricks Connect 9.1.24

September 12, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of September 6, 2022.

Databricks Connect 9.1.23

August 25, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of August 25, 2022.

Databricks Connect 9.1.22

August 12, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of August 12, 2022.

Databricks Connect 9.1.21

July 29, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of July 29, 2022.

Databricks Connect 9.1.20

July 15, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of July 14, 2022.

Databricks Connect 9.1.18

June 16, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of June 12, 2022.

Databricks Connect 9.1.17

June 7, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of June 2, 2022.

Databricks Connect 9.1.16

May 20, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of May 20, 2022.

Databricks Connect 9.1.15

May 5, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of May 5, 2022.

Databricks Connect 9.1.14

April 21, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of April 21, 2022.

Databricks Connect 9.1.13

April 7, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of April 7, 2022.

Databricks Connect 9.1.12

March 29, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of March 29, 2022.

Databricks Connect 9.1.11

March 10, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of March 10, 2022.
• Fix ClassNotFoundException error message

Databricks Connect 9.1.10

February 23, 2022

• Security enhancements

Databricks Connect 9.1.9

February 8, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of February 8, 2022.

Databricks Connect 9.1.8

January 26, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of January 26, 2022.

Databricks Connect 9.1.7

January 19, 2022

• [ES-216554] Fixed an issue where cluster-attached libraries were not loaded correctly server-side in some cases

Databricks Connect 9.1.5

December 7, 2021

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of December 7, 2021.

Databricks Connect 9.1.4

November 16, 2021

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of November 16, 2021.

Databricks Connect 9.1.3

November 4, 2021

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of November 4, 2021.

Databricks Connect 9.1.2

October 20, 2021

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of October 20, 2021.

Databricks Connect 9.1.1

October 5, 2021

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of October 5, 2021.

Databricks Connect 9.1.0

September 23, 2021

Databricks Connect 9.1 is now GA.

Databricks Connect for Databricks Runtime 8.1

Databricks Connect 8.1.14

September 23, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of September 23, 2021.

Databricks Connect 8.1.13

September 15, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of September 15, 2021.

Databricks Connect 8.1.12

September 8, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of September 8, 2021.

Databricks Connect 8.1.11

August 25, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of August 25, 2021.

Databricks Connect 8.1.10

August 11, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of August 11, 2021.

Databricks Connect 8.1.9

July 30, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of July 30, 2021.

Databricks Connect 8.1.8

July 14, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of July 14, 2021.

Databricks Connect 8.1.7

June 29, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of June 29, 2021.

Databricks Connect 8.1.6

June 15, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of June 15, 2021.

Databricks Connect 8.1.5

June 7, 2021

• [ES-102916] Support no_proxy environment variable in DBConnnect client. Set it to a comma-separated list of fully qualified domain names for which the proxy set in https_proxy or http_proxy environment variable should not be used.

Databricks Connect 8.1.4

May 26, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of May 26, 2021.

Databricks Connect 8.1.3

April 30, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of April 30, 2021.

Databricks Connect 8.1.2

April 27, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of April 27, 2021.

Databricks Connect 8.1.1

April 7, 2021

• Improved robustness of Databricks Connect startup process.
• The previous mechanism for accessing secrets over Databricks Connect (using a dkea token) has been deprecated. Contact support to enable the new access mechanism for your workspace.
• Fixed a bug that occasionally caused Spark configurations to fail to be picked up correctly.
• [ES-77737] Fixed a bug preventing proper installation of cluster-wide libraries on Spark executors.

Databricks Connect 8.1.0

March 22, 2021

Databricks Connect 8.1 is now GA.

• Improved support of Catalog API V2 commands
• Improved support of SQL expressions in ANSI mode

Databricks Connect for Databricks Runtime 7.3 LTS

Databricks Connect 7.3.72

September 11, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

Databricks Connect 7.3.71

August 28, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

Databricks Connect 7.3.70

August 18, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

Databricks Connect 7.3.69

August 7, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

Databricks Connect 7.3.68

July 19, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

Databricks Connect 7.3.67

July 5, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

Databricks Connect 7.3.66

June 15, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

Databricks Connect 7.3.65

June 12, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

Databricks Connect 7.3.63

May 16, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

May 1, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

Databricks Connect 7.3.62

April 11, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

Databricks Connect 7.3.59

March 8, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of March 8, 2023.

Databricks Connect 7.3.58

February 20, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of February 20, 2023.
• [ES-573062] Fixes java.lang.ClassCastException when assigning fields to org.apache.spark.sql.catalyst.json.JSONOptions

Databricks Connect 7.3.55

December 1, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of December 1, 2022.

Databricks Connect 7.3.49

September 12, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of September 6, 2022.

Databricks Connect 7.3.48

August 25, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of August 25, 2022.

Databricks Connect 7.3.47

August 12, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of August 12, 2022.

Databricks Connect 7.3.46

July 29, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of July 29, 2022.

Databricks Connect 7.3.45

July 15, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of July 14, 2022.

Databricks Connect 7.3.43

June 16, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of June 12, 2022.

Databricks Connect 7.3.42

June 7, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of June 2, 2022.

Databricks Connect 7.3.41

May 20, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of May 20, 2022.

Databricks Connect 7.3.40

May 5, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of May 5, 2022.

Databricks Connect 7.3.39

April 21, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of April 21, 2022.

Databricks Connect 7.3.38

April 7, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of April 7, 2022.

Databricks Connect 7.3.37

March 29, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of March 29, 2022.

Databricks Connect 7.3.36

March 10, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of March 10, 2022.

Databricks Connect 7.3.35

February 23, 2022

• Security enhancements

Databricks Connect 7.3.34

February 8, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of February 8, 2022.

Databricks Connect 7.3.33

January 26, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of January 26, 2022.

Databricks Connect 7.3.32

January 19, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of January 19, 2022.

Databricks Connect 7.3.30

December 7, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of December 7, 2021.

Databricks Connect 7.3.29

November 16, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of November 16, 2021.

Databricks Connect 7.3.28

November 4, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of November 4, 2021.

Databricks Connect 7.3.27

October 20, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of October 20, 2021.

Databricks Connect 7.3.26

October 5, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of October 5, 2021.

Databricks Connect 7.3.25

September 23, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of September 23, 2021.

Databricks Connect 7.3.24

September 15, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of September 15, 2021.

Databricks Connect 7.3.23

September 8, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of September 8, 2021.

Databricks Connect 7.3.22

August 25, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of August 25, 2021.

Databricks Connect 7.3.21

August 11, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of August 11, 2021.

Databricks Connect 7.3.20

July 30, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of July 30, 2021.

Databricks Connect 7.3.19

July 14, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of July 14, 2021.

Databricks Connect 7.3.18

June 29, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of June 29, 2021.

Databricks Connect 7.3.17

June 15, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of June 15, 2021.

Databricks Connect 7.3.16

June 7, 2021

• [ES-102916] Support no_proxy environment variable in DBConnnect client. Set it to a comma-separated list of fully qualified domain names for which the proxy set in https_proxy or http_proxy environment variable should not be used.

Databricks Connect 7.3.15

May 26, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of May 26, 2021.

Databricks Connect 7.3.14

April 30, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of April 30, 2021.

Databricks Connect 7.3.13

April 27, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of April 27, 2021.

Databricks Connect 7.3.12

April 7, 2021

• Improved robustness of Databricks Connect startup process.
• The previous mechanism for accessing secrets over Databricks Connect (using a dkea token) has been deprecated. Contact support to enable the new access mechanism for your workspace.
• Fixed a bug that occasionally caused Spark configurations to fail to be picked up correctly.
• [ES-77737] Fixed a bug preventing proper installation of cluster-wide libraries on Spark executors.

Databricks Connect 7.3.11

March 24, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of March 24, 2021

Databricks Connect 7.3.10

March 9, 2021

• [ES-69946] Fix path separator on Windows for databricks-connect get-jar-dir

• [ES-68776] Update Azure Blob File System client

• Add support for FlatMapCoGroupsInPandas

• Make initial connection process during cluster startup more robust

  Note

  The Databricks Runtime cluster must be running the latest maintenance release, and the Databricks Connect client must be updated to version 7.3.10 to continue to use the Azure Blob File System and mapPartitions and mapElements RDD operations with Databricks Connect. A restart of the Databricks Runtime cluster is required to update it to the latest maintenance release.

Databricks Connect 7.3.9

February 23, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of February 23, 2021

Databricks Connect 7.3.8

February 4, 2021

• [ES-58153] Fix Analysis Exception when creating tables or views based on a query with aggregations.

Databricks Connect 7.3.7

January 20, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of January 20, 2021

Databricks Connect 7.3.6

January 12, 2021

• [ES-59054] Fix Delta Optimize Table command through Databricks Connect
• [ES-53497] Fix Databricks Connect on Databricks Container Services (DCS)

Databricks Connect 7.3.5

December 9, 2020

Databricks Connect 7.3 is now GA.

• [ES-49865] Prevent 500 Internal Server Errors due to automatic synchronization of dependent libraries when using Databricks Connect with SBT.

Databricks Connect 7.3.4 Beta

November 24, 2020

• [ES-44382] Set spark.databricks.service.allowTerminatedClusterStart=false to prevent Databricks Connect from starting a terminated Databricks Runtime cluster.
• Fix an error where client could not synchronize state with server correctly if INFO logging level was disabled.

Databricks Connect 7.3.3 Beta

November 3, 2020

Initial Databricks Connect release for Databricks Runtime 7.3. This release includes:

• Support for Microsoft Entra ID credential passthrough. You can now use credential passthrough with Standard clusters using Databricks Connect. This enables you to authenticate automatically to Azure Data Lake Storage Gen2 from Databricks Connect using the same Microsoft Entra ID identity that you use to authenticate to Azure Databricks.

• Support for Delta Lake time travel.
• Easy transition between Databricks Connect clients and Databricks Runtime jobs or notebooks when using DBUtils.

Known issues:

Databricks Runtime version validation is disabled in this beta release. Make sure to connect this client to a 7.3 Databricks Runtime cluster.

Databricks Connect for Databricks Runtime 7.1 (Unsupported)

Databricks Connect 7.1.14

February 4, 2021

• Databricks Connect client update to support Databricks Runtime 7.1 maintenance release of February 4, 2021

Databricks Connect 7.1.13

January 20, 2021

• Databricks Connect client update to support Databricks Runtime 7.1 maintenance release of January 20, 2021

Databricks Connect 7.1.12

January 12, 2021

• [ES-59054] Fix Delta Optimize Table command through Databricks Connect
• [ES-53497] Fix Databricks Connect on Databricks Container Services (DCS)

Databricks Connect 7.1.11

December 9, 2020

• [ES-49865] Prevent 500 Internal Server Errors due to automatic synchronization of dependent libraries when using Databricks Connect with SBT.

Databricks Connect 7.1.10

November 24, 2020

• Prevent status polls from restarting the cluster.
• [ES-43656] Fix serialization error related to Azure Data Lake Storage Gen2 support.
• [ES-30594] Allow larger HTTP headers and enlarge buffers in Databricks Connect.
• [ES-34186] Avoid out of memory exception related to logging.
• [ES-39493] Prevent recursive load error in databricks-connect test.
• [ES-44382] Set spark.databricks.service.allowTerminatedClusterStart=false to prevent Databricks Connect from starting a Databricks Runtime cluster that is terminated.
• Fix an error in which client could not synchronize state with server correctly if INFO logging level was disabled.
• Minor fixes.

Databricks Connect 7.1.1

September 23, 2020

• [ES-32536] Fix an issue with serialization of nested timezone-aware expressions.
• [ES-33705] Support Azure Data Lake Storage Gen2 filesystem in DButils.

Databricks Connect 7.1.0

August 13, 2020

Initial Databricks Connect release for Databricks Runtime 7.1.

• Support new Spark 3.0 DataSource V2 APIs.
• Support using multiple SparkSessions in a single Databricks Connect application.
• Optimize Databricks Connect state synchronization between client and server. Avoid excessive polling for status updates when the server is idle.

Databricks Connect for Databricks Runtime 6.4

Databricks Connect 6.4.49

February 23, 2022

• Security enhancements

Databricks Connect 6.4.28

April 7, 2021

• The previous mechanism for accessing secrets over Databricks Connect (using a dkea token) has been deprecated. Contact support to enable the new access mechanism for your workspace.

Databricks Connect 6.4.27

March 24, 2021

• Databricks Connect client update to support Databricks Runtime 6.4 maintenance release of March 24, 2021

Databricks Connect 6.4.26

March 15, 2021

• New version has been released, but there are no changes compared to Databricks Connect 6.4.25

Databricks Connect 6.4.25

March 9, 2021

• [ES-69946] Fix path separator on Windows for databricks-connect get-jar-dir
• Add support for FlatMapCoGroupsInPandas

Databricks Connect 6.4.24

February 23, 2021

• Databricks Connect client update to support Databricks Runtime 6.4 maintenance release of February 23, 2021

Databricks Connect 6.4.23

February 4, 2021

• Databricks Connect client update to support Databricks Runtime 6.4 maintenance release of February 4, 2021

Databricks Connect 6.4.22

January 12, 2021

• [ES-59054] Fix Delta Optimize Table command through Databricks Connect
• [ES-53497] Fix Databricks Connect on Databricks Container Services (DCS)

Databricks Connect 6.4.21

December 9, 2020

• [ES-25890] Fix an error where in some cases DataFrame cache() would have no effect and the query would be executed again.
• [ES-26196] Support using NATURAL and USING syntax in joins.
• [ES-33475][ES-34186] Avoid out of memory exception related to logging.
• [ES-39493] Prevent recursive load error in databricks-connect test.
• [ES-43656] Fix serialization error related to Azure Data Lake Storage Gen2 support.
• [ES-30594] Allow larger HTTP headers and enlarge buffers in Databricks Connect.
• Minor fixes

Databricks Connect 6.4.2

November 5, 2020

• [ES-47216] Fix version check compatibility issue with Databricks Runtime 6.4 November 3, 2020 maintenance update.

Databricks Connect 6.4.1

April 8, 2020

Minor fixes and updates.

Databricks Connect 6.4.0

March 7, 2020

Initial Databricks Connect release for Databricks Runtime 6.4.