Databricks Runtime 14.1
The following release notes provide information about Databricks Runtime 14.1, powered by Apache Spark 3.5.0.
Databricks released this version in October 2023.
Tip
To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.
New features and improvements
- array_insert() is 1-based for negative indexes
- Delta v2 checkpoints enabled by default with liquid clustering
- Drop Delta table feature in Public Preview
- Delta Sharing: Recipients can perform batch queries on shared tables with deletion vectors (Public Preview)
- Delta Sharing: Recipients can perform batch queries on shared tables with column mapping (Public Preview)
- Stream from Unity Catalog views in Public Preview
- Apache Pulsar connector in Public Preview
- Upgraded Snowflake driver
- SQL Session variables
- Named parameter invocation for SQL and Python UDF.
- Table arguments to functions support partitioning and ordering.
- New and enhanced builtin SQL functions
- Improved handling of correlated subqueries
array_insert() is 1-based for negative indexes
The array_insert function is 1-based for both positive and negative indexes. It now inserts new element at the end of input arrays for the index -1. To restore the previous behavior, set spark.sql.legacy.negativeIndexInArrayInsert to true.
Delta v2 checkpoints enabled by default with liquid clustering
Newly created Delta tables with liquid clustering use v2 checkpoints by default. See Compatibility for tables with liquid clustering.
Drop Delta table feature in Public Preview
You can now drop some table features for Delta tables. Current support includes dropping deletionVectors and v2Checkpoint. See Drop Delta table features.
Delta Sharing: Recipients can perform batch queries on shared tables with deletion vectors (Public Preview)
Delta Sharing recipients can now perform batch queries on shared tables that use deletion vectors. See Add tables with deletion vectors or column mapping to a share, Read tables with deletion vectors or column mapping enabled, and Read tables with deletion vectors or column mapping enabled.
Delta Sharing: Recipients can perform batch queries on shared tables with column mapping (Public Preview)
Delta Sharing recipients can now perform batch queries on shared tables that use column mapping. See Add tables with deletion vectors or column mapping to a share, Read tables with deletion vectors or column mapping enabled, and Read tables with deletion vectors or column mapping enabled.
Stream from Unity Catalog views in Public Preview
You can now use Structured Streaming to perform streaming reads from views registered with Unity Catalog. Azure Databricks only supports streaming reads from views defined against Delta tables. See Read a Unity Catalog view as a stream.
Apache Pulsar connector in Public Preview
You can now use Structured Streaming to stream data from Apache Pulsar on Azure Databricks. See Stream from Apache Pulsar.
Upgraded Snowflake driver
The Snowflake JDBC driver now uses version 3.13.33.
SQL Session variables
This release introduces the ability to declare temporary variables in a session which can be set and then referred to from within queries. See Variables.
Named parameter invocation for SQL and Python UDF.
You can now use Named parameter invocation on SQL and Python UDF.
Table arguments to functions support partitioning and ordering.
You can now use PARTITION BY and ORDER BY clauses to control how table arguments are passed to a function during function invocation.
New and enhanced builtin SQL functions
The following builtin functions have been added:
- from_xml: Parses an XML
STRINGinto aSTRUCT. - schema_of_xml: Derives a schema from an XML
STRING. - session_user: Returns the logged-in user.
- try_reflect: Returns
NULLinstead of the exception if a Java method fails.
The following builtin functions have been enhanced:
- mode: Support for an optional parameter forcing a deterministic result.
- to_char: New support for
DATE,TIMESTAMP, andBINARY. - to_varchar: New support for
DATE,TIMESTAMP, andBINARY.
Improved handling of correlated subqueries
The ability to process correlation in subqueries has been extended:
- Handle limit and order by in correlated scalar (lateral) subqueries.
- Support window functions in correlated scalar subqueries.
- Support correlated references in join predicates for scalar and lateral subqueries
Behavior change
Strict type checking in Photon Parquet reader
Photon fails when trying to read a decimal value from a Parquet column that is not a decimal type. Photon also fails when reading a fixed-length byte array from Parquet as a string.
Library upgrades
- Upgraded Python libraries:
- filelock from 3.12.2 to 3.12.3
- s3transfer from 0.6.1 to 0.6.2
- Upgraded Java libraries:
- com.uber.h3 from 3.7.0 to 3.7.3
- io.airlift.aircompressor from 0.24 to 0.25
- io.delta.delta-sharing-spark_2.12 from 0.7.1 to 0.7.5
- io.netty.netty-all from 4.1.93.Final to 4.1.96.Final
- io.netty.netty-buffer from 4.1.93.Final to 4.1.96.Final
- io.netty.netty-codec from 4.1.93.Final to 4.1.96.Final
- io.netty.netty-codec-http from 4.1.93.Final to 4.1.96.Final
- io.netty.netty-codec-http2 from 4.1.93.Final to 4.1.96.Final
- io.netty.netty-codec-socks from 4.1.93.Final to 4.1.96.Final
- io.netty.netty-common from 4.1.93.Final to 4.1.96.Final
- io.netty.netty-handler from 4.1.93.Final to 4.1.96.Final
- io.netty.netty-handler-proxy from 4.1.93.Final to 4.1.96.Final
- io.netty.netty-resolver from 4.1.93.Final to 4.1.96.Final
- io.netty.netty-transport from 4.1.93.Final to 4.1.96.Final
- io.netty.netty-transport-classes-epoll from 4.1.93.Final to 4.1.96.Final
- io.netty.netty-transport-classes-kqueue from 4.1.93.Final to 4.1.96.Final
- io.netty.netty-transport-native-epoll from 4.1.93.Final-linux-x86_64 to 4.1.96.Final-linux-x86_64
- io.netty.netty-transport-native-kqueue from 4.1.93.Final-osx-x86_64 to 4.1.96.Final-osx-x86_64
- io.netty.netty-transport-native-unix-common from 4.1.93.Final to 4.1.96.Final
- net.snowflake.snowflake-jdbc from 3.13.29 to 3.13.33
- org.apache.orc.orc-core from 1.9.0-shaded-protobuf to 1.9.1-shaded-protobuf
- org.apache.orc.orc-mapreduce from 1.9.0-shaded-protobuf to 1.9.1-shaded-protobuf
- org.apache.orc.orc-shims from 1.9.0 to 1.9.1
Apache Spark
Databricks Runtime 14.1 includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 14.0 (EoS), as well as the following additional bug fixes and improvements made to Spark:
- [SPARK-45088] [DBRRM-557] Revert "[SC-142785][PYTHON][CONNECT] Make getitem work with duplicated column"
- [SPARK-43453] [DBRRM-557]Revert "[SC-143135][PS] Ignore the
namesofMultiIndexwhenaxis=1forconcat" - [SPARK-45225] [SC-143207][SQL] XML: XSD file URL support
- [SPARK-45156] [SC-142782][SQL] Wrap
inputNameby backticks in theNON_FOLDABLE_INPUTerror class - [SPARK-44910] [SC-143082][SQL] Encoders.bean does not support superclasses with generic type arguments
- [SPARK-43453] [SC-143135][PS] Ignore the
namesofMultiIndexwhenaxis=1forconcat - [SPARK-44463] [SS][CONNECT] Improve error handling for Connect steaming Python worker
- [SPARK-44960] [SC-141023][UI] Unescape and consist error summary across UI pages
- [SPARK-44788] [SC-142980][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
- [SPARK-44614] [SC-138460][PYTHON][CONNECT][3.5] Add missing packages in setup.py
- [SPARK-45151] [SC-142861][CORE][UI] Task Level Thread Dump Support
- [SPARK-45056] [SC-142779][PYTHON][SS][CONNECT] Termination tests for streamingQueryListener and foreachBatch
- [SPARK-45084] [SC-142828][SS] StateOperatorProgress to use accurate effective shuffle partition number
- [SPARK-44872] [SC-142405][CONNECT] Server testing infra and ReattachableExecuteSuite
- [SPARK-45197] [SC-142984][CORE] Make
StandaloneRestServeraddJavaModuleOptionsto drivers - [SPARK-44404] [SC-139601][SQL] Assign names to the error class LEGACY_ERROR_TEMP[1009,1010,1013,1015,1016,1278]
- [SPARK-44647] [SC-142297][SQL] Support SPJ where join keys are less than cluster keys
- [SPARK-45088] [SC-142785][PYTHON][CONNECT] Make
getitemwork with duplicated columns - [SPARK-45128] [SC-142851][SQL] Support
CalendarIntervalTypein Arrow - [SPARK-45130] [SC-142976][CONNECT][ML][PYTHON] Avoid Spark connect ML model to change input pandas dataframe
- [SPARK-45034] [SC-142959][SQL] Support deterministic mode function
- [SPARK-45173] [SC-142931][UI] Remove some unnecessary sourceMapping files in UI
- [SPARK-45162] [SC-142781][SQL] Support maps and array parameters constructed via
call_function - [SPARK-45143] [SC-142840][PYTHON][CONNECT] Make PySpark compatible with PyArrow 13.0.0
- [SPARK-45174] [SC-142837][CORE] Support
spark.deploy.maxDrivers - [SPARK-45167] [SC-142956][CONNECT][PYTHON] Python client must call
release_all - [SPARK-36191] [SC-142777][SQL] Handle limit and order by in correlated scalar (lateral) subqueries
- [SPARK-45159] [SC-142829][PYTHON] Handle named arguments only when necessary
- [SPARK-45133] [SC-142512][CONNECT] Make Spark Connect queries be FINISHED when last result task is finished
- [SPARK-44801] [SC-140802][SQL][UI] Capture analyzing failed queries in Listener and UI
- [SPARK-45139] [SC-142527][SQL] Add DatabricksDialect to handle SQL type conversion
- [SPARK-45157] [SC-142546][SQL] Avoid repeated
ifchecks in[On|Off|HeapColumnVector - [SPARK-45077] Revert "[SC-142069][UI] Upgrade dagre-d3.js from 04.3 to 0.6.4"
- [SPARK-45145] [SC-142521][EXAMPLE] Add JavaSparkSQLCli example
- [SPARK-43295] Revert "[SC-142254][PS] Support string type columns for
DataFrameGroupBy.sum" - [SPARK-44915] [SC-142383][CORE] Validate checksum of remounted PVC's shuffle data before recovery
- [SPARK-45147] [SC-142524][CORE] Remove
System.setSecurityManagerusage - [SPARK-45104] [SC-142206][UI] Upgrade
graphlib-dot.min.jsto 1.0.2 - [SPARK-44238] [SC-141606][CORE][SQL] Introduce a new
readFrommethod with byte array input forBloomFilter - [SPARK-45060] [SC-141742][SQL] Fix an internal error from
to_char()onNULLformat - [SPARK-43252] [SC-142381][SQL] Replace the error class
_LEGACY_ERROR_TEMP_2016with an internal error - [SPARK-45069] [SC-142279][SQL] SQL variable should always be resolved after outer reference
- [SPARK-44911] [SC-142388][SQL] Create hive table with invalid column should return error class
- [SPARK-42754] [SC-125567][SQL][UI] Fix backward compatibility issue in nested SQL execution
- [SPARK-45121] [SC-142375][CONNECT][PS] Support
Series.emptyfor Spark Connect. - [SPARK-44805] [SC-142203][SQL] getBytes/getShorts/getInts/etc. should work in a column vector that has a dictionary
- [SPARK-45027] [SC-142248][PYTHON] Hide internal functions/variables in
pyspark.sql.functionsfrom auto-completion - [SPARK-45073] [SC-141791][PS][CONNECT] Replace
LastNotNullwithLast(ignoreNulls=True) - [SPARK-44901] [SC-141676][SQL] Manual backport: Add API in Python UDTF 'analyze' method to return partitioning/ordering expressions
- [SPARK-45076] [SC-141795][PS] Switch to built-in
repeatfunction - [SPARK-44162] [SC-141605][CORE] Support G1GC in spark metrics
- [SPARK-45053] [SC-141733][PYTHON][MINOR] Log improvement in python version mismatch
- [SPARK-44866] [SC-142221][SQL] Add
SnowflakeDialectto handle BOOLEAN type correctly - [SPARK-45064] [SC-141775][PYTHON][CONNECT] Add the missing
scaleparameter inceil/ceiling - [SPARK-45059] [SC-141757][CONNECT][PYTHON] Add
try_reflectfunctions to Scala and Python - [SPARK-43251] [SC-142280][SQL] Replace the error class
_LEGACY_ERROR_TEMP_2015with an internal error - [SPARK-45052] [SC-141736][SQL][PYTHON][CONNECT] Make function aliases output column name consistent with SQL
- [SPARK-44239] [SC-141502][SQL] Free memory allocated by large vectors when vectors are reset
- [SPARK-43295] [SC-142254][PS] Support string type columns for
DataFrameGroupBy.sum - [SPARK-45080] [SC-142062][SS] Explicitly call out support for columnar in DSv2 streaming data sources
- [SPARK-45036] [SC-141768][SQL] SPJ: Simplify the logic to handle partially clustered distribution
- [SPARK-45077] [SC-142069][UI] Upgrade dagre-d3.js from 0.4.3 to 0.6.4
- [SPARK-45091] [SC-142020][PYTHON][CONNECT][SQL] Function
floor/round/broundaccept Column typescale - [SPARK-45090] [SC-142019][PYTHON][CONNECT]
DataFrame.{cube, rollup}support column ordinals - [SPARK-44743] [SC-141625][SQL] Add
try_reflectfunction - [SPARK-45086] [SC-142052][UI] Display hexadecimal for thread lock hash code
- [SPARK-44952] [SC-141644][SQL][PYTHON] Support named arguments in aggregate Pandas UDFs
- [SPARK-44987] [SC-141552][SQL] Assign a name to the error class
_LEGACY_ERROR_TEMP_1100 - [SPARK-45032] [SC-141730][CONNECT] Fix compilation warnings related to
Top-level wildcard is not allowed and will error under -Xsource:3 - [SPARK-45048] [SC-141629][CONNECT] Add additional tests for Python client and attachable execution
- [SPARK-45072] [SC-141807][CONNECT] Fix outer scopes for ammonite classes
- [SPARK-45033] [SC-141759][SQL] Support maps by parameterized
sql() - [SPARK-45066] [SC-141772][SQL][PYTHON][CONNECT] Make function
repeataccept column-typen - [SPARK-44860] [SC-141103][SQL] Add SESSION_USER function
- [SPARK-45074] [SC-141796][PYTHON][CONNECT]
DataFrame.{sort, sortWithinPartitions}support column ordinals - [SPARK-45047] [SC-141774][PYTHON][CONNECT]
DataFrame.groupBysupport ordinals - [SPARK-44863] [SC-140798][UI] Add a button to download thread dump as a txt in Spark UI
- [SPARK-45026] [SC-141604][CONNECT]
spark.sqlshould support datatypes not compatible with arrow - [SPARK-44999] [SC-141145][CORE] Refactor
ExternalSorterto reduce checks onshouldPartitionwhen callinggetPartition - [SPARK-42304] [SC-141501][SQL] Rename
_LEGACY_ERROR_TEMP_2189toGET_TABLES_BY_TYPE_UNSUPPORTED_BY_HIVE_VERSION - [SPARK-43781] [SC-139450][SQL] Fix IllegalStateException when cogrouping two datasets derived from the same source
- [SPARK-45018] [SC-141516][PYTHON][CONNECT] Add CalendarIntervalType to Python Client
- [SPARK-45024] [SC-141513][PYTHON][CONNECT] Filter out some configurations in Session Creation
- [SPARK-45017] [SC-141508][PYTHON] Add
CalendarIntervalTypeto PySpark - [SPARK-44720] [SC-139375][CONNECT] Make Dataset use Encoder instead of AgnosticEncoder
- [SPARK-44982] [SC-141027][CONNECT] Mark Spark Connect server configurations as static
- [SPARK-44839] [SC-140900][SS][CONNECT] Better Error Logging when user tries to serialize spark session
- [SPARK-44865] [SC-140905][SS] Make StreamingRelationV2 support metadata column
- [SPARK-45001] [SC-141141][PYTHON][CONNECT] Implement DataFrame.foreachPartition
- [SPARK-44497] [SC-141017][WEBUI] Show task partition id in Task table
- [SPARK-45006] [SC-141143][UI] Use the same date format of other UI date elements for the x-axis of timelines
- [SPARK-45000] [SC-141135][PYTHON][CONNECT] Implement DataFrame.foreach
- [SPARK-44967] [SC-141137][SQL][CONNECT] Unit should be considered first before using Boolean for TreeNodeTag
- [SPARK-44993] [SC-141088][CORE] Add
ShuffleChecksumUtils.compareChecksumsby reusingShuffleChecksumTestHelp.compareChecksums - [SPARK-44807] [SC-140176][CONNECT] Add Dataset.metadataColumn to Scala Client
- [SPARK-44965] [SC-141098][PYTHON] Hide internal functions/variables from
pyspark.sql.functions - [SPARK-44983] [SC-141030][SQL] Convert binary to string by
to_charfor the formats:hex,base64,utf-8 - [SPARK-44980] [DBRRM-462][SC-141024][PYTHON][CONNECT] Fix inherited namedtuples to work in createDataFrame
- [SPARK-44985] [SC-141033][CORE] Use toString instead of stacktrace for task reaper threadDump
- [SPARK-44984] [SC-141028][PYTHON][CONNECT] Remove
_get_aliasfrom DataFrame - [SPARK-44975] [SC-141013][SQL] Remove BinaryArithmetic useless override resolved
- [SPARK-44969] [SC-140957][SQL] Reuse
ArrayInsertinArrayAppend - [SPARK-44549] [SC-140714][SQL] Support window functions in correlated scalar subqueries
- [SPARK-44938] [SC-140811][SQL] Change default value of
spark.sql.maxSinglePartitionBytesto 128m - [SPARK-44918] [SC-140816][SQL][PYTHON] Support named arguments in scalar Python/Pandas UDFs
- [SPARK-44966] [SC-140907][CORE][CONNECT] Change the never changed
vartoval - [SPARK-41471] [SC-140804][SQL] Reduce Spark shuffle when only one side of a join is KeyGroupedPartitioning
- [SPARK-44214] [SC-140528][CORE] Support Spark Driver Live Log UI
- [SPARK-44861] [SC-140716][CONNECT] jsonignore SparkListenerConnectOperationStarted.planRequest
- [SPARK-44776] [SC-140519][CONNECT] Add ProducedRowCount to SparkListenerConnectOperationFinished
- [SPARK-42017] [SC-140765][PYTHON][CONNECT]
df['col_name']should validate the column name - [SPARK-40178] [SC-140433][SQL][COONECT] Support coalesce hints with ease for PySpark and R
- [SPARK-44840] [SC-140593][SQL] Make
array_insert()1-based for negative indexes - [SPARK-44939] [SC-140778][R] Support Java 21 in SparkR SystemRequirements
- [SPARK-44936] [SC-140770][CORE] Simplify the log when Spark HybridStore hits the memory limit
- [SPARK-44908] [SC-140712][ML][CONNECT] Fix cross validator foldCol param functionality
- [SPARK-44816] [SC-140717][CONNECT] Improve error message when UDF class is not found
- [SPARK-44909] [SC-140710][ML] Skip starting torch distributor log streaming server when it is not available
- [SPARK-44920] [SC-140707][CORE] Use await() instead of awaitUninterruptibly() in TransportClientFactory.createClient()
- [SPARK-44905] [SC-140703][SQL] Stateful lastRegex causes NullPointerException on eval for regexp_replace
- [SPARK-43987] [SC-139594][Shuffle] Separate finalizeShuffleMerge Processing to Dedicated Thread Pools
- [SPARK-42768] [SC-140549][SQL] Enable cached plan apply AQE by default
- [SPARK-44741] [SC-139447][CORE] Support regex-based MetricFilter in
StatsdSink - [SPARK-44751] [SC-140532][SQL] XML FileFormat Interface implementation
- [SPARK-44868] [SC-140438][SQL] Convert datetime to string by
to_char/to_varchar - [SPARK-44748] [SC-140504][SQL] Query execution for the PARTITION BY clause in UDTF TABLE arguments
- [SPARK-44873] [SC-140427] Support alter view with nested columns in Hive client
- [SPARK-44876] [SC-140431][PYTHON] Fix Arrow-optimized Python UDF on Spark Connect
- [SPARK-44520] [SC-137845][SQL] Replace the term UNSUPPORTED_DATA_SOURCE_FOR_DIRECT_QUERY with UNSUPPORTED_DATASOURCE_FOR_DIRECT_QUERY and disclosure root AE
- [SPARK-42664] [SC-139769][CONNECT] Support
bloomFilterfunction forDataFrameStatFunctions - [SPARK-43567] [SC-139227][PS] Support
use_na_sentinelforfactorize - [SPARK-44859] [SC-140254][SS] Fix incorrect property name in structured streaming doc
- [SPARK-44822] [SC-140182][PYTHON][SQL] Make Python UDTFs by default non-deterministic
- [SPARK-44731] [SC-139524][PYTHON][CONNECT] Make TimestampNTZ works with literals in Python Spark Connect
- [SPARK-44836] [SC-140180][PYTHON] Refactor Arrow Python UDTF
- [SPARK-44714] [SC-139238] Ease restriction of LCA resolution regarding queries with having
- [SPARK-44749] [SC-139664][SQL][PYTHON] Support named arguments in Python UDTF
- [SPARK-44737] [SC-139512][SQL][UI] Should not display json format errors on SQL page for non-SparkThrowables on SQL Tab
- [SPARK-44665] [SC-139307][PYTHON] Add support for pandas DataFrame assertDataFrameEqual
- [SPARK-44736] [SC-139622][CONNECT] Add Dataset.explode to Spark Connect Scala Client
- [SPARK-44732] [SC-139422][SQL] Built-in XML data source support
- [SPARK-44694] [SC-139213][PYTHON][CONNECT] Refactor active sessions and expose them as an API
- [SPARK-44652] [SC-138881] Raise error when only one df is None
- [SPARK-44562] [SC-138824][SQL] Add OptimizeOneRowRelationSubquery in batch of Subquery
- [SPARK-44717] [SC-139319][PYTHON][PS] Respect TimestampNTZ in resampling
- [SPARK-42849] [SC-139365] [SQL] Session Variables
- [SPARK-44236] [SC-139239][SQL] Disable WholeStageCodegen when set
spark.sql.codegen.factoryModeto NO_CODEGEN - [SPARK-44695] [SC-139316][PYTHON] Improve error message for
DataFrame.toDF - [SPARK-44680] [SC-139234][SQL] Improve the error for parameters in
DEFAULT - [SPARK-43402] [SC-138321][SQL] FileSourceScanExec supports push down data filter with scalar subquery
- [SPARK-44641] [SC-139216][SQL] Incorrect result in certain scenarios when SPJ is not triggered
- [SPARK-44689] [SC-139219][CONNECT] Make the exception handling of function
SparkConnectPlanner#unpackScalarScalaUDFmore universal - [SPARK-41636] [SC-139061][SQL] Make sure
selectFiltersreturns predicates in deterministic order - [SPARK-44132] [SC-139197][SQL] Materialize
Streamof join column names to avoid codegen failure - [SPARK-42330] [SC-138838][SQL] Assign the name
RULE_ID_NOT_FOUNDto the error class_LEGACY_ERROR_TEMP_2175 - [SPARK-44683] [SC-139214][SS] Logging level isn't passed to RocksDB state store provider correctly
- [SPARK-44658] [SC-138868][CORE]
ShuffleStatus.getMapStatusshould returnNoneinstead ofSome(null) - [SPARK-44603] [SC-138353] Add pyspark.testing to setup.py
- [SPARK-44252] [SC-137505][SS] Define a new error class and apply for the case where loading state from DFS fails
- [SPARK-29497] [DBRRM-396][SC-138477][CONNECT] Throw error when UDF is not deserializable.
- [SPARK-44624] [DBRRM-396][SC-138900][CONNECT] Retry ExecutePlan in case initial request didn't reach server
- [SPARK-41400] [DBRRM-396][SC-138287][CONNECT] Remove Connect Client Catalyst Dependency
- [SPARK-44059] [SC-138833][SQL] Add better error messages for SQL named argumnts
- [SPARK-44620] [SC-138831][SQL][PS][CONNECT] Make
ResolvePivotretain thePlan_ID_TAG - [SPARK-43838] [SC-137413][SQL] Fix subquery on single table with having clause can't be optimized
- [SPARK-44555] [SC-138820][SQL] Use checkError() to check Exception in command Suite & assign some error class names
- [SPARK-44280] [SC-138821][SQL] Add convertJavaTimestampToTimestamp in JDBCDialect API
- [SPARK-44602] [SC-138337][SQL][CONNECT][PS] Make
WidenSetOperationTypesretain thePlan_ID_TAG - [SPARK-42941] [SC-138389][SS][CONNECT] Python StreamingQueryListener
- [SPARK-43838] Revert "[SC-137413][SQL] Fix subquery on single ta…
- [SPARK-44538] [SC-138178][CONNECT][SQL] Reinstate Row.jsonValue and friends
- [SPARK-44421] [SC-138434][SPARK-44423][CONNECT] Reattachable execution in Spark Connect
- [SPARK-43838] [SC-137413][SQL] Fix subquery on single table with having clause can't be optimized
- [SPARK-44587] [SC-138315][SQL][CONNECT] Increase protobuf marshaller recursion limit
- [SPARK-44605] [SC-138338][CORE] Refine internal ShuffleWriteProcessor API
- [SPARK-44394] [SC-138291][CONNECT][WEBUI] Add a Spark UI page for Spark Connect
- [SPARK-44585] [SC-138286][MLLIB] Fix warning condition in MLLib RankingMetrics ndcgAk
- [SPARK-44198] [SC-138137][CORE] Support propagation of the log level to the executors
- [SPARK-44454] [SC-138071][SQL][HIVE] HiveShim getTablesByType support fallback
- [SPARK-44425] [SC-138135][CONNECT] Validate that user provided sessionId is an UUID
- [SPARK-43611] [SC-138051][SQL][PS][CONNCECT] Make
ExtractWindowExpressionsretain thePLAN_ID_TAG - [SPARK-44560] [SC-138117][PYTHON][CONNECT] Improve tests and documentation for Arrow Python UDF
- [SPARK-44482] [SC-138067][CONNECT] Connect server should can specify the bind address
- [SPARK-44528] [SC-138047][CONNECT] Support proper usage of hasattr() for Connect dataframe
- [SPARK-44525] [SC-138043][SQL] Improve error message when Invoke method is not found
- [SPARK-44355] [SC-137878][SQL] Move WithCTE into command queries
Databricks ODBC/JDBC driver support
Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).
System environment
- Operating System: Ubuntu 22.04.3 LTS
- Java: Zulu 8.72.0.17-CA-linux64
- Scala: 2.12.15
- Python: 3.10.12
- R: 4.3.1
- Delta Lake: 3.0.0
Installed Python libraries
| Library | Version | Library | Version | Library | Version |
|---|---|---|---|---|---|
| anyio | 3.5.0 | argon2-cffi | 21.3.0 | argon2-cffi-bindings | 21.2.0 |
| asttokens | 2.0.5 | attrs | 22.1.0 | backcall | 0.2.0 |
| beautifulsoup4 | 4.11.1 | black | 22.6.0 | bleach | 4.1.0 |
| blinker | 1.4 | boto3 | 1.24.28 | botocore | 1.27.96 |
| certifi | 2022.12.7 | cffi | 1.15.1 | chardet | 4.0.0 |
| charset-normalizer | 2.0.4 | click | 8.0.4 | comm | 0.1.2 |
| contourpy | 1.0.5 | cryptography | 39.0.1 | cycler | 0.11.0 |
| Cython | 0.29.32 | databricks-sdk | 0.1.6 | dbus-python | 1.2.18 |
| debugpy | 1.6.7 | decorator | 5.1.1 | defusedxml | 0.7.1 |
| distlib | 0.3.7 | docstring-to-markdown | 0.11 | entrypoints | 0.4 |
| executing | 0.8.3 | facets-overview | 1.1.1 | fastjsonschema | 2.18.0 |
| filelock | 3.12.3 | fonttools | 4.25.0 | GCC runtime library | 1.10.0 |
| googleapis-common-protos | 1.60.0 | grpcio | 1.48.2 | grpcio-status | 1.48.1 |
| httplib2 | 0.20.2 | idna | 3.4 | importlib-metadata | 4.6.4 |
| ipykernel | 6.25.0 | ipython | 8.14.0 | ipython-genutils | 0.2.0 |
| ipywidgets | 7.7.2 | jedi | 0.18.1 | jeepney | 0.7.1 |
| Jinja2 | 3.1.2 | jmespath | 0.10.0 | joblib | 1.2.0 |
| jsonschema | 4.17.3 | jupyter-client | 7.3.4 | jupyter-server | 1.23.4 |
| jupyter_core | 5.2.0 | jupyterlab-pygments | 0.1.2 | jupyterlab-widgets | 1.0.0 |
| keyring | 23.5.0 | kiwisolver | 1.4.4 | launchpadlib | 1.10.16 |
| lazr.restfulclient | 0.14.4 | lazr.uri | 1.0.6 | lxml | 4.9.1 |
| MarkupSafe | 2.1.1 | matplotlib | 3.7.0 | matplotlib-inline | 0.1.6 |
| mccabe | 0.7.0 | mistune | 0.8.4 | more-itertools | 8.10.0 |
| mypy-extensions | 0.4.3 | nbclassic | 0.5.2 | nbclient | 0.5.13 |
| nbconvert | 6.5.4 | nbformat | 5.7.0 | nest-asyncio | 1.5.6 |
| nodeenv | 1.8.0 | notebook | 6.5.2 | notebook_shim | 0.2.2 |
| numpy | 1.23.5 | oauthlib | 3.2.0 | packaging | 22.0 |
| pandas | 1.5.3 | pandocfilters | 1.5.0 | parso | 0.8.3 |
| pathspec | 0.10.3 | patsy | 0.5.3 | pexpect | 4.8.0 |
| pickleshare | 0.7.5 | Pillow | 9.4.0 | pip | 22.3.1 |
| platformdirs | 2.5.2 | plotly | 5.9.0 | pluggy | 1.0.0 |
| prometheus-client | 0.14.1 | prompt-toolkit | 3.0.36 | protobuf | 4.24.0 |
| psutil | 5.9.0 | psycopg2 | 2.9.3 | ptyprocess | 0.7.0 |
| pure-eval | 0.2.2 | pyarrow | 8.0.0 | pycparser | 2.21 |
| pydantic | 1.10.6 | pyflakes | 3.0.1 | Pygments | 2.11.2 |
| PyGObject | 3.42.1 | PyJWT | 2.3.0 | pyodbc | 4.0.32 |
| pyparsing | 3.0.9 | pyright | 1.1.294 | pyrsistent | 0.18.0 |
| python-dateutil | 2.8.2 | python-lsp-jsonrpc | 1.0.0 | python-lsp-server | 1.7.1 |
| pytoolconfig | 1.2.5 | pytz | 2022.7 | pyzmq | 23.2.0 |
| requests | 2.28.1 | rope | 1.7.0 | s3transfer | 0.6.2 |
| scikit-learn | 1.1.1 | seaborn | 0.12.2 | SecretStorage | 3.3.1 |
| Send2Trash | 1.8.0 | setuptools | 65.6.3 | six | 1.16.0 |
| sniffio | 1.2.0 | soupsieve | 2.3.2.post1 | ssh-import-id | 5.11 |
| stack-data | 0.2.0 | statsmodels | 0.13.5 | tenacity | 8.1.0 |
| terminado | 0.17.1 | threadpoolctl | 2.2.0 | tinycss2 | 1.2.1 |
| tokenize-rt | 4.2.1 | tomli | 2.0.1 | tornado | 6.1 |
| traitlets | 5.7.1 | typing_extensions | 4.4.0 | ujson | 5.4.0 |
| unattended-upgrades | 0.1 | urllib3 | 1.26.14 | virtualenv | 20.16.7 |
| wadllib | 1.3.6 | wcwidth | 0.2.5 | webencodings | 0.5.1 |
| websocket-client | 0.58.0 | whatthepatch | 1.0.2 | wheel | 0.38.4 |
| widgetsnbextension | 3.6.1 | yapf | 0.31.0 | zipp | 1.0.0 |
| SciPy | 1.10.1 |
Installed R libraries
R libraries are installed from the Posit Package Manager CRAN snapshot on 2023-02-10.
| Library | Version | Library | Version | Library | Version |
|---|---|---|---|---|---|
| arrow | 12.0.1 | askpass | 1.1 | assertthat | 0.2.1 |
| backports | 1.4.1 | base | 4.3.1 | base64enc | 0.1-3 |
| bit | 4.0.5 | bit64 | 4.0.5 | blob | 1.2.4 |
| boot | 1.3-28 | brew | 1.0-8 | brio | 1.1.3 |
| broom | 1.0.5 | bslib | 0.5.0 | cachem | 1.0.8 |
| callr | 3.7.3 | caret | 6.0-94 | cellranger | 1.1.0 |
| chron | 2.3-61 | class | 7.3-22 | cli | 3.6.1 |
| clipr | 0.8.0 | clock | 0.7.0 | cluster | 2.1.4 |
| codetools | 0.2-19 | colorspace | 2.1-0 | commonmark | 1.9.0 |
| compiler | 4.3.1 | config | 0.3.1 | conflicted | 1.2.0 |
| cpp11 | 0.4.4 | crayon | 1.5.2 | credentials | 1.3.2 |
| curl | 5.0.1 | data.table | 1.14.8 | datasets | 4.3.1 |
| DBI | 1.1.3 | dbplyr | 2.3.3 | desc | 1.4.2 |
| devtools | 2.4.5 | diagram | 1.6.5 | diffobj | 0.3.5 |
| digest | 0.6.33 | downlit | 0.4.3 | dplyr | 1.1.2 |
| dtplyr | 1.3.1 | e1071 | 1.7-13 | ellipsis | 0.3.2 |
| evaluate | 0.21 | fansi | 1.0.4 | farver | 2.1.1 |
| fastmap | 1.1.1 | fontawesome | 0.5.1 | forcats | 1.0.0 |
| foreach | 1.5.2 | foreign | 0.8-82 | forge | 0.2.0 |
| fs | 1.6.2 | future | 1.33.0 | future.apply | 1.11.0 |
| gargle | 1.5.1 | generics | 0.1.3 | gert | 1.9.2 |
| ggplot2 | 3.4.2 | gh | 1.4.0 | gitcreds | 0.1.2 |
| glmnet | 4.1-7 | globals | 0.16.2 | glue | 1.6.2 |
| googledrive | 2.1.1 | googlesheets4 | 1.1.1 | gower | 1.0.1 |
| graphics | 4.3.1 | grDevices | 4.3.1 | grid | 4.3.1 |
| gridExtra | 2.3 | gsubfn | 0.7 | gtable | 0.3.3 |
| hardhat | 1.3.0 | haven | 2.5.3 | highr | 0.10 |
| hms | 1.1.3 | htmltools | 0.5.5 | htmlwidgets | 1.6.2 |
| httpuv | 1.6.11 | httr | 1.4.6 | httr2 | 0.2.3 |
| ids | 1.0.1 | ini | 0.3.1 | ipred | 0.9-14 |
| isoband | 0.2.7 | iterators | 1.0.14 | jquerylib | 0.1.4 |
| jsonlite | 1.8.7 | KernSmooth | 2.23-21 | knitr | 1.43 |
| labeling | 0.4.2 | later | 1.3.1 | lattice | 0.21-8 |
| lava | 1.7.2.1 | lifecycle | 1.0.3 | listenv | 0.9.0 |
| lubridate | 1.9.2 | magrittr | 2.0.3 | markdown | 1.7 |
| MASS | 7.3-60 | Matrix | 1.5-4.1 | memoise | 2.0.1 |
| methods | 4.3.1 | mgcv | 1.8-42 | mime | 0.12 |
| miniUI | 0.1.1.1 | ModelMetrics | 1.2.2.2 | modelr | 0.1.11 |
| munsell | 0.5.0 | nlme | 3.1-162 | nnet | 7.3-19 |
| numDeriv | 2016.8-1.1 | openssl | 2.0.6 | parallel | 4.3.1 |
| parallelly | 1.36.0 | pillar | 1.9.0 | pkgbuild | 1.4.2 |
| pkgconfig | 2.0.3 | pkgdown | 2.0.7 | pkgload | 1.3.2.1 |
| plogr | 0.2.0 | plyr | 1.8.8 | praise | 1.0.0 |
| prettyunits | 1.1.1 | pROC | 1.18.4 | processx | 3.8.2 |
| prodlim | 2023.03.31 | profvis | 0.3.8 | progress | 1.2.2 |
| progressr | 0.13.0 | promises | 1.2.0.1 | proto | 1.0.0 |
| proxy | 0.4-27 | ps | 1.7.5 | purrr | 1.0.1 |
| r2d3 | 0.2.6 | R6 | 2.5.1 | ragg | 1.2.5 |
| randomForest | 4.7-1.1 | rappdirs | 0.3.3 | rcmdcheck | 1.4.0 |
| RColorBrewer | 1.1-3 | Rcpp | 1.0.11 | RcppEigen | 0.3.3.9.3 |
| readr | 2.1.4 | readxl | 1.4.3 | recipes | 1.0.6 |
| rematch | 1.0.1 | rematch2 | 2.1.2 | remotes | 2.4.2 |
| reprex | 2.0.2 | reshape2 | 1.4.4 | rlang | 1.1.1 |
| rmarkdown | 2.23 | RODBC | 1.3-20 | roxygen2 | 7.2.3 |
| rpart | 4.1.19 | rprojroot | 2.0.3 | Rserve | 1.8-11 |
| RSQLite | 2.3.1 | rstudioapi | 0.15.0 | rversions | 2.1.2 |
| rvest | 1.0.3 | sass | 0.4.6 | scales | 1.2.1 |
| selectr | 0.4-2 | sessioninfo | 1.2.2 | shape | 1.4.6 |
| shiny | 1.7.4.1 | sourcetools | 0.1.7-1 | sparklyr | 1.8.1 |
| SparkR | 3.5.0 | spatial | 7.3-15 | splines | 4.3.1 |
| sqldf | 0.4-11 | SQUAREM | 2021.1 | stats | 4.3.1 |
| stats4 | 4.3.1 | stringi | 1.7.12 | stringr | 1.5.0 |
| survival | 3.5-5 | sys | 3.4.2 | systemfonts | 1.0.4 |
| tcltk | 4.3.1 | testthat | 3.1.10 | textshaping | 0.3.6 |
| tibble | 3.2.1 | tidyr | 1.3.0 | tidyselect | 1.2.0 |
| tidyverse | 2.0.0 | timechange | 0.2.0 | timeDate | 4022.108 |
| tinytex | 0.45 | tools | 4.3.1 | tzdb | 0.4.0 |
| urlchecker | 1.0.1 | usethis | 2.2.2 | utf8 | 1.2.3 |
| utils | 4.3.1 | uuid | 1.1-0 | vctrs | 0.6.3 |
| viridisLite | 0.4.2 | vroom | 1.6.3 | waldo | 0.5.1 |
| whisker | 0.4.1 | withr | 2.5.0 | xfun | 0.39 |
| xml2 | 1.3.5 | xopen | 1.0.0 | xtable | 1.8-4 |
| yaml | 2.3.7 | zip | 2.3.0 |
Installed Java and Scala libraries (Scala 2.12 cluster version)
| Group ID | Artifact ID | Version |
|---|---|---|
| antlr | antlr | 2.7.7 |
| com.amazonaws | amazon-kinesis-client | 1.12.0 |
| com.amazonaws | aws-java-sdk-autoscaling | 1.12.390 |
| com.amazonaws | aws-java-sdk-cloudformation | 1.12.390 |
| com.amazonaws | aws-java-sdk-cloudfront | 1.12.390 |
| com.amazonaws | aws-java-sdk-cloudhsm | 1.12.390 |
| com.amazonaws | aws-java-sdk-cloudsearch | 1.12.390 |
| com.amazonaws | aws-java-sdk-cloudtrail | 1.12.390 |
| com.amazonaws | aws-java-sdk-cloudwatch | 1.12.390 |
| com.amazonaws | aws-java-sdk-cloudwatchmetrics | 1.12.390 |
| com.amazonaws | aws-java-sdk-codedeploy | 1.12.390 |
| com.amazonaws | aws-java-sdk-cognitoidentity | 1.12.390 |
| com.amazonaws | aws-java-sdk-cognitosync | 1.12.390 |
| com.amazonaws | aws-java-sdk-config | 1.12.390 |
| com.amazonaws | aws-java-sdk-core | 1.12.390 |
| com.amazonaws | aws-java-sdk-datapipeline | 1.12.390 |
| com.amazonaws | aws-java-sdk-directconnect | 1.12.390 |
| com.amazonaws | aws-java-sdk-directory | 1.12.390 |
| com.amazonaws | aws-java-sdk-dynamodb | 1.12.390 |
| com.amazonaws | aws-java-sdk-ec2 | 1.12.390 |
| com.amazonaws | aws-java-sdk-ecs | 1.12.390 |
| com.amazonaws | aws-java-sdk-efs | 1.12.390 |
| com.amazonaws | aws-java-sdk-elasticache | 1.12.390 |
| com.amazonaws | aws-java-sdk-elasticbeanstalk | 1.12.390 |
| com.amazonaws | aws-java-sdk-elasticloadbalancing | 1.12.390 |
| com.amazonaws | aws-java-sdk-elastictranscoder | 1.12.390 |
| com.amazonaws | aws-java-sdk-emr | 1.12.390 |
| com.amazonaws | aws-java-sdk-glacier | 1.12.390 |
| com.amazonaws | aws-java-sdk-glue | 1.12.390 |
| com.amazonaws | aws-java-sdk-iam | 1.12.390 |
| com.amazonaws | aws-java-sdk-importexport | 1.12.390 |
| com.amazonaws | aws-java-sdk-kinesis | 1.12.390 |
| com.amazonaws | aws-java-sdk-kms | 1.12.390 |
| com.amazonaws | aws-java-sdk-lambda | 1.12.390 |
| com.amazonaws | aws-java-sdk-logs | 1.12.390 |
| com.amazonaws | aws-java-sdk-machinelearning | 1.12.390 |
| com.amazonaws | aws-java-sdk-opsworks | 1.12.390 |
| com.amazonaws | aws-java-sdk-rds | 1.12.390 |
| com.amazonaws | aws-java-sdk-redshift | 1.12.390 |
| com.amazonaws | aws-java-sdk-route53 | 1.12.390 |
| com.amazonaws | aws-java-sdk-s3 | 1.12.390 |
| com.amazonaws | aws-java-sdk-ses | 1.12.390 |
| com.amazonaws | aws-java-sdk-simpledb | 1.12.390 |
| com.amazonaws | aws-java-sdk-simpleworkflow | 1.12.390 |
| com.amazonaws | aws-java-sdk-sns | 1.12.390 |
| com.amazonaws | aws-java-sdk-sqs | 1.12.390 |
| com.amazonaws | aws-java-sdk-ssm | 1.12.390 |
| com.amazonaws | aws-java-sdk-storagegateway | 1.12.390 |
| com.amazonaws | aws-java-sdk-sts | 1.12.390 |
| com.amazonaws | aws-java-sdk-support | 1.12.390 |
| com.amazonaws | aws-java-sdk-swf-libraries | 1.11.22 |
| com.amazonaws | aws-java-sdk-workspaces | 1.12.390 |
| com.amazonaws | jmespath-java | 1.12.390 |
| com.clearspring.analytics | stream | 2.9.6 |
| com.databricks | Rserve | 1.8-3 |
| com.databricks | databricks-sdk-java | 0.7.0 |
| com.databricks | jets3t | 0.7.1-0 |
| com.databricks.scalapb | compilerplugin_2.12 | 0.4.15-10 |
| com.databricks.scalapb | scalapb-runtime_2.12 | 0.4.15-10 |
| com.esotericsoftware | kryo-shaded | 4.0.2 |
| com.esotericsoftware | minlog | 1.3.0 |
| com.fasterxml | classmate | 1.3.4 |
| com.fasterxml.jackson.core | jackson-annotations | 2.15.2 |
| com.fasterxml.jackson.core | jackson-core | 2.15.2 |
| com.fasterxml.jackson.core | jackson-databind | 2.15.2 |
| com.fasterxml.jackson.dataformat | jackson-dataformat-cbor | 2.15.2 |
| com.fasterxml.jackson.datatype | jackson-datatype-joda | 2.15.2 |
| com.fasterxml.jackson.datatype | jackson-datatype-jsr310 | 2.15.1 |
| com.fasterxml.jackson.module | jackson-module-paranamer | 2.15.2 |
| com.fasterxml.jackson.module | jackson-module-scala_2.12 | 2.15.2 |
| com.github.ben-manes.caffeine | caffeine | 2.9.3 |
| com.github.fommil | jniloader | 1.1 |
| com.github.fommil.netlib | native_ref-java | 1.1 |
| com.github.fommil.netlib | native_ref-java | 1.1-natives |
| com.github.fommil.netlib | native_system-java | 1.1 |
| com.github.fommil.netlib | native_system-java | 1.1-natives |
| com.github.fommil.netlib | netlib-native_ref-linux-x86_64 | 1.1-natives |
| com.github.fommil.netlib | netlib-native_system-linux-x86_64 | 1.1-natives |
| com.github.luben | zstd-jni | 1.5.5-4 |
| com.github.wendykierp | JTransforms | 3.1 |
| com.google.code.findbugs | jsr305 | 3.0.0 |
| com.google.code.gson | gson | 2.10.1 |
| com.google.crypto.tink | tink | 1.9.0 |
| com.google.errorprone | error_prone_annotations | 2.10.0 |
| com.google.flatbuffers | flatbuffers-java | 1.12.0 |
| com.google.guava | guava | 15.0 |
| com.google.protobuf | protobuf-java | 2.6.1 |
| com.helger | profiler | 1.1.1 |
| com.jcraft | jsch | 0.1.55 |
| com.jolbox | bonecp | 0.8.0.RELEASE |
| com.lihaoyi | sourcecode_2.12 | 0.1.9 |
| com.microsoft.azure | azure-data-lake-store-sdk | 2.3.9 |
| com.microsoft.sqlserver | mssql-jdbc | 11.2.2.jre8 |
| com.ning | compress-lzf | 1.1.2 |
| com.sun.mail | javax.mail | 1.5.2 |
| com.sun.xml.bind | jaxb-core | 2.2.11 |
| com.sun.xml.bind | jaxb-impl | 2.2.11 |
| com.tdunning | json | 1.8 |
| com.thoughtworks.paranamer | paranamer | 2.8 |
| com.trueaccord.lenses | lenses_2.12 | 0.4.12 |
| com.twitter | chill-java | 0.10.0 |
| com.twitter | chill_2.12 | 0.10.0 |
| com.twitter | util-app_2.12 | 7.1.0 |
| com.twitter | util-core_2.12 | 7.1.0 |
| com.twitter | util-function_2.12 | 7.1.0 |
| com.twitter | util-jvm_2.12 | 7.1.0 |
| com.twitter | util-lint_2.12 | 7.1.0 |
| com.twitter | util-registry_2.12 | 7.1.0 |
| com.twitter | util-stats_2.12 | 7.1.0 |
| com.typesafe | config | 1.2.1 |
| com.typesafe.scala-logging | scala-logging_2.12 | 3.7.2 |
| com.uber | h3 | 3.7.3 |
| com.univocity | univocity-parsers | 2.9.1 |
| com.zaxxer | HikariCP | 4.0.3 |
| commons-cli | commons-cli | 1.5.0 |
| commons-codec | commons-codec | 1.16.0 |
| commons-collections | commons-collections | 3.2.2 |
| commons-dbcp | commons-dbcp | 1.4 |
| commons-fileupload | commons-fileupload | 1.5 |
| commons-httpclient | commons-httpclient | 3.1 |
| commons-io | commons-io | 2.13.0 |
| commons-lang | commons-lang | 2.6 |
| commons-logging | commons-logging | 1.1.3 |
| commons-pool | commons-pool | 1.5.4 |
| dev.ludovic.netlib | arpack | 3.0.3 |
| dev.ludovic.netlib | blas | 3.0.3 |
| dev.ludovic.netlib | lapack | 3.0.3 |
| info.ganglia.gmetric4j | gmetric4j | 1.0.10 |
| io.airlift | aircompressor | 0.25 |
| io.delta | delta-sharing-spark_2.12 | 0.7.5 |
| io.dropwizard.metrics | metrics-annotation | 4.2.19 |
| io.dropwizard.metrics | metrics-core | 4.2.19 |
| io.dropwizard.metrics | metrics-graphite | 4.2.19 |
| io.dropwizard.metrics | metrics-healthchecks | 4.2.19 |
| io.dropwizard.metrics | metrics-jetty9 | 4.2.19 |
| io.dropwizard.metrics | metrics-jmx | 4.2.19 |
| io.dropwizard.metrics | metrics-json | 4.2.19 |
| io.dropwizard.metrics | metrics-jvm | 4.2.19 |
| io.dropwizard.metrics | metrics-servlets | 4.2.19 |
| io.netty | netty-all | 4.1.96.Final |
| io.netty | netty-buffer | 4.1.96.Final |
| io.netty | netty-codec | 4.1.96.Final |
| io.netty | netty-codec-http | 4.1.96.Final |
| io.netty | netty-codec-http2 | 4.1.96.Final |
| io.netty | netty-codec-socks | 4.1.96.Final |
| io.netty | netty-common | 4.1.96.Final |
| io.netty | netty-handler | 4.1.96.Final |
| io.netty | netty-handler-proxy | 4.1.96.Final |
| io.netty | netty-resolver | 4.1.96.Final |
| io.netty | netty-tcnative-boringssl-static | 2.0.61.Final-linux-aarch_64 |
| io.netty | netty-tcnative-boringssl-static | 2.0.61.Final-linux-x86_64 |
| io.netty | netty-tcnative-boringssl-static | 2.0.61.Final-osx-aarch_64 |
| io.netty | netty-tcnative-boringssl-static | 2.0.61.Final-osx-x86_64 |
| io.netty | netty-tcnative-boringssl-static | 2.0.61.Final-windows-x86_64 |
| io.netty | netty-tcnative-classes | 2.0.61.Final |
| io.netty | netty-transport | 4.1.96.Final |
| io.netty | netty-transport-classes-epoll | 4.1.96.Final |
| io.netty | netty-transport-classes-kqueue | 4.1.96.Final |
| io.netty | netty-transport-native-epoll | 4.1.96.Final |
| io.netty | netty-transport-native-epoll | 4.1.96.Final-linux-aarch_64 |
| io.netty | netty-transport-native-epoll | 4.1.96.Final-linux-x86_64 |
| io.netty | netty-transport-native-kqueue | 4.1.96.Final-osx-aarch_64 |
| io.netty | netty-transport-native-kqueue | 4.1.96.Final-osx-x86_64 |
| io.netty | netty-transport-native-unix-common | 4.1.96.Final |
| io.prometheus | simpleclient | 0.7.0 |
| io.prometheus | simpleclient_common | 0.7.0 |
| io.prometheus | simpleclient_dropwizard | 0.7.0 |
| io.prometheus | simpleclient_pushgateway | 0.7.0 |
| io.prometheus | simpleclient_servlet | 0.7.0 |
| io.prometheus.jmx | collector | 0.12.0 |
| jakarta.annotation | jakarta.annotation-api | 1.3.5 |
| jakarta.servlet | jakarta.servlet-api | 4.0.3 |
| jakarta.validation | jakarta.validation-api | 2.0.2 |
| jakarta.ws.rs | jakarta.ws.rs-api | 2.1.6 |
| javax.activation | activation | 1.1.1 |
| javax.el | javax.el-api | 2.2.4 |
| javax.jdo | jdo-api | 3.0.1 |
| javax.transaction | jta | 1.1 |
| javax.transaction | transaction-api | 1.1 |
| javax.xml.bind | jaxb-api | 2.2.11 |
| javolution | javolution | 5.5.1 |
| jline | jline | 2.14.6 |
| joda-time | joda-time | 2.12.1 |
| net.java.dev.jna | jna | 5.8.0 |
| net.razorvine | pickle | 1.3 |
| net.sf.jpam | jpam | 1.1 |
| net.sf.opencsv | opencsv | 2.3 |
| net.sf.supercsv | super-csv | 2.2.0 |
| net.snowflake | snowflake-ingest-sdk | 0.9.6 |
| net.snowflake | snowflake-jdbc | 3.13.33 |
| net.sourceforge.f2j | arpack_combined_all | 0.1 |
| org.acplt.remotetea | remotetea-oncrpc | 1.1.2 |
| org.antlr | ST4 | 4.0.4 |
| org.antlr | antlr-runtime | 3.5.2 |
| org.antlr | antlr4-runtime | 4.9.3 |
| org.antlr | stringtemplate | 3.2.1 |
| org.apache.ant | ant | 1.9.16 |
| org.apache.ant | ant-jsch | 1.9.16 |
| org.apache.ant | ant-launcher | 1.9.16 |
| org.apache.arrow | arrow-format | 12.0.1 |
| org.apache.arrow | arrow-memory-core | 12.0.1 |
| org.apache.arrow | arrow-memory-netty | 12.0.1 |
| org.apache.arrow | arrow-vector | 12.0.1 |
| org.apache.avro | avro | 1.11.2 |
| org.apache.avro | avro-ipc | 1.11.2 |
| org.apache.avro | avro-mapred | 1.11.2 |
| org.apache.commons | commons-collections4 | 4.4 |
| org.apache.commons | commons-compress | 1.23.0 |
| org.apache.commons | commons-crypto | 1.1.0 |
| org.apache.commons | commons-lang3 | 3.12.0 |
| org.apache.commons | commons-math3 | 3.6.1 |
| org.apache.commons | commons-text | 1.10.0 |
| org.apache.curator | curator-client | 2.13.0 |
| org.apache.curator | curator-framework | 2.13.0 |
| org.apache.curator | curator-recipes | 2.13.0 |
| org.apache.datasketches | datasketches-java | 3.1.0 |
| org.apache.datasketches | datasketches-memory | 2.0.0 |
| org.apache.derby | derby | 10.14.2.0 |
| org.apache.hadoop | hadoop-client-runtime | 3.3.6 |
| org.apache.hive | hive-beeline | 2.3.9 |
| org.apache.hive | hive-cli | 2.3.9 |
| org.apache.hive | hive-jdbc | 2.3.9 |
| org.apache.hive | hive-llap-client | 2.3.9 |
| org.apache.hive | hive-llap-common | 2.3.9 |
| org.apache.hive | hive-serde | 2.3.9 |
| org.apache.hive | hive-shims | 2.3.9 |
| org.apache.hive | hive-storage-api | 2.8.1 |
| org.apache.hive.shims | hive-shims-0.23 | 2.3.9 |
| org.apache.hive.shims | hive-shims-common | 2.3.9 |
| org.apache.hive.shims | hive-shims-scheduler | 2.3.9 |
| org.apache.httpcomponents | httpclient | 4.5.14 |
| org.apache.httpcomponents | httpcore | 4.4.16 |
| org.apache.ivy | ivy | 2.5.1 |
| org.apache.logging.log4j | log4j-1.2-api | 2.20.0 |
| org.apache.logging.log4j | log4j-api | 2.20.0 |
| org.apache.logging.log4j | log4j-core | 2.20.0 |
| org.apache.logging.log4j | log4j-slf4j2-impl | 2.20.0 |
| org.apache.mesos | mesos | 1.11.0-shaded-protobuf |
| org.apache.orc | orc-core | 1.9.1-shaded-protobuf |
| org.apache.orc | orc-mapreduce | 1.9.1-shaded-protobuf |
| org.apache.orc | orc-shims | 1.9.1 |
| org.apache.thrift | libfb303 | 0.9.3 |
| org.apache.thrift | libthrift | 0.12.0 |
| org.apache.ws.xmlschema | xmlschema-core | 2.3.0 |
| org.apache.xbean | xbean-asm9-shaded | 4.23 |
| org.apache.yetus | audience-annotations | 0.13.0 |
| org.apache.zookeeper | zookeeper | 3.6.3 |
| org.apache.zookeeper | zookeeper-jute | 3.6.3 |
| org.checkerframework | checker-qual | 3.31.0 |
| org.codehaus.jackson | jackson-core-asl | 1.9.13 |
| org.codehaus.jackson | jackson-mapper-asl | 1.9.13 |
| org.codehaus.janino | commons-compiler | 3.0.16 |
| org.codehaus.janino | janino | 3.0.16 |
| org.datanucleus | datanucleus-api-jdo | 4.2.4 |
| org.datanucleus | datanucleus-core | 4.1.17 |
| org.datanucleus | datanucleus-rdbms | 4.1.19 |
| org.datanucleus | javax.jdo | 3.2.0-m3 |
| org.eclipse.jetty | jetty-client | 9.4.52.v20230823 |
| org.eclipse.jetty | jetty-continuation | 9.4.52.v20230823 |
| org.eclipse.jetty | jetty-http | 9.4.52.v20230823 |
| org.eclipse.jetty | jetty-io | 9.4.52.v20230823 |
| org.eclipse.jetty | jetty-jndi | 9.4.52.v20230823 |
| org.eclipse.jetty | jetty-plus | 9.4.52.v20230823 |
| org.eclipse.jetty | jetty-proxy | 9.4.52.v20230823 |
| org.eclipse.jetty | jetty-security | 9.4.52.v20230823 |
| org.eclipse.jetty | jetty-server | 9.4.52.v20230823 |
| org.eclipse.jetty | jetty-servlet | 9.4.52.v20230823 |
| org.eclipse.jetty | jetty-servlets | 9.4.52.v20230823 |
| org.eclipse.jetty | jetty-util | 9.4.52.v20230823 |
| org.eclipse.jetty | jetty-util-ajax | 9.4.52.v20230823 |
| org.eclipse.jetty | jetty-webapp | 9.4.52.v20230823 |
| org.eclipse.jetty | jetty-xml | 9.4.52.v20230823 |
| org.eclipse.jetty.websocket | websocket-api | 9.4.52.v20230823 |
| org.eclipse.jetty.websocket | websocket-client | 9.4.52.v20230823 |
| org.eclipse.jetty.websocket | websocket-common | 9.4.52.v20230823 |
| org.eclipse.jetty.websocket | websocket-server | 9.4.52.v20230823 |
| org.eclipse.jetty.websocket | websocket-servlet | 9.4.52.v20230823 |
| org.fusesource.leveldbjni | leveldbjni-all | 1.8 |
| org.glassfish.hk2 | hk2-api | 2.6.1 |
| org.glassfish.hk2 | hk2-locator | 2.6.1 |
| org.glassfish.hk2 | hk2-utils | 2.6.1 |
| org.glassfish.hk2 | osgi-resource-locator | 1.0.3 |
| org.glassfish.hk2.external | aopalliance-repackaged | 2.6.1 |
| org.glassfish.hk2.external | jakarta.inject | 2.6.1 |
| org.glassfish.jersey.containers | jersey-container-servlet | 2.40 |
| org.glassfish.jersey.containers | jersey-container-servlet-core | 2.40 |
| org.glassfish.jersey.core | jersey-client | 2.40 |
| org.glassfish.jersey.core | jersey-common | 2.40 |
| org.glassfish.jersey.core | jersey-server | 2.40 |
| org.glassfish.jersey.inject | jersey-hk2 | 2.40 |
| org.hibernate.validator | hibernate-validator | 6.1.7.Final |
| org.ini4j | ini4j | 0.5.4 |
| org.javassist | javassist | 3.29.2-GA |
| org.jboss.logging | jboss-logging | 3.3.2.Final |
| org.jdbi | jdbi | 2.63.1 |
| org.jetbrains | annotations | 17.0.0 |
| org.joda | joda-convert | 1.7 |
| org.jodd | jodd-core | 3.5.2 |
| org.json4s | json4s-ast_2.12 | 3.7.0-M11 |
| org.json4s | json4s-core_2.12 | 3.7.0-M11 |
| org.json4s | json4s-jackson_2.12 | 3.7.0-M11 |
| org.json4s | json4s-scalap_2.12 | 3.7.0-M11 |
| org.lz4 | lz4-java | 1.8.0 |
| org.mariadb.jdbc | mariadb-java-client | 2.7.9 |
| org.mlflow | mlflow-spark | 2.2.0 |
| org.objenesis | objenesis | 2.5.1 |
| org.postgresql | postgresql | 42.6.0 |
| org.roaringbitmap | RoaringBitmap | 0.9.45 |
| org.roaringbitmap | shims | 0.9.45 |
| org.rocksdb | rocksdbjni | 8.3.2 |
| org.rosuda.REngine | REngine | 2.1.0 |
| org.scala-lang | scala-compiler_2.12 | 2.12.15 |
| org.scala-lang | scala-library_2.12 | 2.12.15 |
| org.scala-lang | scala-reflect_2.12 | 2.12.15 |
| org.scala-lang.modules | scala-collection-compat_2.12 | 2.9.0 |
| org.scala-lang.modules | scala-parser-combinators_2.12 | 1.1.2 |
| org.scala-lang.modules | scala-xml_2.12 | 1.2.0 |
| org.scala-sbt | test-interface | 1.0 |
| org.scalacheck | scalacheck_2.12 | 1.14.2 |
| org.scalactic | scalactic_2.12 | 3.2.15 |
| org.scalanlp | breeze-macros_2.12 | 2.1.0 |
| org.scalanlp | breeze_2.12 | 2.1.0 |
| org.scalatest | scalatest-compatible | 3.2.15 |
| org.scalatest | scalatest-core_2.12 | 3.2.15 |
| org.scalatest | scalatest-diagrams_2.12 | 3.2.15 |
| org.scalatest | scalatest-featurespec_2.12 | 3.2.15 |
| org.scalatest | scalatest-flatspec_2.12 | 3.2.15 |
| org.scalatest | scalatest-freespec_2.12 | 3.2.15 |
| org.scalatest | scalatest-funspec_2.12 | 3.2.15 |
| org.scalatest | scalatest-funsuite_2.12 | 3.2.15 |
| org.scalatest | scalatest-matchers-core_2.12 | 3.2.15 |
| org.scalatest | scalatest-mustmatchers_2.12 | 3.2.15 |
| org.scalatest | scalatest-propspec_2.12 | 3.2.15 |
| org.scalatest | scalatest-refspec_2.12 | 3.2.15 |
| org.scalatest | scalatest-shouldmatchers_2.12 | 3.2.15 |
| org.scalatest | scalatest-wordspec_2.12 | 3.2.15 |
| org.scalatest | scalatest_2.12 | 3.2.15 |
| org.slf4j | jcl-over-slf4j | 2.0.7 |
| org.slf4j | jul-to-slf4j | 2.0.7 |
| org.slf4j | slf4j-api | 2.0.7 |
| org.threeten | threeten-extra | 1.7.1 |
| org.tukaani | xz | 1.9 |
| org.typelevel | algebra_2.12 | 2.0.1 |
| org.typelevel | cats-kernel_2.12 | 2.1.1 |
| org.typelevel | spire-macros_2.12 | 0.17.0 |
| org.typelevel | spire-platform_2.12 | 0.17.0 |
| org.typelevel | spire-util_2.12 | 0.17.0 |
| org.typelevel | spire_2.12 | 0.17.0 |
| org.wildfly.openssl | wildfly-openssl | 1.1.3.Final |
| org.xerial | sqlite-jdbc | 3.42.0.0 |
| org.xerial.snappy | snappy-java | 1.1.10.3 |
| org.yaml | snakeyaml | 2.0 |
| oro | oro | 2.0.8 |
| pl.edu.icm | JLargeArrays | 1.5 |
| software.amazon.cryptools | AmazonCorrettoCryptoProvider | 1.6.1-linux-x86_64 |
| software.amazon.ion | ion-java | 1.0.2 |
| stax | stax-api | 1.0.1 |