Databricks Runtime 12.2 LTS

The following release notes provide information about Databricks Runtime 12.2 LTS, powered by Apache Spark 3.3.2.

Databricks released this version in March, 2023.

Behavioral changes

[Breaking change] New Python version requires updating Databricks Connect V1 Python clients

To apply required security patches, the Python version in Databricks Runtime 12.2 LTS is upgraded from 3.9.5 to 3.9.19. Because these changes might cause errors in clients that use specific PySpark functions, any clients that use Databricks Connect V1 for Python with Databricks Runtime 12.2 LTS must be updated to Python 3.9.7 or later.

New features and improvements

• Delta Lake schema evolution supports specifying source columns in merge statements
• Structured Streaming workloads are supported on clusters with shared access mode
• New features for Predictive I/O
• Implicit lateral column aliasing support
• New forEachBatch feature
• Standardized Connection Options for Query Federation
• Extended SQL function library for array management
• New mask function to anonymize strings
• Common error conditions now return SQLSTATEs
• Invoke generator functions in the FROM clause
• Support for protocol buffers is generally available
• Go-to-definition for notebook variables and functions
• Notebook quick-fix for auto-importing libraries
• Bug fixes

Delta Lake schema evolution supports specifying source columns in merge statements

You can now specify columns present only in the source table in insert or update actions for merge statements when schema evolution is enabled. In Databricks Runtime 12.1 and below, only INSERT * or UPDATE SET * actions can be used for schema evolution with merge. See Automatic schema evolution for Delta Lake merge.

Structured Streaming workloads are supported on clusters with shared access mode

You can now use Structured Streaming to interact with Unity Catalog on shared clusters. Some limitations apply. See What Structured Streaming functionality does Unity Catalog support?.

New features for Predictive I/O

Photon support for the Foreachbatch sink is now available. Workloads that stream from a source and merge into Delta Tables or write to multiple sinks can now benefit from Photonized Foreachbatch sink.

Implicit lateral column aliasing support

Azure Databricks now supports implicit lateral column aliasing by default. You can now reuse an expression specified earlier in the same SELECT list. For example, given SELECT 1 AS a, a + 1 AS b, the a in a + 1 can be resolved as the previously defined 1 AS a. Check Name resolution for more details in resolution order. To turn off this feature, you can set spark.sql.lateralColumnAlias.enableImplicitResolution to false.

New forEachBatch feature

Photon is now supported when using foreachBatch to write to a data sink.

Standardized Connection Options for Query Federation

You can now use a unified set of options (host, port, database, user, password)to connect to data sources supported in Query Federation. Port is optional and uses the default port number for each data source, unless specified.

Extended SQL function library for array management

You can now remove all NULL elements from an array using array_compact. To append elements to an array, use array_append.

New mask function to anonymize strings

Invoke the mask function to anonymize sensitive string values.

Common error conditions now return SQLSTATEs

Most error conditions that Databricks Runtime now include documented SQLSTATE values which can be used to test for errors in a SQL standard compliant way.

Invoke generator functions in the FROM clause

You can now invoke table-valued generator functions such as explode in the regular FROM clause of a query. This aligns generator function invocation with other built-in and user defined table functions.

Support for protocol buffers is generally available

You can use the from_protobuf and to_protobuf functions to exchange data between binary and struct types. See Read and write protocol buffers.

Go-to-definition for notebook variables and functions

In notebooks, you can quickly go to the definition of a variable, function, or the code behind a %run statement by right-clicking the variable or function name.

Notebook quick-fix for auto-importing libraries

Databricks Notebooks now offer a quick-fix feature for auto-importing libraries. If you forget to import a library like pandas, hover over the underlined syntax warning, and then click Quick Fix, this features requires the Databricks Assistant to be enabled in your Workspace.

Bug fixes

• Improved consistency for Delta commit behavior for empty transactions relating to update, delete, and merge commands. In WriteSerializable isolation level, commands that result in no changes now create an empty commit. In a Serializable isolation level, such empty transactions now do not create a commit.

Behavior changes

Behavior changes with the new lateral column alias feature

The new lateral column alias feature introduces behavior changes for the following cases during name resolution:

• Lateral column alias now takes precedence over correlated references with the same name. For example, for this query SELECT (SELECT c2 FROM (SELECT 1 AS c1, c1 AS c2) WHERE c2 > 5) FROM VALUES(6) AS t(c1), the c1 in the inner c1 AS c2 was resolved to the correlated reference t.c1, but now changes to lateral column alias 1 AS c1. The query now returns NULL.
• Lateral column alias now takes precedence over function parameters with the same name. For example, for the function, CREATE OR REPLACE TEMPORARY FUNCTION func(x INT) RETURNS TABLE (a INT, b INT, c DOUBLE) RETURN SELECT x + 1 AS x, x, the x in the function body was resolved to the function parameter x, but changes to lateral column alias x + 1 in the function body. The query, SELECT * FROM func(1) now returns 2, 2.
• To turn off the lateral column alias feature, set spark.sql.lateralColumnAlias.enableImplicitResolution to false. For more information, see Name resolution.

Library upgrades

• Upgraded Python libraries:
  • filelock from 3.8.2 to 3.9.0
  • joblib from 1.1.0 to 1.1.1
  • platformdirs from 2.6.0 to 2.6.2
  • whatthepatch from 1.0.3 to 1.0.4
• Upgraded R libraries:
  • class from 7.3-20 to 7.3-21
  • codetools from 0.2-18 to 0.2-19
  • MASS from 7.3-58 to 7.3-58.2
  • nlme from 3.1-160 to 3.1-162
  • Rserve from 1.8-11 to 1.8-12
  • SparkR from 3.3.1 to 3.3.2

Behavior changes

• Users are now required to have SELECT and MODIFY privileges on any file when creating a schema with a defined location.

Apache Spark

Databricks Runtime 12.2 includes Apache Spark 3.3.2. This release includes all Spark fixes and improvements included in Databricks Runtime 12.1 (EoS), as well as the following additional bug fixes and improvements made to Spark:

• [SPARK-42416] [SC-123205][SC-122851][SQL] Dateset operations should not resolve the analyzed logical plan again
• [SPARK-41848] Revert "[CHERRY-PICK][12.x][12.1][12.0][SC-120037][CORE] Fixing task over-scheduled with TaskResourceProfile"
• [SPARK-42162] [SC-122711][ES-556261] Introduce MultiCommutativeOp expression as a memory optimization for canonicalizing large trees of commutative expressions
• [SPARK-42406] [SC-122998][PROTOBUF][Cherry-pick] Fix recursive depth setting for Protobuf functions
• [SPARK-42002] [SC-122476][CONNECT][PYTHON] Implement DataFrameWriterV2
• [SPARK-41716] [SC-122545][CONNECT] Rename _catalog_to_pandas to _execute_and_fetch in Catalog
• [SPARK-41490] [SC-121774][SQL] Assign name to _LEGACY_ERROR_TEMP_2441
• [SPARK-41600] [SC-122538][SPARK-41623][SPARK-41612][CONNECT] Implement Catalog.cacheTable, isCached and uncache
• [SPARK-42191] [SC-121990][SQL] Support udf 'luhn_check'
• [SPARK-42253] [SC-121976][PYTHON] Add test for detecting duplicated error class
• [SPARK-42268] [SC-122251][CONNECT][PYTHON] Add UserDefinedType in protos
• [SPARK-42231] [SC-121841][SQL] Turn MISSING_STATIC_PARTITION_COLUMN into internalError
• [SPARK-42136] [SC-122554] Refactor BroadcastHashJoinExec output partitioning calculation
• [SPARK-42158] [SC-121610][SQL] Integrate _LEGACY_ERROR_TEMP_1003 into FIELD_NOT_FOUND
• [SPARK-42192] [12.x][SC-121820][PYTHON] Migrate the TypeError from pyspark/sql/dataframe.py into PySparkTypeError
• [SPARK-35240] Revert "[SC-118242][SS] Use CheckpointFileManager …
• [SPARK-41488] [SC-121858][SQL] Assign name to _LEGACY_ERROR_TEMP_1176 (and 1177)
• [SPARK-42232] [SC-122267][SQL] Rename error class: UNSUPPORTED_FEATURE.JDBC_TRANSACTION
• [SPARK-42346] [SC-122480][SQL] Rewrite distinct aggregates after subquery merge
• [SPARK-42306] [SC-122539][SQL] Integrate _LEGACY_ERROR_TEMP_1317 into UNRESOLVED_COLUMN.WITH_SUGGESTION
• [SPARK-42234] [SC-122354][SQL] Rename error class: UNSUPPORTED_FEATURE.REPEATED_PIVOT
• [SPARK-42343] [SC-122437][CORE] Ignore IOException in handleBlockRemovalFailure if SparkContext is stopped
• [SPARK-41295] [SC-122442][SPARK-41296][SQL] Rename the error classes
• [SPARK-42320] [SC-122478][SQL] Assign name to _LEGACY_ERROR_TEMP_2188
• [SPARK-42255] [SC-122483][SQL] Assign name to _LEGACY_ERROR_TEMP_2430
• [SPARK-42156] [SC-121851][CONNECT] SparkConnectClient supports RetryPolicies now
• [SPARK-38728] [SC-116723][SQL] Test the error class: FAILED_RENAME_PATH
• [SPARK-40005] [12.X] Self contained examples in PySpark
• [SPARK-39347] [SC-122457][SS] Bug fix for time window calculation when event time < 0
• [SPARK-42336] [SC-122458][CORE] Use getOrElse() instead of contains() in ResourceAllocator
• [SPARK-42125] [SC-121827][CONNECT][PYTHON] Pandas UDF in Spark Connect
• [SPARK-42217] [SC-122263][SQL] Support implicit lateral column alias in queries with Window
• [SPARK-35240] [SC-118242][SS] Use CheckpointFileManager for checkpoint file manipulation
• [SPARK-42294] [SC-122337][SQL] Include column default values in DESCRIBE output for V2 tables
• [SPARK-41979] Revert "Revert "[12.x][SC-121190][SQL] Add missing dots for error messages in error classes.""
• [SPARK-42286] [SC-122336][SQL] Fallback to previous codegen code path for complex expr with CAST
• [SPARK-42275] [SC-122249][CONNECT][PYTHON] Avoid using built-in list, dict in static typing
• [SPARK-41985] [SC-122172][SQL] Centralize more column resolution rules
• [SPARK-42126] [SC-122330][PYTHON][CONNECT] Accept return type in DDL strings for Python Scalar UDFs in Spark Connect
• [SPARK-42197] [SC-122328][SC-121514][CONNECT] Reuses JVM initialization, and separate configuration groups to set in remote local mode
• [SPARK-41575] [SC-120118][SQL] Assign name to _LEGACY_ERROR_TEMP_2054
• [SPARK-41985] Revert "[SC-122172][SQL] Centralize more column resolution rules"
• [SPARK-42123] [SC-122234][SC-121453][SQL] Include column default values in DESCRIBE and SHOW CREATE TABLE output
• [SPARK-41985] [SC-122172][SQL] Centralize more column resolution rules
• [SPARK-42284] [SC-122233][CONNECT] Make sure connect server assembly is built before running client tests - SBT
• [SPARK-42239] [SC-121790][SQL] Integrate MUST_AGGREGATE_CORRELATED_SCALAR_SUBQUERY
• [SPARK-42278] [SC-122170][SQL] DS V2 pushdown supports supports JDBC dialects compile SortOrder by themselves
• [SPARK-42259] [SC-122168][SQL] ResolveGroupingAnalytics should take care of Python UDAF
• [SPARK-41979] Revert "[12.x][SC-121190][SQL] Add missing dots for error messages in error classes."
• [SPARK-42224] [12.x][SC-121708][CONNECT] Migrate TypeError into error framework for Spark Connect functions
• [SPARK-41712] [12.x][SC-121189][PYTHON][CONNECT] Migrate the Spark Connect errors into PySpark error framework.
• [SPARK-42119] [SC-121913][SC-121342][SQL] Add built-in table-valued functions inline and inline_outer
• [SPARK-41489] [SC-121713][SQL] Assign name to _LEGACY_ERROR_TEMP_2415
• [SPARK-42082] [12.x][SC-121163][SPARK-41598][PYTHON][CONNECT] Introduce PySparkValueError and PySparkTypeError
• [SPARK-42081] [SC-121723][SQL] Improve the plan change validation
• [SPARK-42225] [12.x][SC-121714][CONNECT] Add SparkConnectIllegalArgumentException to handle Spark Connect error precisely.
• [SPARK-42044] [12.x][SC-121280][SQL] Fix incorrect error message for MUST_AGGREGATE_CORRELATED_SCALAR_SUBQUERY
• [SPARK-42194] [12.x][SC-121712][PS] Allow columns parameter when creating DataFrame with Series.
• [SPARK-42078] [12.x][SC-120761][PYTHON] Migrate errors thrown by JVM into PySparkException.
• [SPARK-42133] [12.x][SC-121250] Add basic Dataset API methods to SparkConnect Scala Client
• [SPARK-41979] [12.x][SC-121190][SQL] Add missing dots for error messages in error classes.
• [SPARK-42124] [12.x][SC-121420][PYTHON][CONNECT] Scalar Inline Python UDF in Spark Connect
• [SPARK-42051] [SC-121994][SQL] Codegen Support for HiveGenericUDF
• [SPARK-42257] [SC-121948][CORE] Remove unused variable external sorter
• [SPARK-41735] [SC-121771][SQL] Use MINIMAL instead of STANDARD for SparkListenerSQLExecutionEnd
• [SPARK-42236] [SC-121882][SQL] Refine NULLABLE_ARRAY_OR_MAP_ELEMENT
• [SPARK-42233] [SC-121775][SQL] Improve error message for PIVOT_AFTER_GROUP_BY
• [SPARK-42229] [SC-121856][CORE] Migrate SparkCoreErrors into error classes
• [SPARK-42163] [SC-121839][SQL] Fix schema pruning for non-foldable array index or map key
• [SPARK-40711] [SC-119990][SQL] Add spill size metrics for window
• [SPARK-42023] [SC-121847][SPARK-42024][CONNECT][PYTHON] Make createDataFrame support AtomicType -> StringType coercion
• [SPARK-42202] [SC-121837][Connect][Test] Improve the E2E test server stop logic
• [SPARK-41167] [SC-117425][SQL] Improve multi like performance by creating a balanced expression tree predicate
• [SPARK-41931] [SC-121618][SQL] Better error message for incomplete complex type definition
• [SPARK-36124] [SC-121339][SC-110446][SQL] Support subqueries with correlation through UNION
• [SPARK-42090] [SC-121290][3.3] Introduce sasl retry count in RetryingBlockTransferor
• [SPARK-42157] [SC-121264][CORE] spark.scheduler.mode=FAIR should provide FAIR scheduler
• [SPARK-41572] [SC-120772][SQL] Assign name to _LEGACY_ERROR_TEMP_2149
• [SPARK-41983] [SC-121224][SQL] Rename & improve error message for NULL_COMPARISON_RESULT
• [SPARK-41976] [SC-121024][SQL] Improve error message for INDEX_NOT_FOUND
• [SPARK-41994] [SC-121210][SC-120573] Assign SQLSTATE's (1/2)
• [SPARK-41415] [SC-121117][3.3] SASL Request Retries
• [SPARK-38591] [SC-121018][SQL] Add flatMapSortedGroups and cogroupSorted
• [SPARK-41975] [SC-120767][SQL] Improve error message for INDEX_ALREADY_EXISTS
• [SPARK-42056] [SC-121158][SQL][PROTOBUF] Add missing options for Protobuf functions
• [SPARK-41984] [SC-120769][SQL] Rename & improve error message for RESET_PERMISSION_TO_ORIGINAL
• [SPARK-41948] [SC-121196][SQL] Fix NPE for error classes: CANNOT_PARSE_JSON_FIELD
• [SPARK-41772] [SC-121176][CONNECT][PYTHON] Fix incorrect column name in withField's doctest
• [SPARK-41283] [SC-121175][CONNECT][PYTHON] Add array_append to Connect
• [SPARK-41960] [SC-120773][SQL] Assign name to _LEGACY_ERROR_TEMP_1056
• [SPARK-42134] [SC-121116][SQL] Fix getPartitionFiltersAndDataFilters() to handle filters without referenced attributes
• [SPARK-42096] [SC-121012][CONNECT] Some code cleanup for connect module
• [SPARK-42099] [SC-121114][SPARK-41845][CONNECT][PYTHON] Fix count(*) and count(col(*))
• [SPARK-42045] [SC-120958][SC-120450][SQL] ANSI SQL mode: Round/Bround should return an error on integer overflow
• [SPARK-42043] [SC-120968][CONNECT] Scala Client Result with E2E Tests
• [SPARK-41884] [SC-121022][CONNECT] Support naive tuple as a nested row
• [SPARK-42112] [SC-121011][SQL][SS] Add null check before ContinuousWriteRDD#compute function close dataWriter
• [SPARK-42077] [SC-120553][CONNECT][PYTHON] Literal should throw TypeError for unsupported DataType
• [SPARK-42108] [SC-120898][SQL] Make Analyzer transform Count(*) into Count(1)
• [SPARK-41666] [SC-120928][SC-119009][PYTHON] Support parameterized SQL by sql()
• [SPARK-40599] [SC-120930][SQL] Relax multiTransform rule type to allow alternatives to be any kinds of Seq
• [SPARK-41574] [SC-120771][SQL] Update _LEGACY_ERROR_TEMP_2009 as INTERNAL_ERROR.
• [SPARK-41579] [SC-120770][SQL] Assign name to _LEGACY_ERROR_TEMP_1249
• [SPARK-41974] [SC-120766][SQL] Turn INCORRECT_END_OFFSET into INTERNAL_ERROR
• [SPARK-41530] [SC-120916][SC-118513][CORE] Rename MedianHeap to PercentileMap and support percentile
• [SPARK-41757] [SC-120608][SPARK-41901][CONNECT] Fix string representation for Column class
• [SPARK-42084] [SC-120775][SQL] Avoid leaking the qualified-access-only restriction
• [SPARK-41973] [SC-120765][SQL] Assign name to _LEGACY_ERROR_TEMP_1311
• [SPARK-42039] [SC-120655][SQL] SPJ: Remove Option in KeyGroupedPartitioning#partitionValuesOpt
• [SPARK-42079] [SC-120712][CONNECT][PYTHON] Rename proto messages for toDF and withColumnsRenamed
• [SPARK-42089] [SC-120605][CONNECT][PYTHON] Fix variable name issues in nested lambda functions
• [SPARK-41982] [SC-120604][SQL] Partitions of type string should not be treated as numeric types
• [SPARK-40599] [SC-120620][SQL] Add multiTransform methods to TreeNode to generate alternatives
• [SPARK-42085] [SC-120556][CONNECT][PYTHON] Make from_arrow_schema support nested types
• [SPARK-42057] [SC-120507][SQL][PROTOBUF] Fix how exception is handled in error reporting.
• [SPARK-41586] [12.x][ALL TESTS][SC-120544][PYTHON] Introduce pyspark.errors and error classes for PySpark.
• [SPARK-41903] [SC-120543][CONNECT][PYTHON] Literal should support 1-dim ndarray
• [SPARK-42021] [SC-120584][CONNECT][PYTHON] Make createDataFrame support array.array
• [SPARK-41896] [SC-120506][SQL] Filtering by row index returns empty results
• [SPARK-41162] [SC-119742][SQL] Fix anti- and semi-join for self-join with aggregations
• [SPARK-41961] [SC-120501][SQL] Support table-valued functions with LATERAL
• [SPARK-41752] [SC-120550][SQL][UI] Group nested executions under the root execution
• [SPARK-42047] [SC-120586][SPARK-41900][CONNECT][PYTHON][12.X] Literal should support Numpy datatypes
• [SPARK-42028] [SC-120344][CONNECT][PYTHON] Truncating nanoseconds timestampsl
• [SPARK-42011] [SC-120534][CONNECT][PYTHON] Implement DataFrameReader.csv
• [SPARK-41990] [SC-120532][SQL] Use FieldReference.column instead of apply in V1 to V2 filter conversion
• [SPARK-39217] [SC-120446][SQL] Makes DPP support the pruning side has Union
• [SPARK-42076] [SC-120551][CONNECT][PYTHON] Factor data conversion arrow -> rows out to conversion.py
• [SPARK-42074] [SC-120540][SQL] Enable KryoSerializer in TPCDSQueryBenchmark to enforce SQL class registration
• [SPARK-42012] [SC-120517][CONNECT][PYTHON] Implement DataFrameReader.orc
• [SPARK-41832] [SC-120513][CONNECT][PYTHON] Fix DataFrame.unionByName, add allow_missing_columns
• [SPARK-38651] [SC-120514] [SQL] Add spark.sql.legacy.allowEmptySchemaWrite
• [SPARK-41991] [SC-120406][SQL] CheckOverflowInTableInsert should accept ExpressionProxy as child
• [SPARK-41232] [SC-120073][SQL][PYTHON] Adding array_append function
• [SPARK-42041] [SC-120512][SPARK-42013][CONNECT][PYTHON] DataFrameReader should support list of paths
• [SPARK-42071] [SC-120533][CORE] Register scala.math.Ordering$Reverse to KyroSerializer
• [SPARK-41986] [SC-120429][SQL] Introduce shuffle on SinglePartition
• [SPARK-42016] [SC-120428][CONNECT][PYTHON] Enable tests related to the nested column
• [SPARK-42042] [SC-120427][CONNECT][PYTHON] DataFrameReader should support StructType schema
• [SPARK-42031] [SC-120389][CORE][SQL] Clean up remove methods that do not need override
• [SPARK-41746] [SC-120463][SPARK-41838][SPARK-41837][SPARK-41835][SPARK-41836][SPARK-41847][CONNECT][PYTHON] Make createDataFrame(rows/lists/tuples/dicts) support nested types
• [SPARK-41437] [SC-117601][SQL][ALL TESTS] Do not optimize the input query twice for v1 write fallback
• [SPARK-41840] [SC-119719][CONNECT][PYTHON] Add the missing alias groupby
• [SPARK-41846] [SC-119717][CONNECT][PYTHON] Enable doctests for window functions
• [SPARK-41914] [SC-120094][SQL] FileFormatWriter materializes AQE plan before accessing outputOrdering
• [SPARK-41805] [SC-119992][SQL] Reuse expressions in WindowSpecDefinition
• [SPARK-41977] [SC-120269][SPARK-41978][CONNECT] SparkSession.range to take float as arguments
• [SPARK-42029] [SC-120336][CONNECT] Add Guava Shading rules to connect-common to avoid startup failure
• [SPARK-41989] [SC-120334][PYTHON] Avoid breaking logging config from pyspark.pandas
• [SPARK-42003] [SC-120331][SQL] Reduce duplicate code in ResolveGroupByAll
• [SPARK-41635] [SC-120313][SQL] Fix group by all error reporting
• [SPARK-41047] [SC-120291][SQL] Improve docs for round
• [SPARK-41822] [SC-120122][CONNECT] Setup gRPC connection for Scala/JVM client
• [SPARK-41879] [SC-120264][CONNECT][PYTHON] Make DataFrame.collect support nested types
• [SPARK-41887] [SC-120268][CONNECT][PYTHON] Make DataFrame.hint accept list typed parameter
• [SPARK-41964] [SC-120210][CONNECT][PYTHON] Add the list of unsupported IO functions
• [SPARK-41595] [SC-120097][SQL] Support generator function explode/explode_outer in the FROM clause
• [SPARK-41957] [SC-120121][CONNECT][PYTHON] Enable the doctest for DataFrame.hint
• [SPARK-41886] [SC-120141][CONNECT][PYTHON] DataFrame.intersect doctest output has different order
• [SPARK-41442] [SC-117795][SQL][ALL TESTS] Only update SQLMetric value if merging with valid metric
• [SPARK-41944] [SC-120046][CONNECT] Pass configurations when local remote mode is on
• [SPARK-41708] [SC-119838][SQL] Pull v1write information to WriteFiles
• [SPARK-41780] [SC-120000][SQL] Should throw INVALID_PARAMETER_VALUE.PATTERN when the parameters regexp is invalid
• [SPARK-41889] [SC-119975][SQL] Attach root cause to invalidPatternError & refactor error classes INVALID_PARAMETER_VALUE
• [SPARK-41860] [SC-120028][SQL] Make AvroScanBuilder and JsonScanBuilder case classes
• [SPARK-41945] [SC-120010][CONNECT][PYTHON] Python: connect client lost column data with pyarrow.Table.to_pylist
• [SPARK-41690] [SC-119102][SC-119087][SQL][CONNECT] Agnostic Encoders
• [SPARK-41354] [SC-119995][CONNECT][PYTHON] Implement RepartitionByExpression
• [SPARK-41581] [SC-119997][SQL] Update _LEGACY_ERROR_TEMP_1230 as INTERNAL_ERROR
• [SPARK-41928] [SC-119972][CONNECT][PYTHON] Add the unsupported list for functions
• [SPARK-41933] [SC-119980][CONNECT] Provide local mode that automatically starts the server
• [SPARK-41899] [SC-119971][CONNECT][PYTHON] createDataFrame` should respect user provided DDL schema
• [SPARK-41936] [SC-119978][CONNECT][PYTHON] Make withMetadata reuse the withColumns proto
• [SPARK-41898] [SC-119931][CONNECT][PYTHON] Window.rowsBetween, Window.rangeBetween parameters typechecking parity with pyspark
• [SPARK-41939] [SC-119977][CONNECT][PYTHON] Add the unsupported list for catalog functions
• [SPARK-41924] [SC-119946][CONNECT][PYTHON] Make StructType support metadata and Implement DataFrame.withMetadata
• [SPARK-41934] [SC-119967][CONNECT][PYTHON] Add the unsupported function list for session
• [SPARK-41875] [SC-119969][CONNECT][PYTHON] Add test cases for Dataset.to()
• [SPARK-41824] [SC-119970][CONNECT][PYTHON] Ingore the doctest for explain of connect
• [SPARK-41880] [SC-119959][CONNECT][PYTHON] Make function from_json accept non-literal schema
• [SPARK-41927] [SC-119952][CONNECT][PYTHON] Add the unsupported list for GroupedData
• [SPARK-41929] [SC-119949][CONNECT][PYTHON] Add function array_compact
• [SPARK-41827] [SC-119841][CONNECT][PYTHON] Make GroupBy accept column list
• [SPARK-41925] [SC-119905][SQL] Enable spark.sql.orc.enableNestedColumnVectorizedReader by default
• [SPARK-41831] [SC-119853][CONNECT][PYTHON] Make DataFrame.select accept column list
• [SPARK-41455] [SC-119858][CONNECT][PYTHON] Make DataFrame.collect discard the timezone info
• [SPARK-41923] [SC-119861][CONNECT][PYTHON] Add DataFrame.writeTo to the unsupported list
• [SPARK-41912] [SC-119837][SQL] Subquery should not validate CTE
• [SPARK-41828] [SC-119832][CONNECT][PYTHON][12.X] Make createDataFrame support empty dataframe
• [SPARK-41905] [SC-119848][CONNECT] Support name as strings in slice
• [SPARK-41869] [SC-119845][CONNECT] Reject single string in dropDuplicates
• [SPARK-41830] [SC-119840][CONNECT][PYTHON] Make DataFrame.sample accept the same parameters as PySpark
• [SPARK-41849] [SC-119835][CONNECT] Implement DataFrameReader.text
• [SPARK-41861] [SC-119834][SQL] Make v2 ScanBuilders' build() return typed scan
• [SPARK-41825] [SC-119710][CONNECT][PYTHON] Enable doctests related to DataFrame.show
• [SPARK-41855] [SC-119804][SC-119410][SPARK-41814][SPARK-41851][SPARK-41852][CONNECT][PYTHON][12.X] Make createDataFrame handle None/NaN properly
• [SPARK-41833] [SC-119685][SPARK-41881][SPARK-41815][CONNECT][PYTHON] Make DataFrame.collect handle None/NaN/Array/Binary porperly
• [SPARK-39318] [SC-119713][SQL] Remove tpch-plan-stability WithStats golden files
• [SPARK-41791] [SC-119745] Add new file source metadata column types
• [SPARK-41790] [SC-119729][SQL] Set TRANSFORM reader and writer's format correctly
• [SPARK-41829] [SC-119725][CONNECT][PYTHON] Add the missing ordering parameter in Sort and sortWithinPartitions
• [SPARK-41576] [SC-119718][SQL] Assign name to _LEGACY_ERROR_TEMP_2051
• [SPARK-41821] [SC-119716][CONNECT][PYTHON] Fix doc test for DataFrame.describe
• [SPARK-41871] [SC-119714][CONNECT] DataFrame hint parameter can be str, float or int
• [SPARK-41720] [SC-119076][SQL] Rename UnresolvedFunc to UnresolvedFunctionName
• [SPARK-41573] [SC-119567][SQL] Assign name to _LEGACY_ERROR_TEMP_2136
• [SPARK-41862] [SC-119492][SQL] Fix correctness bug related to DEFAULT values in Orc reader
• [SPARK-41582] [SC-119482][SC-118701][CORE][SQL] Reuse INVALID_TYPED_LITERAL instead of _LEGACY_ERROR_TEMP_0022

Maintenance updates

See Databricks Runtime 12.2 maintenance updates.

System environment

• Operating System: Ubuntu 20.04.5 LTS
• Java: Zulu 8.68.0.21-CA-linux64
• Scala: 2.12.15
• Python: 3.9.19
• R: 4.2.2
• Delta Lake: 2.2.0

Installed Python libraries

Library	Version	Library	Version	Library	Version
argon2-cffi	21.3.0	argon2-cffi-bindings	21.2.0	asttokens	2.0.5
attrs	21.4.0	backcall	0.2.0	backports.entry-points-selectable	1.2.0
beautifulsoup4	4.11.1	black	22.3.0	bleach	4.1.0
boto3	1.21.32	botocore	1.24.32	certifi	2021.10.8
cffi	1.15.0	chardet	4.0.0	charset-normalizer	2.0.4
click	8.0.4	cryptography	3.4.8	cycler	0.11.0
Cython	0.29.28	dbus-python	1.2.16	debugpy	1.5.1
decorator	5.1.1	defusedxml	0.7.1	distlib	0.3.6
docstring-to-markdown	0.11	entrypoints	0.4	executing	0.8.3
facets-overview	1.0.0	fastjsonschema	2.16.2	filelock	3.9.0
fonttools	4.25.0	idna	3.3	ipykernel	6.15.3
ipython	8.5.0	ipython-genutils	0.2.0	ipywidgets	7.7.2
jedi	0.18.1	Jinja2	2.11.3	jmespath	0.10.0
joblib	1.1.1	jsonschema	4.4.0	jupyter-client	6.1.12
jupyter_core	4.11.2	jupyterlab-pygments	0.1.2	jupyterlab-widgets	1.0.0
kiwisolver	1.3.2	MarkupSafe	2.0.1	matplotlib	3.5.1
matplotlib-inline	0.1.2	mccabe	0.7.0	mistune	0.8.4
mypy-extensions	0.4.3	nbclient	0.5.13	nbconvert	6.4.4
nbformat	5.3.0	nest-asyncio	1.5.5	nodeenv	1.7.0
notebook	6.4.8	numpy	1.21.5	packaging	21.3
pandas	1.4.2	pandocfilters	1.5.0	parso	0.8.3
pathspec	0.9.0	patsy	0.5.2	pexpect	4.8.0
pickleshare	0.7.5	Pillow	9.0.1	pip	21.2.4
platformdirs	2.6.2	plotly	5.6.0	pluggy	1.0.0
prometheus-client	0.13.1	prompt-toolkit	3.0.20	protobuf	3.19.4
psutil	5.8.0	psycopg2	2.9.3	ptyprocess	0.7.0
pure-eval	0.2.2	pyarrow	7.0.0	pycparser	2.21
pyflakes	2.5.0	Pygments	2.11.2	PyGObject	3.36.0
pyodbc	4.0.32	pyparsing	3.0.4	pyright	1.1.283
pyrsistent	0.18.0	python-dateutil	2.8.2	python-lsp-jsonrpc	1.0.0
python-lsp-server	1.6.0	pytz	2021.3	pyzmq	22.3.0
requests	2.27.1	requests-unixsocket	0.2.0	rope	0.22.0
s3transfer	0.5.0	scikit-learn	1.0.2	scipy	1.7.3
seaborn	0.11.2	Send2Trash	1.8.0	setuptools	61.2.0
six	1.16.0	soupsieve	2.3.1	ssh-import-id	5.10
stack-data	0.2.0	statsmodels	0.13.2	tenacity	8.0.1
terminado	0.13.1	testpath	0.5.0	threadpoolctl	2.2.0
tokenize-rt	4.2.1	tomli	1.2.2	tornado	6.1
traitlets	5.1.1	typing_extensions	4.1.1	ujson	5.1.0
unattended-upgrades	0.1	urllib3	1.26.9	virtualenv	20.8.0
wcwidth	0.2.5	webencodings	0.5.1	whatthepatch	1.0.4
wheel	0.37.0	widgetsnbextension	3.6.1	yapf	0.31.0

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2022-11-11.

Library	Version	Library	Version	Library	Version
arrow	10.0.0	askpass	1.1	assertthat	0.2.1
backports	1.4.1	base	4.2.2	base64enc	0.1-3
bit	4.0.4	bit64	4.0.5	blob	1.2.3
boot	1.3-28	brew	1.0-8	brio	1.1.3
broom	1.0.1	bslib	0.4.1	cachem	1.0.6
callr	3.7.3	caret	6.0-93	cellranger	1.1.0
chron	2.3-58	class	7.3-21	cli	3.4.1
clipr	0.8.0	clock	0.6.1	cluster	2.1.4
codetools	0.2-19	colorspace	2.0-3	commonmark	1.8.1
compiler	4.2.2	config	0.3.1	cpp11	0.4.3
crayon	1.5.2	credentials	1.3.2	curl	4.3.3
data.table	1.14.4	datasets	4.2.2	DBI	1.1.3
dbplyr	2.2.1	desc	1.4.2	devtools	2.4.5
diffobj	0.3.5	digest	0.6.30	downlit	0.4.2
dplyr	1.0.10	dtplyr	1.2.2	e1071	1.7-12
ellipsis	0.3.2	evaluate	0.18	fansi	1.0.3
farver	2.1.1	fastmap	1.1.0	fontawesome	0.4.0
forcats	0.5.2	foreach	1.5.2	foreign	0.8-82
forge	0.2.0	fs	1.5.2	future	1.29.0
future.apply	1.10.0	gargle	1.2.1	generics	0.1.3
gert	1.9.1	ggplot2	3.4.0	gh	1.3.1
gitcreds	0.1.2	glmnet	4.1-4	globals	0.16.1
glue	1.6.2	googledrive	2.0.0	googlesheets4	1.0.1
gower	1.0.0	graphics	4.2.2	grDevices	4.2.2
grid	4.2.2	gridExtra	2.3	gsubfn	0.7
gtable	0.3.1	hardhat	1.2.0	haven	2.5.1
highr	0.9	hms	1.1.2	htmltools	0.5.3
htmlwidgets	1.5.4	httpuv	1.6.6	httr	1.4.4
ids	1.0.1	ini	0.3.1	ipred	0.9-13
isoband	0.2.6	iterators	1.0.14	jquerylib	0.1.4
jsonlite	1.8.3	KernSmooth	2.23-20	knitr	1.40
labeling	0.4.2	later	1.3.0	lattice	0.20-45
lava	1.7.0	lifecycle	1.0.3	listenv	0.8.0
lubridate	1.9.0	magrittr	2.0.3	markdown	1.3
MASS	7.3-58.2	Matrix	1.5-1	memoise	2.0.1
methods	4.2.2	mgcv	1.8-41	mime	0.12
miniUI	0.1.1.1	ModelMetrics	1.2.2.2	modelr	0.1.9
munsell	0.5.0	nlme	3.1-162	nnet	7.3-18
numDeriv	2016.8-1.1	openssl	2.0.4	parallel	4.2.2
parallelly	1.32.1	pillar	1.8.1	pkgbuild	1.3.1
pkgconfig	2.0.3	pkgdown	2.0.6	pkgload	1.3.1
plogr	0.2.0	plyr	1.8.7	praise	1.0.0
prettyunits	1.1.1	pROC	1.18.0	processx	3.8.0
prodlim	2019.11.13	profvis	0.3.7	progress	1.2.2
progressr	0.11.0	promises	1.2.0.1	proto	1.0.0
proxy	0.4-27	ps	1.7.2	purrr	0.3.5
r2d3	0.2.6	R6	2.5.1	ragg	1.2.4
randomForest	4.7-1.1	rappdirs	0.3.3	rcmdcheck	1.4.0
RColorBrewer	1.1-3	Rcpp	1.0.9	RcppEigen	0.3.3.9.3
readr	2.1.3	readxl	1.4.1	recipes	1.0.3
rematch	1.0.1	rematch2	2.1.2	remotes	2.4.2
reprex	2.0.2	reshape2	1.4.4	rlang	1.0.6
rmarkdown	2.18	RODBC	1.3-19	roxygen2	7.2.1
rpart	4.1.19	rprojroot	2.0.3	Rserve	1.8-12
RSQLite	2.2.18	rstudioapi	0.14	rversions	2.1.2
rvest	1.0.3	sass	0.4.2	scales	1.2.1
selectr	0.4-2	sessioninfo	1.2.2	shape	1.4.6
shiny	1.7.3	sourcetools	0.1.7	sparklyr	1.7.8
SparkR	3.3.2	spatial	7.3-11	splines	4.2.2
sqldf	0.4-11	SQUAREM	2021.1	stats	4.2.2
stats4	4.2.2	stringi	1.7.8	stringr	1.4.1
survival	3.4-0	sys	3.4.1	systemfonts	1.0.4
tcltk	4.2.2	testthat	3.1.5	textshaping	0.3.6
tibble	3.1.8	tidyr	1.2.1	tidyselect	1.2.0
tidyverse	1.3.2	timechange	0.1.1	timeDate	4021.106
tinytex	0.42	tools	4.2.2	tzdb	0.3.0
urlchecker	1.0.1	usethis	2.1.6	utf8	1.2.2
utils	4.2.2	uuid	1.1-0	vctrs	0.5.0
viridisLite	0.4.1	vroom	1.6.0	waldo	0.4.0
whisker	0.4	withr	2.5.0	xfun	0.34
xml2	1.3.3	xopen	1.0.0	xtable	1.8-4
yaml	2.3.6	zip	2.2.2

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID	Artifact ID	Version
antlr	antlr	2.7.7
com.amazonaws	amazon-kinesis-client	1.12.0
com.amazonaws	aws-java-sdk-autoscaling	1.12.189
com.amazonaws	aws-java-sdk-cloudformation	1.12.189
com.amazonaws	aws-java-sdk-cloudfront	1.12.189
com.amazonaws	aws-java-sdk-cloudhsm	1.12.189
com.amazonaws	aws-java-sdk-cloudsearch	1.12.189
com.amazonaws	aws-java-sdk-cloudtrail	1.12.189
com.amazonaws	aws-java-sdk-cloudwatch	1.12.189
com.amazonaws	aws-java-sdk-cloudwatchmetrics	1.12.189
com.amazonaws	aws-java-sdk-codedeploy	1.12.189
com.amazonaws	aws-java-sdk-cognitoidentity	1.12.189
com.amazonaws	aws-java-sdk-cognitosync	1.12.189
com.amazonaws	aws-java-sdk-config	1.12.189
com.amazonaws	aws-java-sdk-core	1.12.189
com.amazonaws	aws-java-sdk-datapipeline	1.12.189
com.amazonaws	aws-java-sdk-directconnect	1.12.189
com.amazonaws	aws-java-sdk-directory	1.12.189
com.amazonaws	aws-java-sdk-dynamodb	1.12.189
com.amazonaws	aws-java-sdk-ec2	1.12.189
com.amazonaws	aws-java-sdk-ecs	1.12.189
com.amazonaws	aws-java-sdk-efs	1.12.189
com.amazonaws	aws-java-sdk-elasticache	1.12.189
com.amazonaws	aws-java-sdk-elasticbeanstalk	1.12.189
com.amazonaws	aws-java-sdk-elasticloadbalancing	1.12.189
com.amazonaws	aws-java-sdk-elastictranscoder	1.12.189
com.amazonaws	aws-java-sdk-emr	1.12.189
com.amazonaws	aws-java-sdk-glacier	1.12.189
com.amazonaws	aws-java-sdk-glue	1.12.189
com.amazonaws	aws-java-sdk-iam	1.12.189
com.amazonaws	aws-java-sdk-importexport	1.12.189
com.amazonaws	aws-java-sdk-kinesis	1.12.189
com.amazonaws	aws-java-sdk-kms	1.12.189
com.amazonaws	aws-java-sdk-lambda	1.12.189
com.amazonaws	aws-java-sdk-logs	1.12.189
com.amazonaws	aws-java-sdk-machinelearning	1.12.189
com.amazonaws	aws-java-sdk-opsworks	1.12.189
com.amazonaws	aws-java-sdk-rds	1.12.189
com.amazonaws	aws-java-sdk-redshift	1.12.189
com.amazonaws	aws-java-sdk-route53	1.12.189
com.amazonaws	aws-java-sdk-s3	1.12.189
com.amazonaws	aws-java-sdk-ses	1.12.189
com.amazonaws	aws-java-sdk-simpledb	1.12.189
com.amazonaws	aws-java-sdk-simpleworkflow	1.12.189
com.amazonaws	aws-java-sdk-sns	1.12.189
com.amazonaws	aws-java-sdk-sqs	1.12.189
com.amazonaws	aws-java-sdk-ssm	1.12.189
com.amazonaws	aws-java-sdk-storagegateway	1.12.189
com.amazonaws	aws-java-sdk-sts	1.12.189
com.amazonaws	aws-java-sdk-support	1.12.189
com.amazonaws	aws-java-sdk-swf-libraries	1.11.22
com.amazonaws	aws-java-sdk-workspaces	1.12.189
com.amazonaws	jmespath-java	1.12.189
com.chuusai	shapeless_2.12	2.3.3
com.clearspring.analytics	stream	2.9.6
com.databricks	Rserve	1.8-3
com.databricks	jets3t	0.7.1-0
com.databricks.scalapb	compilerplugin_2.12	0.4.15-10
com.databricks.scalapb	scalapb-runtime_2.12	0.4.15-10
com.esotericsoftware	kryo-shaded	4.0.2
com.esotericsoftware	minlog	1.3.0
com.fasterxml	classmate	1.3.4
com.fasterxml.jackson.core	jackson-annotations	2.13.4
com.fasterxml.jackson.core	jackson-core	2.13.4
com.fasterxml.jackson.core	jackson-databind	2.13.4.2
com.fasterxml.jackson.dataformat	jackson-dataformat-cbor	2.13.4
com.fasterxml.jackson.datatype	jackson-datatype-joda	2.13.4
com.fasterxml.jackson.datatype	jackson-datatype-jsr310	2.13.4
com.fasterxml.jackson.module	jackson-module-paranamer	2.13.4
com.fasterxml.jackson.module	jackson-module-scala_2.12	2.13.4
com.github.ben-manes.caffeine	caffeine	2.3.4
com.github.fommil	jniloader	1.1
com.github.fommil.netlib	core	1.1.2
com.github.fommil.netlib	native_ref-java	1.1
com.github.fommil.netlib	native_ref-java-natives	1.1
com.github.fommil.netlib	native_system-java	1.1
com.github.fommil.netlib	native_system-java-natives	1.1
com.github.fommil.netlib	netlib-native_ref-linux-x86_64-natives	1.1
com.github.fommil.netlib	netlib-native_system-linux-x86_64-natives	1.1
com.github.luben	zstd-jni	1.5.2-1
com.github.wendykierp	JTransforms	3.1
com.google.code.findbugs	jsr305	3.0.0
com.google.code.gson	gson	2.8.6
com.google.crypto.tink	tink	1.6.1
com.google.flatbuffers	flatbuffers-java	1.12.0
com.google.guava	guava	15.0
com.google.protobuf	protobuf-java	2.6.1
com.h2database	h2	2.0.204
com.helger	profiler	1.1.1
com.jcraft	jsch	0.1.50
com.jolbox	bonecp	0.8.0.RELEASE
com.lihaoyi	sourcecode_2.12	0.1.9
com.microsoft.azure	azure-data-lake-store-sdk	2.3.9
com.microsoft.sqlserver	mssql-jdbc	11.2.2.jre8
com.ning	compress-lzf	1.1
com.sun.mail	javax.mail	1.5.2
com.tdunning	json	1.8
com.thoughtworks.paranamer	paranamer	2.8
com.trueaccord.lenses	lenses_2.12	0.4.12
com.twitter	chill-java	0.10.0
com.twitter	chill_2.12	0.10.0
com.twitter	util-app_2.12	7.1.0
com.twitter	util-core_2.12	7.1.0
com.twitter	util-function_2.12	7.1.0
com.twitter	util-jvm_2.12	7.1.0
com.twitter	util-lint_2.12	7.1.0
com.twitter	util-registry_2.12	7.1.0
com.twitter	util-stats_2.12	7.1.0
com.typesafe	config	1.2.1
com.typesafe.scala-logging	scala-logging_2.12	3.7.2
com.uber	h3	3.7.0
com.univocity	univocity-parsers	2.9.1
com.zaxxer	HikariCP	4.0.3
commons-cli	commons-cli	1.5.0
commons-codec	commons-codec	1.15
commons-collections	commons-collections	3.2.2
commons-dbcp	commons-dbcp	1.4
commons-fileupload	commons-fileupload	1.3.3
commons-httpclient	commons-httpclient	3.1
commons-io	commons-io	2.11.0
commons-lang	commons-lang	2.6
commons-logging	commons-logging	1.1.3
commons-pool	commons-pool	1.5.4
dev.ludovic.netlib	arpack	2.2.1
dev.ludovic.netlib	blas	2.2.1
dev.ludovic.netlib	lapack	2.2.1
info.ganglia.gmetric4j	gmetric4j	1.0.10
io.airlift	aircompressor	0.21
io.delta	delta-sharing-spark_2.12	0.6.3
io.dropwizard.metrics	metrics-core	4.1.1
io.dropwizard.metrics	metrics-graphite	4.1.1
io.dropwizard.metrics	metrics-healthchecks	4.1.1
io.dropwizard.metrics	metrics-jetty9	4.1.1
io.dropwizard.metrics	metrics-jmx	4.1.1
io.dropwizard.metrics	metrics-json	4.1.1
io.dropwizard.metrics	metrics-jvm	4.1.1
io.dropwizard.metrics	metrics-servlets	4.1.1
io.netty	netty-all	4.1.74.Final
io.netty	netty-buffer	4.1.74.Final
io.netty	netty-codec	4.1.74.Final
io.netty	netty-common	4.1.74.Final
io.netty	netty-handler	4.1.74.Final
io.netty	netty-resolver	4.1.74.Final
io.netty	netty-tcnative-classes	2.0.48.Final
io.netty	netty-transport	4.1.74.Final
io.netty	netty-transport-classes-epoll	4.1.74.Final
io.netty	netty-transport-classes-kqueue	4.1.74.Final
io.netty	netty-transport-native-epoll-linux-aarch_64	4.1.74.Final
io.netty	netty-transport-native-epoll-linux-x86_64	4.1.74.Final
io.netty	netty-transport-native-kqueue-osx-aarch_64	4.1.74.Final
io.netty	netty-transport-native-kqueue-osx-x86_64	4.1.74.Final
io.netty	netty-transport-native-unix-common	4.1.74.Final
io.prometheus	simpleclient	0.7.0
io.prometheus	simpleclient_common	0.7.0
io.prometheus	simpleclient_dropwizard	0.7.0
io.prometheus	simpleclient_pushgateway	0.7.0
io.prometheus	simpleclient_servlet	0.7.0
io.prometheus.jmx	collector	0.12.0
jakarta.annotation	jakarta.annotation-api	1.3.5
jakarta.servlet	jakarta.servlet-api	4.0.3
jakarta.validation	jakarta.validation-api	2.0.2
jakarta.ws.rs	jakarta.ws.rs-api	2.1.6
javax.activation	activation	1.1.1
javax.el	javax.el-api	2.2.4
javax.jdo	jdo-api	3.0.1
javax.transaction	jta	1.1
javax.transaction	transaction-api	1.1
javax.xml.bind	jaxb-api	2.2.11
javolution	javolution	5.5.1
jline	jline	2.14.6
joda-time	joda-time	2.10.13
net.java.dev.jna	jna	5.8.0
net.razorvine	pickle	1.2
net.sf.jpam	jpam	1.1
net.sf.opencsv	opencsv	2.3
net.sf.supercsv	super-csv	2.2.0
net.snowflake	snowflake-ingest-sdk	0.9.6
net.snowflake	snowflake-jdbc	3.13.22
net.sourceforge.f2j	arpack_combined_all	0.1
org.acplt.remotetea	remotetea-oncrpc	1.1.2
org.antlr	ST4	4.0.4
org.antlr	antlr-runtime	3.5.2
org.antlr	antlr4-runtime	4.8
org.antlr	stringtemplate	3.2.1
org.apache.ant	ant	1.9.2
org.apache.ant	ant-jsch	1.9.2
org.apache.ant	ant-launcher	1.9.2
org.apache.arrow	arrow-format	7.0.0
org.apache.arrow	arrow-memory-core	7.0.0
org.apache.arrow	arrow-memory-netty	7.0.0
org.apache.arrow	arrow-vector	7.0.0
org.apache.avro	avro	1.11.0
org.apache.avro	avro-ipc	1.11.0
org.apache.avro	avro-mapred	1.11.0
org.apache.commons	commons-collections4	4.4
org.apache.commons	commons-compress	1.21
org.apache.commons	commons-crypto	1.1.0
org.apache.commons	commons-lang3	3.12.0
org.apache.commons	commons-math3	3.6.1
org.apache.commons	commons-text	1.10.0
org.apache.curator	curator-client	2.13.0
org.apache.curator	curator-framework	2.13.0
org.apache.curator	curator-recipes	2.13.0
org.apache.derby	derby	10.14.2.0
org.apache.hadoop	hadoop-client-api	3.3.4-databricks
org.apache.hadoop	hadoop-client-runtime	3.3.4
org.apache.hive	hive-beeline	2.3.9
org.apache.hive	hive-cli	2.3.9
org.apache.hive	hive-jdbc	2.3.9
org.apache.hive	hive-llap-client	2.3.9
org.apache.hive	hive-llap-common	2.3.9
org.apache.hive	hive-serde	2.3.9
org.apache.hive	hive-shims	2.3.9
org.apache.hive	hive-storage-api	2.8.1
org.apache.hive.shims	hive-shims-0.23	2.3.9
org.apache.hive.shims	hive-shims-common	2.3.9
org.apache.hive.shims	hive-shims-scheduler	2.3.9
org.apache.httpcomponents	httpclient	4.5.13
org.apache.httpcomponents	httpcore	4.4.14
org.apache.ivy	ivy	2.5.0
org.apache.logging.log4j	log4j-1.2-api	2.18.0
org.apache.logging.log4j	log4j-api	2.18.0
org.apache.logging.log4j	log4j-core	2.18.0
org.apache.logging.log4j	log4j-slf4j-impl	2.18.0
org.apache.mesos	mesos-shaded-protobuf	1.4.0
org.apache.orc	orc-core	1.7.6
org.apache.orc	orc-mapreduce	1.7.6
org.apache.orc	orc-shims	1.7.6
org.apache.parquet	parquet-column	1.12.3-databricks-0002
org.apache.parquet	parquet-common	1.12.3-databricks-0002
org.apache.parquet	parquet-encoding	1.12.3-databricks-0002
org.apache.parquet	parquet-format-structures	1.12.3-databricks-0002
org.apache.parquet	parquet-hadoop	1.12.3-databricks-0002
org.apache.parquet	parquet-jackson	1.12.3-databricks-0002
org.apache.thrift	libfb303	0.9.3
org.apache.thrift	libthrift	0.12.0
org.apache.xbean	xbean-asm9-shaded	4.20
org.apache.yetus	audience-annotations	0.13.0
org.apache.zookeeper	zookeeper	3.6.2
org.apache.zookeeper	zookeeper-jute	3.6.2
org.checkerframework	checker-qual	3.5.0
org.codehaus.jackson	jackson-core-asl	1.9.13
org.codehaus.jackson	jackson-mapper-asl	1.9.13
org.codehaus.janino	commons-compiler	3.0.16
org.codehaus.janino	janino	3.0.16
org.datanucleus	datanucleus-api-jdo	4.2.4
org.datanucleus	datanucleus-core	4.1.17
org.datanucleus	datanucleus-rdbms	4.1.19
org.datanucleus	javax.jdo	3.2.0-m3
org.eclipse.jetty	jetty-client	9.4.46.v20220331
org.eclipse.jetty	jetty-continuation	9.4.46.v20220331
org.eclipse.jetty	jetty-http	9.4.46.v20220331
org.eclipse.jetty	jetty-io	9.4.46.v20220331
org.eclipse.jetty	jetty-jndi	9.4.46.v20220331
org.eclipse.jetty	jetty-plus	9.4.46.v20220331
org.eclipse.jetty	jetty-proxy	9.4.46.v20220331
org.eclipse.jetty	jetty-security	9.4.46.v20220331
org.eclipse.jetty	jetty-server	9.4.46.v20220331
org.eclipse.jetty	jetty-servlet	9.4.46.v20220331
org.eclipse.jetty	jetty-servlets	9.4.46.v20220331
org.eclipse.jetty	jetty-util	9.4.46.v20220331
org.eclipse.jetty	jetty-util-ajax	9.4.46.v20220331
org.eclipse.jetty	jetty-webapp	9.4.46.v20220331
org.eclipse.jetty	jetty-xml	9.4.46.v20220331
org.eclipse.jetty.websocket	websocket-api	9.4.46.v20220331
org.eclipse.jetty.websocket	websocket-client	9.4.46.v20220331
org.eclipse.jetty.websocket	websocket-common	9.4.46.v20220331
org.eclipse.jetty.websocket	websocket-server	9.4.46.v20220331
org.eclipse.jetty.websocket	websocket-servlet	9.4.46.v20220331
org.fusesource.leveldbjni	leveldbjni-all	1.8
org.glassfish.hk2	hk2-api	2.6.1
org.glassfish.hk2	hk2-locator	2.6.1
org.glassfish.hk2	hk2-utils	2.6.1
org.glassfish.hk2	osgi-resource-locator	1.0.3
org.glassfish.hk2.external	aopalliance-repackaged	2.6.1
org.glassfish.hk2.external	jakarta.inject	2.6.1
org.glassfish.jersey.containers	jersey-container-servlet	2.36
org.glassfish.jersey.containers	jersey-container-servlet-core	2.36
org.glassfish.jersey.core	jersey-client	2.36
org.glassfish.jersey.core	jersey-common	2.36
org.glassfish.jersey.core	jersey-server	2.36
org.glassfish.jersey.inject	jersey-hk2	2.36
org.hibernate.validator	hibernate-validator	6.1.0.Final
org.javassist	javassist	3.25.0-GA
org.jboss.logging	jboss-logging	3.3.2.Final
org.jdbi	jdbi	2.63.1
org.jetbrains	annotations	17.0.0
org.joda	joda-convert	1.7
org.jodd	jodd-core	3.5.2
org.json4s	json4s-ast_2.12	3.7.0-M11
org.json4s	json4s-core_2.12	3.7.0-M11
org.json4s	json4s-jackson_2.12	3.7.0-M11
org.json4s	json4s-scalap_2.12	3.7.0-M11
org.lz4	lz4-java	1.8.0
org.mariadb.jdbc	mariadb-java-client	2.7.4
org.mlflow	mlflow-spark	2.1.1
org.objenesis	objenesis	2.5.1
org.postgresql	postgresql	42.3.3
org.roaringbitmap	RoaringBitmap	0.9.25
org.roaringbitmap	shims	0.9.25
org.rocksdb	rocksdbjni	6.28.2
org.rosuda.REngine	REngine	2.1.0
org.scala-lang	scala-compiler_2.12	2.12.14
org.scala-lang	scala-library_2.12	2.12.14
org.scala-lang	scala-reflect_2.12	2.12.14
org.scala-lang.modules	scala-collection-compat_2.12	2.4.3
org.scala-lang.modules	scala-parser-combinators_2.12	1.1.2
org.scala-lang.modules	scala-xml_2.12	1.2.0
org.scala-sbt	test-interface	1.0
org.scalacheck	scalacheck_2.12	1.14.2
org.scalactic	scalactic_2.12	3.0.8
org.scalanlp	breeze-macros_2.12	1.2
org.scalanlp	breeze_2.12	1.2
org.scalatest	scalatest_2.12	3.0.8
org.slf4j	jcl-over-slf4j	1.7.36
org.slf4j	jul-to-slf4j	1.7.36
org.slf4j	slf4j-api	1.7.36
org.spark-project.spark	unused	1.0.0
org.threeten	threeten-extra	1.5.0
org.tukaani	xz	1.9
org.typelevel	algebra_2.12	2.0.1
org.typelevel	cats-kernel_2.12	2.1.1
org.typelevel	macro-compat_2.12	1.1.1
org.typelevel	spire-macros_2.12	0.17.0
org.typelevel	spire-platform_2.12	0.17.0
org.typelevel	spire-util_2.12	0.17.0
org.typelevel	spire_2.12	0.17.0
org.wildfly.openssl	wildfly-openssl	1.0.7.Final
org.xerial	sqlite-jdbc	3.8.11.2
org.xerial.snappy	snappy-java	1.1.8.4
org.yaml	snakeyaml	1.24
oro	oro	2.0.8
pl.edu.icm	JLargeArrays	1.5
software.amazon.cryptools	AmazonCorrettoCryptoProvider	1.6.1-linux-x86_64
software.amazon.ion	ion-java	1.0.2
stax	stax-api	1.0.1