What are Unity Catalog volumes?

Volumes are Unity Catalog objects that enable governance over non-tabular datasets. Volumes represent a logical volume of storage in a cloud object storage location. Volumes provide capabilities for accessing, storing, governing, and organizing files.

While tables provide governance over tabular datasets, volumes add governance over non-tabular datasets. You can use volumes to store and access files in any format, including structured, semi-structured, and unstructured data.

Databricks recommends using volumes to govern access to all non-tabular data. Like tables, volumes can be managed or external.

The following articles provide more information about working with volumes:

• Create and manage volumes.
• Manage files in volumes.
• Explore storage and find data files.
• Managed vs. external volumes.
• What are the privileges for volumes?.

What is a managed volume?

A managed volume is a Unity Catalog-governed storage volume created within the managed storage location of the containing schema. See Specify a managed storage location in Unity Catalog.

Managed volumes allow the creation of governed storage for working with files without the overhead of external locations and storage credentials. You do not need to specify a location when creating a managed volume, and all file access for data in managed volumes is through paths managed by Unity Catalog.

What is an external volume?

An external volume is a Unity Catalog-governed storage volume registered against a directory within an external location using Unity Catalog-governed storage credentials.

Unity Catalog does not manage the lifecycle and layout of the files in external volumes. When you drop an external volume, Unity Catalog does not delete the underlying data.

What path is used for accessing files in a volume?

Volumes sit at the third level of the Unity Catalog three-level namespace (catalog.schema.volume):

The path to access volumes is the same whether you use Apache Spark, SQL, Python, or other languages and libraries. This differs from legacy access patterns for files in object storage bound to an Azure Databricks workspace.

The path to access files in volumes uses the following format:

/Volumes/<catalog>/<schema>/<volume>/<path>/<file-name>

Azure Databricks also supports an optional dbfs:/ scheme when working with Apache Spark, so the following path also works:

dbfs:/Volumes/<catalog>/<schema>/<volume>/<path>/<file-name>

The sequence /<catalog>/<schema>/<volume> in the path corresponds to the three Unity Catalog object names associated with the file. These path elements are read-only and not directly writeable by users, meaning it is not possible to create or delete these directories using filesystem operations. They are automatically managed and kept in sync with the corresponding Unity Catalog entities.

Reserved paths for volumes

Volumes introduces the following reserved paths used for accessing volumes:

• dbfs:/Volumes
• /Volumes

Volumes are only supported on Databricks Runtime 13.3 LTS and above. In Databricks Runtime 12.2 LTS and below, operations against /Volumes paths might succeed, but can write data to ephemeral storage disks attached to compute clusters rather than persisting data to Unity Catalog volumes as expected.

Limitations

You must use Unity Catalog-enabled compute to interact with Unity Catalog volumes. Volumes do not support all workloads.

The following limitations apply:

In Databricks Runtime 14.3 LTS and above:

• On single user user clusters, you cannot access volumes from threads and subprocesses in Scala.

In Databricks Runtime 14.2 and below:

• On compute configured with shared access mode, you can't use UDFs to access volumes.
  • Both Python or Scala have access to FUSE from the driver but not from executors.
  • Scala code that performs I/O operations can run on the driver but not the executors.
• On compute configured with single user access mode, there is no support for FUSE in Scala, Scala IO code accessing data using volume paths, or Scala UDFs. Python UDFs are supported in single user access mode.

On all supported Databricks Runtime versions:

• Unity Catalog UDFs do not support accessing volume file paths.

• You cannot access volumes from RDDs.

• You cannot use spark-submit with JARs stored in a volume.

• You cannot define dependencies to other libraries accessed via volume paths inside a wheel or JAR file.

• You cannot list Unity Catalog objects using the /Volumes/<catalog-name> or /Volumes/<catalog-name>/<schema-name> patterns. You must use a fully-qualified path that includes a volume name.

• The DBFS endpoint for the REST API does not support volumes paths.

• You cannot specify volumes as the destination for cluster log delivery.

• %sh mv is not supported for moving files between volumes. Use dbutils.fs.mv or %sh cp instead.

• You cannot create a custom Hadoop file system with volumes, meaning the following is not supported:

  import org.apache.hadoop.fs.Path
  val path =  new Path("dbfs:/Volumes/main/default/test-volume/file.txt")
  val fs = path.getFileSystem(sc.hadoopConfiguration)
  fs.listStatus(path)
  

• You must use the path format with a dbfs:/ scheme in the Azure Data Factory library configuration panel. For example, dbfs:/Volumes/<catalog-name>/<schema-name>/<volume-name>/file.