Synchronize with Azure Blob storage by using AzCopy

  • Article

You can synchronize local storage with Azure Blob storage by using the AzCopy v10 command-line utility.

You can synchronize the contents of a local file system with a blob container. You can also synchronize containers and virtual directories with one another. Synchronization is one way. In other words, you choose which of these two endpoints is the source and which one is the destination. Synchronization also uses server to server APIs. The examples presented in this section also work with accounts that have a hierarchical namespace.

Note

The current release of AzCopy doesn't synchronize between other sources and destinations (For example: File storage or Amazon Web Services (AWS) S3 buckets).

To see examples for other types of tasks such as uploading files, downloading blobs, or copying blobs between accounts, see the links presented in the Next Steps section of this article.

Get started

See the Get started with AzCopy article to download AzCopy and learn about the ways that you can provide authorization credentials to the storage service.

Note

The examples in this article assume that you've provided authorization credentials by using Microsoft Entra ID.

If you'd rather use a SAS token to authorize access to blob data, then you can append that token to the resource URL in each AzCopy command. For example: 'https://<storage-account-name>.blob.core.chinacloudapi.cn/<container-name><SAS-token>'.

Guidelines

By default, the sync command compares file names and last modified timestamps. You can override that behavior to use MD5 hashes instead of last modified timestamps by using the --compare-hash flag. Set the --delete-destination optional flag to a value of true or prompt to delete files in the destination directory if those files no longer exist in the source directory.

  • If you set the --delete-destination flag to true, AzCopy deletes files without providing a prompt. If you want a prompt to appear before AzCopy deletes a file, set the --delete-destination flag to prompt.

  • If you plan to set the --delete-destination flag to prompt or false, consider using the copy command instead of the sync command and set the --overwrite parameter to ifSourceNewer. The copy command consumes less memory and incurs less billing costs because a copy operation doesn't have to index the source or destination prior to moving files.

  • If you don't plan to use the --compare-hash flag, then the machine on which you run the sync command should have an accurate system clock because the last modified times are critical in determining whether a file should be transferred. If your system has significant clock skew, avoid modifying files at the destination too close to the time that you plan to run a sync command.

  • AzCopy uses server-to-server APIs to synchronize data between storage accounts. That means that data is copied directly between storage servers. However, AzCopy does set up and monitor each transfer, and for larger storage accounts (For example, accounts that contain millions of blobs), AzCopy might require a substantial amount of compute resources to accomplish these tasks. Therefore, if you are running AzCopy from Virtual Machine (VM), make sure that the VM has enough cores/memory to handle the load.

  • To prevent accidental deletions, make sure to enable the soft delete feature before you use the --delete-destination=prompt|true flag.

Update a container with changes to a local file system

In this case, the container is the destination, and the local file system is the source.

Tip

This example encloses path arguments with single quotes (''). Use single quotes in all command shells except for the Windows Command Shell (cmd.exe). If you're using a Windows Command Shell (cmd.exe), enclose path arguments with double quotes ("") instead of single quotes ('').

Syntax

azcopy sync '<local-directory-path>' 'https://<storage-account-name>.blob.core.chinacloudapi.cn/<container-name>' --recursive

Example

azcopy sync 'C:\myDirectory' 'https://mystorageaccount.blob.core.chinacloudapi.cn/mycontainer' --recursive

Update a local file system with changes to a container

In this case, the local file system is the destination, and the container is the source.

Tip

This example encloses path arguments with single quotes (''). Use single quotes in all command shells except for the Windows Command Shell (cmd.exe). If you're using a Windows Command Shell (cmd.exe), enclose path arguments with double quotes ("") instead of single quotes ('').

Syntax

azcopy sync 'https://<storage-account-name>.blob.core.chinacloudapi.cn/<container-name>' 'C:\myDirectory' --recursive

Example

azcopy sync 'https://mystorageaccount.blob.core.chinacloudapi.cn/mycontainer' 'C:\myDirectory' --recursive

Update a container with changes in another container

The first container that appears in this command is the source. The second one is the destination.

If you provide authorization credentials by using Microsoft Entra ID, make sure that you've set up the proper roles in your source and destination account. See Option 1: Use Microsoft Entra ID.

Tip

This example encloses path arguments with single quotes (''). Use single quotes in all command shells except for the Windows Command Shell (cmd.exe). If you're using a Windows Command Shell (cmd.exe), enclose path arguments with double quotes ("") instead of single quotes ('').

Syntax

azcopy sync 'https://<source-storage-account-name>.blob.core.chinacloudapi.cn/<container-name>/<SAS-token>' 'https://<destination-storage-account-name>.blob.core.chinacloudapi.cn/<container-name>' --recursive

Example

azcopy sync 'https://mysourceaccount.blob.core.chinacloudapi.cn/mycontainer?sv=2018-03-28&ss=bfqt&srt=sco&sp=rwdlacup&se=2019-07-04T05:30:08Z&st=2019-07-03T21:30:08Z&spr=https&sig=CAfhgnc9gdGktvB=ska7bAiqIddM845yiyFwdMH481QA8%3D' 'https://mydestinationaccount.blob.core.chinacloudapi.cn/mycontainer' --recursive

Update a directory with changes to a directory in another container

The first directory that appears in this command is the source. The second one is the destination.

If you provide authorization credentials by using Microsoft Entra ID, make sure that you've set up the proper roles in your source and destination account. See Option 1: Use Microsoft Entra ID.

Tip

This example encloses path arguments with single quotes (''). Use single quotes in all command shells except for the Windows Command Shell (cmd.exe). If you're using a Windows Command Shell (cmd.exe), enclose path arguments with double quotes ("") instead of single quotes ('').

Syntax

azcopy sync 'https://<source-storage-account-name>.blob.core.chinacloudapi.cn/<container-name>/<directory-name>/<SAS-token>' 'https://<destination-storage-account-name>.blob.core.chinacloudapi.cn/<container-name>/<directory-name>' --recursive

Example

azcopy sync 'https://mysourceaccount.blob.core.chinacloudapi.cn/<container-name>/myDirectory?sv=2018-03-28&ss=bfqt&srt=sco&sp=rwdlacup&se=2019-07-04T05:30:08Z&st=2019-07-03T21:30:08Z&spr=https&sig=CAfhgnc9gdGktvB=ska7bAiqIddM845yiyFwdMH481QA8%3D' 'https://mydestinationaccount.blob.core.chinacloudapi.cn/mycontainer/myDirectory' --recursive

Synchronize with optional flags

You can tweak your sync operation by using optional flags. Here's a few examples.

Scenario Flag
Specify how strictly MD5 hashes should be validated when downloading. --check-md5=[NoCheck|LogOnly|FailIfDifferent|FailIfDifferentOrMissing]
Exclude files based on a pattern. --exclude-path
Specify how detailed you want your sync-related log entries to be. --log-level=[WARNING|ERROR|INFO|NONE]

For a complete list of flags, see options.

Note

The --recursive flag is set to true by default. The --exclude-pattern and --include-pattern flags apply to only to file names and not other parts of the file path.

Next steps

Find more examples in these articles:

See these articles to configure settings, optimize performance, and troubleshoot issues: