Outbound network and FQDN rules for Azure Kubernetes Service (AKS) clusters

This article provides the necessary details that allow you to secure outbound traffic from your Azure Kubernetes Service (AKS). It contains the cluster requirements for a base AKS deployment and additional requirements for optional addons and features. You can apply this information to any outbound restriction method or appliance.

To see an example configuration using Azure Firewall, visit Control egress traffic using Azure Firewall in AKS.

Background

AKS clusters are deployed on a virtual network. This network can either be customized and pre-configured by you or it can be created and managed by AKS. In either case, the cluster has outbound, or egress, dependencies on services outside of the virtual network.

For management and operational purposes, nodes in an AKS cluster need to access certain ports and fully qualified domain names (FQDNs). These endpoints are required for the nodes to communicate with the API server or to download and install core Kubernetes cluster components and node security updates. For example, the cluster needs to pull base system container images from Microsoft Container Registry (MCR).

The AKS outbound dependencies are almost entirely defined with FQDNs, which don't have static addresses behind them. The lack of static addresses means you can't use network security groups (NSGs) to lock down the outbound traffic from an AKS cluster.

By default, AKS clusters have unrestricted outbound internet access. This level of network access allows nodes and services you run to access external resources as needed. If you wish to restrict egress traffic, a limited number of ports and addresses must be accessible to maintain healthy cluster maintenance tasks. The simplest solution to securing outbound addresses is using a firewall device that can control outbound traffic based on domain names. Azure Firewall can restrict outbound HTTP and HTTPS traffic based on the FQDN of the destination. You can also configure your preferred firewall and security rules to allow these required ports and addresses.

Required outbound network rules and FQDNs for AKS clusters

The following network and FQDN/application rules are required for an AKS cluster. You can use them if you wish to configure a solution other than Azure Firewall.

• IP address dependencies are for non-HTTP/S traffic (both TCP and UDP traffic).
• FQDN HTTP/HTTPS endpoints can be placed in your firewall device.
• Wildcard HTTP/HTTPS endpoints are dependencies that can vary with your AKS cluster based on a number of qualifiers.
• AKS uses an admission controller to inject the FQDN as an environment variable to all deployments under kube-system and gatekeeper-system. This ensures all system communication between nodes and API server uses the API server FQDN and not the API server IP. You can get the same behavior on your own pods, in any namespace, by annotating the pod spec with an annotation named kubernetes.azure.com/set-kube-service-host-fqdn. If that annotation is present, AKS will set the KUBERNETES_SERVICE_HOST variable to the domain name of the API server instead of the in-cluster service IP. This is useful in cases where the cluster egress is via a layer 7 firewall.
• If you have an app or solution that needs to talk to the API server, you must either add an additional network rule to allow TCP communication to port 443 of your API server's IP OR , if you have a layer 7 firewall configured to allow traffic to the API Server's domain name, set kubernetes.azure.com/set-kube-service-host-fqdn in your pod specs.
• On rare occasions, if there's a maintenance operation, your API server IP might change. Planned maintenance operations that can change the API server IP are always communicated in advance.
• Under certain circumstances, it might happen that traffic towards "md-*.blob.storage.chinacloudapi.cn" is required. This dependency is due to some internal mechanisms of Azure Managed Disks. You might also want to use the Storage service tag.
• You might notice traffic towards "umsa*.blob.core.chinacloudapi.cn" endpoint. This endpoint is used to store manifests for Azure Linux VM Agent & Extensions and is regularly checked to download new versions.

Azure Global required network rules

Destination Endpoint	Protocol	Port	Use
*:1194 Or ServiceTag - AzureCloud.<Region>:1194 Or Regional CIDRs - RegionCIDRs:1194 Or APIServerPublicIP:1194 (only known after cluster creation)	UDP	1194	For tunneled secure communication between the nodes and the control plane. This isn't required for private clusters, or for clusters with the konnectivity-agent enabled.
*:9000 Or ServiceTag - AzureCloud.<Region>:9000 Or Regional CIDRs - RegionCIDRs:9000 Or APIServerPublicIP:9000 (only known after cluster creation)	TCP	9000	For tunneled secure communication between the nodes and the control plane. This isn't required for private clusters, or for clusters with the konnectivity-agent enabled.
*:123 or ntp.ubuntu.com:123 (if using Azure Firewall network rules)	UDP	123	Required for Network Time Protocol (NTP) time synchronization on Linux nodes. This isn't required for nodes provisioned after March 2021.
CustomDNSIP:53 (if using custom DNS servers)	UDP	53	If you're using custom DNS servers, you must ensure they're accessible by the cluster nodes.
APIServerPublicIP:443 (if running pods/deployments that access the API Server)	TCP	443	Required if running pods/deployments that access the API Server, those pods/deployments would use the API IP. This port isn't required for private clusters.

Azure Global required FQDN / application rules

Destination FQDN	Port	Use
*.hcp.<location>.cx.prod.service.azk8s.cn	HTTPS:443	Required for Node <-> API server communication. Replace <location> with the region where your AKS cluster is deployed. This is required for clusters with konnectivity-agent enabled. Konnectivity also uses Application-Layer Protocol Negotiation (ALPN) to communicate between agent and server. Blocking or rewriting the ALPN extension will cause a failure. This isn't required for private clusters.
mcr.azk8s.cn	HTTPS:443	Required to access images in Microsoft Container Registry (MCR). This registry contains first-party images/charts (for example, coreDNS, etc.). These images are required for the correct creation and functioning of the cluster, including scale and upgrade operations.
*.data.mcr.azk8s.cn, mcr-0001.mcr-msedge.net	HTTPS:443	Required for MCR storage backed by the Azure content delivery network (CDN).
management.chinacloudapi.cn	HTTPS:443	Required for Kubernetes operations against the Azure API.
login.partner.microsoftonline.cn	HTTPS:443	Required for Microsoft Entra authentication.
packages.microsoft.com	HTTPS:443	This address is the Microsoft packages repository used for cached apt-get operations. Example packages include Moby, PowerShell, and Azure CLI.
acs-mirror.azureedge.net	HTTPS:443	This address is for the repository required to download and install required binaries like kubenet and Azure CNI.

Azure operated by 21Vianet required network rules

Destination Endpoint	Protocol	Port	Use
*:1194 Or ServiceTag - AzureCloud.Region:1194 Or Regional CIDRs - RegionCIDRs:1194 Or APIServerPublicIP:1194 (only known after cluster creation)	UDP	1194	For tunneled secure communication between the nodes and the control plane.
*:9000 Or ServiceTag - AzureCloud.<Region>:9000 Or Regional CIDRs - RegionCIDRs:9000 Or APIServerPublicIP:9000 (only known after cluster creation)	TCP	9000	For tunneled secure communication between the nodes and the control plane.
*:22 Or ServiceTag - AzureCloud.<Region>:22 Or Regional CIDRs - RegionCIDRs:22 Or APIServerPublicIP:22 (only known after cluster creation)	TCP	22	For tunneled secure communication between the nodes and the control plane.
*:123 or ntp.ubuntu.com:123 (if using Azure Firewall network rules)	UDP	123	Required for Network Time Protocol (NTP) time synchronization on Linux nodes.
CustomDNSIP:53 (if using custom DNS servers)	UDP	53	If you're using custom DNS servers, you must ensure they're accessible by the cluster nodes.
APIServerPublicIP:443 (if running pods/deployments that access the API Server)	TCP	443	Required if running pods/deployments that access the API Server, those pod/deployments would use the API IP.

Azure operated by 21Vianet required FQDN / application rules

Destination FQDN	Port	Use
*.hcp.<location>.cx.prod.service.azk8s.cn	HTTPS:443	Required for Node <-> API server communication. Replace <location> with the region where your AKS cluster is deployed.
*.tun.<location>.cx.prod.service.azk8s.cn	HTTPS:443	Required for Node <-> API server communication. Replace <location> with the region where your AKS cluster is deployed.
mcr.azk8s.cn	HTTPS:443	Required to access images in Microsoft Container Registry (MCR). This registry contains first-party images/charts (for example, coreDNS, etc.). These images are required for the correct creation and functioning of the cluster, including scale and upgrade operations.
.data.mcr.azk8s.cn	HTTPS:443	Required for MCR storage backed by the Azure Content Delivery Network (CDN).
management.chinacloudapi.cn	HTTPS:443	Required for Kubernetes operations against the Azure API.
login.chinacloudapi.cn	HTTPS:443	Required for Microsoft Entra authentication.
packages.microsoft.com	HTTPS:443	This address is the Microsoft packages repository used for cached apt-get operations. Example packages include Moby, PowerShell, and Azure CLI.
*.azk8s.cn	HTTPS:443	This address is for the repository required to download and install required binaries like kubenet and Azure CNI.

Optional recommended FQDN / application rules for AKS clusters

The following FQDN / application rules aren't required, but are recommended for AKS clusters:

Destination FQDN	Port	Use
security.ubuntu.com, azure.archive.ubuntu.com, changelogs.ubuntu.com	HTTP:80	This address lets the Linux cluster nodes download the required security patches and updates.
snapshot.ubuntu.com	HTTPS:443	This address lets the Linux cluster nodes download the required security patches and updates from ubuntu snapshot service.

If you choose to block/not allow these FQDNs, the nodes will only receive OS updates when you do a node image upgrade or cluster upgrade. Keep in mind that node image upgrades also come with updated packages including security fixes.

GPU enabled AKS clusters required FQDN / application rules

Destination FQDN	Port	Use
nvidia.github.io	HTTPS:443	This address is used for correct driver installation and operation on GPU-based nodes.
us.download.nvidia.com	HTTPS:443	This address is used for correct driver installation and operation on GPU-based nodes.
download.docker.com	HTTPS:443	This address is used for correct driver installation and operation on GPU-based nodes.

Windows Server based node pools required FQDN / application rules

Destination FQDN	Port	Use
onegetcdn.azureedge.net, go.microsoft.com	HTTPS:443	To install windows-related binaries
*.mp.microsoft.com, www.msftconnecttest.com, ctldl.windowsupdate.com	HTTP:80	To install windows-related binaries

If you choose to block/not allow these FQDNs, the nodes will only receive OS updates when you do a node image upgrade or cluster upgrade. Keep in mind that Node Image Upgrades also come with updated packages including security fixes.

AKS addons and integrations

Microsoft Defender for Containers

Required FQDN / application rules

FQDN	Port	Use
login.chinacloudapi.cn	HTTPS:443	Required for Active Directory Authentication.
*.ods.opinsights.azure.com	HTTPS:443	Required for Microsoft Defender to upload security events to the cloud.
*.oms.opinsights.azure.com	HTTPS:443	Required to Authenticate with LogAnalytics workspaces.

CSI Secret Store

Required FQDN / application rules

FQDN	Port	Use
vault.azure.cn	HTTPS:443	Required for CSI Secret Store addon pods to talk to Azure KeyVault server.

Azure Monitor for containers

There are two options to provide access to Azure Monitor for containers:

• Allow the Azure Monitor ServiceTag.
• Provide access to the required FQDN/application rules.

Required network rules

Destination Endpoint	Protocol	Port	Use
ServiceTag - AzureMonitor:443	TCP	443	This endpoint is used to send metrics data and logs to Azure Monitor and Log Analytics.

Required FQDN / application rules

FQDN	Port	Use
dc.services.visualstudio.com	HTTPS:443	This endpoint is used by Azure Monitor for Containers Agent Telemetry.
*.ods.opinsights.azure.com	HTTPS:443	This endpoint is used by Azure Monitor for ingesting log analytics data.
*.oms.opinsights.azure.com	HTTPS:443	This endpoint is used by omsagent, which is used to authenticate the log analytics service.
*.monitoring.azure.com	HTTPS:443	This endpoint is used to send metrics data to Azure Monitor.
<cluster-region-name>.ingest.monitor.azure.com	HTTPS:443	This endpoint is used by Azure Monitor managed service for Prometheus metrics ingestion.
<cluster-region-name>.handler.control.monitor.azure.com	HTTPS:443	This endpoint is used to fetch data collection rules for a specific cluster.

Azure operated by 21Vianet required FQDN / application rules

FQDN	Port	Use
dc.services.visualstudio.cn	HTTPS:443	This endpoint is used by Azure Monitor for Containers Agent Telemetry.
*.ods.opinsights.azure.cn	HTTPS:443	This endpoint is used by Azure Monitor for ingesting log analytics data.
*.oms.opinsights.azure.cn	HTTPS:443	This endpoint is used by omsagent, which is used to authenticate the log analytics service.
global.handler.control.monitor.azure.cn	HTTPS:443	This endpoint is used by Azure Monitor for accessing the control service.
<cluster-region-name>.handler.control.monitor.azure.cn	HTTPS:443	This endpoint is used to fetch data collection rules for a specific cluster.

Azure Policy

Required FQDN / application rules

FQDN	Port	Use
data.policy.core.windows.net	HTTPS:443	This address is used to pull the Kubernetes policies and to report cluster compliance status to policy service.
store.policy.core.windows.net	HTTPS:443	This address is used to pull the Gatekeeper artifacts of built-in policies.
dc.services.visualstudio.com	HTTPS:443	Azure Policy add-on that sends telemetry data to applications insights endpoint.

Azure operated by 21Vianet required FQDN / application rules

FQDN	Port	Use
data.policy.azure.cn	HTTPS:443	This address is used to pull the Kubernetes policies and to report cluster compliance status to policy service.
store.policy.azure.cn	HTTPS:443	This address is used to pull the Gatekeeper artifacts of built-in policies.

Next steps

In this article, you learned what ports and addresses to allow if you want to restrict egress traffic for the cluster.

If you want to restrict how pods communicate between themselves and East-West traffic restrictions within cluster see Secure traffic between pods using network policies in AKS.