Orchestrate updates across multiple clusters using Azure Kubernetes Fleet Manager
Platform admins managing Kubernetes fleets with large number of clusters often have problems with staging updates across multiple clusters in a safe and predictable way. To address this pain point, Kubernetes Fleet Manager (Fleet) allows you to orchestrate updates across multiple clusters using update runs, stages, groups, and strategies.
Prerequisites
Read the conceptual overview of this feature, which provides an explanation of update strategies, runs, stages, and groups references in this document.
You need a Fleet resource with one or more member clusters. If you don't have one, follow the quickstart to create a Fleet resource and join Azure Kubernetes Service (AKS) clusters as members.
Set the following environment variables:
export GROUP=<resource-group> export FLEET=<fleet-name>If you're using the Azure CLI, you need Azure CLI version 2.53.1 or later installed. To install or upgrade, see Install the Azure CLI.
You need the
fleetAzure CLI extension, which you can install using theaz extension addcommand.az extension add --name fleetRun the
az extension updatecommand to update to the latest version of the extension released:az extension update --name fleet
Creating update runs
Note
Update runs honor the planned maintenance windows that you set at the AKS cluster level. For more information, see planned maintenance across multiple member clusters which explains how update runs handle member clusters that have been configured with planned maintenance windows.
Update run supports two options for the cluster upgrade sequence:
- One by one: If you don't care about controlling the cluster upgrade sequence,
one-by-oneprovides a simple approach to upgrade all member clusters of the fleet in sequence one at a time. - Control sequence of clusters using update groups and stages: If you want to control the cluster upgrade sequence, you can structure member clusters in update groups and update stages. You can store this sequence as a template in the form of update strategy. You can create update runs later using the update strategies instead of defining the sequence every time you need to create an update run.
Update all clusters one by one
In the Azure portal, navigate to your Azure Kubernetes Fleet Manager resource.
From the service menu, under Settings, select Multi-cluster update > Create a run.
Enter a name for the update run, and then select One by one for the upgrade type.
Select one of the following options for the Upgrade scope:
- Kubernetes version for both control plane and node pools
- Kubernetes version for only control plane of the cluster
- Node image version only
Select one of the following options for the Node image:
- Latest image: Updates every AKS cluster in the update run to the latest image available for that cluster in its region.
- Consistent image: As it's possible for an update run to have AKS clusters across multiple regions where the latest available node images can be different (check release tracker for more information). The update run picks the latest common image across all these regions to achieve consistency.
Select Create to create the update run.
Assign clusters to update groups and stages
Update groups and stages provide more control over the sequence that update runs follow when you're updating the clusters. Within an update stage, updates are applied to all the different update groups in parallel. Within an update group, member clusters update sequentially.
You can assign a member cluster to a specific update group in one of two ways:
- Assign to group when adding member cluster to the fleet.
- Assign an existing fleet member to an update group.
Assign to group when adding member cluster to the fleet
In the Azure portal, navigate to your Azure Kubernetes Fleet Manager resource.
From the service menu, under Settings, select Member clusters > Add.
Select the cluster that you want to add, and then select Next: Review + add.
Enter the name of the update group that you want to assign the cluster to, and then select Add.
Assign an existing fleet member to an update group
In the Azure portal, navigate to your Azure Kubernetes Fleet Manager resource.
From the service menu, under Settings, select Member clusters.
Select the cluster or clusters that you want to assign to an update group, and then select Assign update group
Enter the name of the update group that you want to assign the cluster to, and then select Assign.
Note
A fleet member can only be a part of one update group, but an update group can have multiple fleet members assigned to it. An update group itself is not a separate resource type. Update groups are only strings representing references from the fleet members. So, if all fleet members with references to a common update group are deleted, that specific update group will cease to exist as well.
Define an update run and stages
You can define an update run using update stages to sequentially order the application of updates to different update groups. For example, a first update stage might update test environment member clusters, and a second update stage would then update production environment member clusters. You can also specify a wait time between the update stages.
In the Azure portal, navigate to your Azure Kubernetes Fleet Manager resource.
From the service menu, under Settings, select Multi-cluster update > Create a run.
Enter a name for the update run, and then select Stages for the update sequence type.
Select Create stage, and then enter a name for the stage and the wait time between stages.
Select the update groups that you want to include in this stage. You can also specify the order of the update groups if you want to update them in a specific sequence. When you're done, select Create.
Select one of the following options for the Upgrade scope:
- Kubernetes version for both control plane and node pools
- Kubernetes version for only control plane of the cluster
- Node image version only
Select one of the following options for the Node image:
- Latest image: Updates every AKS cluster in the update run to the latest image available for that cluster in its region.
- Consistent image: As it's possible for an update run to have AKS clusters across multiple regions where the latest available node images can be different (check release tracker for more information). The update run picks the latest common image across all these regions to achieve consistency.
Select Create to create the update run.
Specifying stages and their order every time when creating an update run can get repetitive and cumbersome. Update strategies simplify this process by allowing you to store templates for update runs. For more information, see update strategy creation and usage.
In the Multi-cluster update menu, select the update run, and then select Start.
Create an update run using update strategies
Creating an update run requires you to specify the stages, groups, order each time. Update strategies simplify this process by allowing you to store templates for update runs.
Note
It's possible to create multiple update runs with unique names from the same update strategy.
You can create an update strategy using one of the following methods:
- Save an update strategy while creating an update run using the Azure portal.
- Create a new update strategy and then reference it when creating an update run.
Save an update strategy while creating an update run
Save an update strategy while creating an update run in the Azure portal:
Create a new update strategy and reference it when creating an update run
Navigate to the Multi-cluster update page, and then select Strategies > Create a strategy:
Configure the update strategy details, and then select Create.
Reference the update strategy when creating new subsequent update runs:
Manage update runs
The following sections explain how to manage an update run using the Azure portal and Azure CLI.
On the Multi-cluster update page of the fleet resource, you can Start an update run that's either in Not started or Failed state:
On the Multi-cluster update page of the fleet resource, you can Stop a currently Running update run:
Within any update run in the Not Started, Failed, or Running state, you can select any Stage and Skip the upgrade:
You can similarly skip the upgrade at the update group or member cluster level too.
For more information, see the conceptual overview on the update run states and skip behavior on runs/stages/groups.
Next steps
Learn more about Azure Kubernetes Fleet Manager.