Install AGIC by using an existing Application Gateway deployment

Article
11/08/2024

The Application Gateway Ingress Controller (AGIC) is a pod within your Azure Kubernetes Service (AKS) cluster. AGIC monitors the Kubernetes Ingress resources. It creates and applies an Azure Application Gateway configuration based on the status of the Kubernetes cluster.

Prerequisites

This article assumes that you already installed the following tools and infrastructure:

An AKS cluster with Azure Container Networking Interface (CNI).
Application Gateway v2 in the same virtual network as the AKS cluster.
Microsoft Entra Workload ID configured for your AKS cluster.

Add the Helm repository

Helm is a package manager for Kubernetes. You use it to install the application-gateway-kubernetes-ingress package.

Install Helm. Run the following commands to add the AGIC Helm repository for an AKS cluster that's enabled with Kubernetes role-based access control (RBAC):

kubectl create serviceaccount --namespace kube-system tiller-sa
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller-sa
helm init --tiller-namespace kube-system --service-account tiller-sa

Back up the Application Gateway deployment

Before you install AGIC, back up your Application Gateway deployment's configuration:

In the Azure portal, go to your Application Gateway deployment.
In the Automation section, select Export template and then select Download.

The downloaded .zip file contains JSON templates, Bash scripts, and PowerShell scripts that you can use to restore Application Gateway, if a restoration becomes necessary.

Set up an identity for Resource Manager authentication

AGIC communicates with the Kubernetes API server and Azure Resource Manager. It requires an identity to access these APIs. You can use either Microsoft Entra Workload ID or a service principal.

Set up Microsoft Entra Workload ID

Microsoft Entra Workload ID is an identity that you assign to a software workload. This identity enables your AKS pod to authenticate with other Azure resources.

For this configuration, you need authorization for the AGIC pod to make HTTP requests to Azure Resource Manager.

Use the Azure CLI az account set command to set a specific subscription to be the current active subscription:
```
az account set --subscription "subscriptionID"
```
Then use the az identity create command to create a managed identity. You must create the identity in the node resource group. The node resource group is assigned a name by default, such as MC_myResourceGroup_myAKSCluster_chinanorth2.
```
az identity create --name "userAssignedIdentityName" --resource-group "resourceGroupName" --location "location" --subscription "subscriptionID"
```

For the role assignment, run the following command to identify the principalId value for the newly created identity:

$resourceGroup="resource-group-name"
$identityName="identity-name"
az identity list -g $resourceGroup --query "[?name == '$identityName'].principalId | [0]" -o tsv

Grant the identity Contributor access to your Application Gateway deployment. You need the ID of the Application Gateway deployment, which looks like /subscriptions/A/resourceGroups/B/providers/Microsoft.Network/applicationGateways/C.

First, get the list of Application Gateway IDs in your subscription by running the following command:

az network application-gateway list --query '[].id'

To assign the identity Contributor access, run the following command:

$resourceGroup="resource-group-name"
$identityName="identity-Name"
# Get the Application Gateway ID
$AppGatewayID=$(az network application-gateway list --query '[].id' -o tsv)
$role="contributor"
# Get the principal ID for the user-assigned identity
$principalId=$(az identity list -g $resourceGroup --query "[?name == '$identityName'].principalId | [0]" -o tsv)
az role assignment create --assignee $principalId --role $role --scope $AppGatewayID

Grant the identity Reader access to the Application Gateway resource group. The resource group ID looks like /subscriptions/A/resourceGroups/B. You can get all resource groups by running az group list --query '[].id'.

$resourceGroup="resource-group-name"
$identityName="identity-Name"
# Get the Application Gateway resource group
$AppGatewayResourceGroup=$(az network application-gateway list --query '[].resourceGroup' -o tsv)
# Get the Application Gateway resource group ID
$AppGatewayResourceGroupID=$(az group show --name $AppGatewayResourceGroup --query id -o tsv)
$role="Reader"
# Get the principal ID for the user-assigned identity
$principalId=$(az identity list -g $resourceGroup --query "[?name == '$identityName'].principalId | [0]" -o tsv)
# Assign the Reader role to the user-assigned identity at the resource group scope
az role assignment create --role $role --assignee $principalId  --scope $AppGatewayResourceGroupID

Note

Make sure the identity that AGIC uses has the Microsoft.Network/virtualNetworks/subnets/join/action permission delegated to the subnet where Application Gateway is deployed. If you didn't define a custom role that has this permission, you can use the built-in Network Contributor role.

Set up a service principal

It's also possible to provide AGIC access to Azure Resource Manager by using a Kubernetes secret:

Create an Active Directory service principal and encode it with Base64. The Base64 encoding is required for the JSON blob to be saved to Kubernetes.
```
az ad sp create-for-rbac --role Contributor --sdk-auth | base64 -w0
```
Add the Base64-encoded JSON blob to the helm-config.yaml file. The helm-config.yaml file configures AGIC.
```
armAuth:
    type: servicePrincipal
    secretJSON: <Base64-Encoded-Credentials>
```

Deploy the AGIC add-on

Create a deployment manifest for the ingress controller

---
# file: pet-supplies-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pet-supplies-ingress
spec:
  ingressClassName: azure-application-gateway
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: store-front
            port:
              number: 80
      - path: /order-service
        pathType: Prefix
        backend:
          service:
            name: order-service
            port:
              number: 3000
      - path: /product-service
        pathType: Prefix
        backend:
          service:
            name: product-service
            port:
              number: 3002

Deploy the ingress controller

$namespace="namespace"
$file="pet-supplies-ingress.yaml"
kubectl apply -f $file -n $namespace

Install the ingress controller as a Helm chart

In the first few steps, we installed Helm's Tiller on your Kubernetes cluster. Install the AGIC Helm package:

Perform a Helm update:
```
helm repo update
```

Download helm-config.yaml:

wget https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/sample-helm-config.yaml -O helm-config.yaml

Or copy the following YAML file:

# This file contains the essential configs for the ingress controller helm chart

# Verbosity level of the App Gateway Ingress Controller
verbosityLevel: 3

################################################################################
# Specify which application gateway the ingress controller must manage
#
appgw:
    subscriptionId: <subscriptionId>
    resourceGroup: <resourceGroupName>
    name: <applicationGatewayName>
    environment: AzureChinaCloud

    # Setting appgw.shared to "true" creates an AzureIngressProhibitedTarget CRD.
    # This prohibits AGIC from applying config for any host/path.
    # Use "kubectl get AzureIngressProhibitedTargets" to view and change this.
    shared: false

################################################################################
# Specify which kubernetes namespace the ingress controller must watch
# Default value is "default"
# Leaving this variable out or setting it to blank or empty string would
# result in Ingress Controller observing all accessible namespaces.
#
# kubernetes:
#   watchNamespace: <namespace>

################################################################################
# Specify the authentication with Azure Resource Manager
#
# Two authentication methods are available:
# - Option 1: Azure-AD-workload-identity
armAuth:
    type: workloadIdentity
    identityClientID:  <identityClientId>

## Alternatively you can use Service Principal credentials
# armAuth:
#    type: servicePrincipal
#    secretJSON: <<Generate this value with: "az ad sp create-for-rbac --role Contributor --sdk-auth | base64 -w0" >>

################################################################################
# Specify if the cluster is Kubernetes RBAC enabled or not
rbac:
    enabled: false # true/false

# Specify aks cluster related information. THIS IS BEING DEPRECATED.
aksClusterConfiguration:
    apiServerAddress: <aks-api-server-address>

Edit helm-config.yaml and fill in the values for appgw and armAuth.

Note

<identity-client-id> is a property of the Microsoft Entra Workload ID value that you set up in the previous section. You can retrieve this information by running the following command: az identity show -g <resourcegroup> -n <identity-name>. In that command, <resourcegroup> is the resource group that hosts the infrastructure resources related to the AKS cluster, Application Gateway, and the managed identity.

Install the Helm chart with the helm-config.yaml configuration from the previous step:

helm install agic-controller oci://mcr.microsoft.com/azure-application-gateway/charts/ingress-azure --version 1.7.5 -f helm-config.yaml

Alternatively, you can combine helm-config.yaml and the Helm command in one step:

helm install oci://mcr.microsoft.com/azure-application-gateway/charts/ingress-azure \
     --name agic-controller \
     --version 1.7.5 \
     --namespace default \
     --debug \
     --set appgw.name=applicationgatewayABCD \
     --set appgw.resourceGroup=your-resource-group \
     --set appgw.subscriptionId=subscription-uuid \
     --set appgw.environment=AzureChinaCloud \
     --set appgw.shared=false \
     --set armAuth.type=servicePrincipal \
     --set armAuth.secretJSON=$(az ad sp create-for-rbac --role Contributor --sdk-auth | base64 -w0) \
     --set rbac.enabled=true \
     --set verbosityLevel=3 \
     --set kubernetes.watchNamespace=default \
     --set aksClusterConfiguration.apiServerAddress=aks-abcdefg.hcp.chinanorth2.azmk8s.io

Check the log of the newly created pod to verify that it started properly.

To understand how you can expose an AKS service to the internet over HTTP or HTTPS by using an Azure Application Gateway deployment, see this how-to guide.

Set up a shared Application Gateway deployment

By default, AGIC assumes full ownership of the Application Gateway deployment that it's linked to. AGIC version 0.8.0 and later can share a single Application Gateway deployment with other Azure components. For example, you could use the same Application Gateway deployment for an app that's hosted on an Azure virtual machine scale set and an AKS cluster.

Example scenario

Let's look at an imaginary Application Gateway deployment that manages traffic for two websites:

dev.contoso.com: Hosted on a new AKS cluster by using Application Gateway and AGIC.
prod.contoso.com: Hosted on a virtual machine scale set.

With default settings, AGIC assumes 100% ownership of the Application Gateway deployment that it's pointed to. AGIC overwrites all of the App Gateway configuration. If you manually create a listener for prod.contoso.com on Application Gateway without defining it in the Kubernetes ingress, AGIC deletes the prod.contoso.com configuration within seconds.

To install AGIC and also serve prod.contoso.com from the machines that use the virtual machine scale set, you must constrain AGIC to configuring dev.contoso.com only. You facilitate this constraint by instantiating the following custom resource definition (CRD):

cat <<EOF | kubectl apply -f -
apiVersion: "appgw.ingress.k8s.io/v1"
kind: AzureIngressProhibitedTarget
metadata:
  name: prod-contoso-com
spec:
  hostname: prod.contoso.com
EOF

The preceding command creates an AzureIngressProhibitedTarget object. This object makes AGIC (version 0.8.0 and later) aware of the existence of the Application Gateway configuration for prod.contoso.com. This object also explicitly instructs AGIC to avoid changing any configuration related to that host name.

Enable a shared Application Gateway deployment by using a new AGIC installation

To limit AGIC (version 0.8.0 and later) to a subset of the Application Gateway configuration, modify the helm-config.yaml template. In the appgw: section, add a shared key and set it to true:

appgw:
    subscriptionId: <subscriptionId>    # existing field
    resourceGroup: <resourceGroupName>  # existing field
    name: <applicationGatewayName>      # existing field
    environment: AzureChinaCloud        # existing field
    shared: true                        # Add this field to enable shared Application Gateway

Apply the Helm changes:

Ensure that the AzureIngressProhibitedTarget CRD is installed:

kubectl apply -f https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/7b55ad194e7582c47589eb9e78615042e00babf3/crds/AzureIngressProhibitedTarget-v1-CRD-v1.yaml

Update Helm:

helm upgrade \
    --recreate-pods \
    -f helm-config.yaml \
    agic-controller
    oci://mcr.microsoft.com/azure-application-gateway/charts/ingress-azure

As a result, your AKS cluster has a new instance of AzureIngressProhibitedTarget called prohibit-all-targets:

kubectl get AzureIngressProhibitedTargets prohibit-all-targets -o yaml

The prohibit-all-targets object prohibits AGIC from changing the configuration for any host and path. Helm installed with appgw.shared=true deploys AGIC, but it doesn't make any changes to Application Gateway.

Broaden permissions

Because Helm with appgw.shared=true and the default prohibit-all-targets blocks AGIC from applying a configuration, you must broaden AGIC permissions:

Create a new YAML file named AzureIngressProhibitedTarget with the following snippet that contains your specific setup:

cat <<EOF | kubectl apply -f -
apiVersion: "appgw.ingress.k8s.io/v1"
kind: AzureIngressProhibitedTarget
metadata:
  name: your-custom-prohibitions
spec:
  hostname: your.own-hostname.com
EOF

Now that you've created your own custom prohibition, you can delete the default one, which is too broad:
```
kubectl delete AzureIngressProhibitedTarget prohibit-all-targets
```

Enable a shared Application Gateway deployment for an existing AGIC installation

Assume that you already have a working AKS cluster and an Application Gateway deployment, and you configured AGIC in your cluster. You have an Ingress for prod.contoso.com and are successfully serving traffic for it from the cluster.

You want to add staging.contoso.com to your existing Application Gateway deployment, but you need to host it on a virtual machine. You're going to reuse the existing Application Gateway deployment and manually configure a listener and backend pools for staging.contoso.com. But manually tweaking the Application Gateway configuration (by using the Azure portal, Resource Manager APIs, or Terraform) would conflict with AGIC's assumptions of full ownership. Shortly after you apply changes, AGIC overwrites or deletes them.

You can prohibit AGIC from making changes to a subset of the configuration:

Create a new YAML file named AzureIngressProhibitedTarget by using the following snippet:

cat <<EOF | kubectl apply -f -
apiVersion: "appgw.ingress.k8s.io/v1"
kind: AzureIngressProhibitedTarget
metadata:
  name: manually-configured-staging-environment
spec:
  hostname: staging.contoso.com
EOF

View the newly created object:

kubectl get AzureIngressProhibitedTargets

Modify the Application Gateway configuration from the Azure portal. For example, add listeners, routing rules, and backends. The new object that you created (manually-configured-staging-environment) prohibits AGIC from overwriting the Application Gateway configuration related to staging.contoso.com.