Set up Secrets Store CSI Driver to enable NGINX Ingress Controller with TLS

Article
06/03/2024

This article walks you through the process of securing an NGINX Ingress Controller with TLS with an Azure Kubernetes Service (AKS) cluster and an Azure Key Vault (AKV) instance. For more information, see TLS in Kubernetes.

You can import the ingress TLS certificate to the cluster using one of the following methods:

Application: The application deployment manifest declares and mounts the provider volume. Only when you deploy the application is the certificate made available in the cluster. When you remove the application, the secret is also removed. This scenario fits development teams responsible for the application’s security infrastructure and its integration with the cluster.
Ingress Controller: The ingress deployment is modified to declare and mount the provider volume. The secret is imported when ingress pods are created. The application’s pods have no access to the TLS certificate. This scenario fits scenarios where one team (for example, IT) manages and creates infrastructure and networking components (including HTTPS TLS certificates) and other teams manage application lifecycle.

Prerequisites

If you don't have an Azure subscription, create a trial subscription before you begin.
Before you start, ensure your Azure CLI version is >= 2.30.0, or install the latest version.
An AKS cluster with the Secrets Store CSI Driver configured.
An Azure Key Vault instance.

Generate a TLS certificate

Generate a TLS certificate using the following command.

export CERT_NAME=aks-ingress-cert
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
    -out aks-ingress-tls.crt \
    -keyout aks-ingress-tls.key \
    -subj "/CN=demo.azure.com/O=aks-ingress-tls"

Import the certificate to AKV

Export the certificate to a PFX file using the following command.

export AKV_NAME="[YOUR AKV NAME]"
openssl pkcs12 -export -in aks-ingress-tls.crt -inkey aks-ingress-tls.key  -out $CERT_NAME.pfx
# skip Password prompt

Import the certificate using the az keyvault certificate import command.

az keyvault certificate import --vault-name $AKV_NAME --name $CERT_NAME --file $CERT_NAME.pfx

Deploy a SecretProviderClass

Export a new namespace using the following command.
```
export NAMESPACE=ingress-basic
```
Create the namespace using the kubectl create namespace command.
```
kubectl create namespace $NAMESPACE
```

Select a method to provide an access identity and configure your SecretProviderClass YAML accordingly.

Be sure to use objectType=secret, which is the only way to obtain the private key and the certificate from AKV.
Set kubernetes.io/tls as the type in your secretObjects section.

See the following example of what your SecretProviderClass might look like:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-tls
spec:
  provider: azure
  secretObjects:                            # secretObjects defines the desired state of synced K8s secret objects
    - secretName: ingress-tls-csi
      type: kubernetes.io/tls
      data: 
        - objectName: $CERT_NAME
          key: tls.key
        - objectName: $CERT_NAME
          key: tls.crt
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: <client id>
    keyvaultName: $AKV_NAME                 # the name of the AKV instance
    objects: |
      array:
        - |
          objectName: $CERT_NAME
          objectType: secret
    tenantId: $TENANT_ID                    # the tenant ID of the AKV instance

Apply the SecretProviderClass to your Kubernetes cluster using the kubectl apply command.
```
kubectl apply -f secretProviderClass.yaml -n $NAMESPACE
```

Deploy the ingress controller

Add the official ingress chart repository

Add the official ingress chart repository using the following helm commands.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

Configure and deploy the NGINX ingress

Depending on your scenario, you can choose to bind the certificate to either the application or to the ingress controller. Follow the below instructions according to your selection:

Bind certificate to application

Bind the certificate to the application using the helm install command. The application’s deployment references the Secrets Store CSI Driver's Azure Key Vault provider.

helm install ingress-nginx/ingress-nginx --generate-name \
    --namespace $NAMESPACE \
    --set controller.replicaCount=2 \
    --set controller.nodeSelector."kubernetes\.io/os"=linux \
    --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz \
    --set defaultBackend.nodeSelector."kubernetes\.io/os"=linux

Bind certificate to ingress controller

Bind the certificate to the ingress controller using the helm install command. The ingress controller’s deployment references the Secrets Store CSI Driver's Azure Key Vault provider.

Note

If not using Microsoft Entra pod-managed identity as your method of access, remove the line with --set controller.podLabels.aadpodidbinding=$AAD_POD_IDENTITY_NAME .
Also, binding the SecretProviderClass to a pod is required for the Secrets Store CSI Driver to mount it and generate the Kubernetes secret. See Sync mounted content with a Kubernetes secret .

helm install ingress-nginx/ingress-nginx --generate-name \
    --namespace $NAMESPACE \
    --set controller.replicaCount=2 \
    --set controller.nodeSelector."kubernetes\.io/os"=linux \
    --set defaultBackend.nodeSelector."kubernetes\.io/os"=linux \
    --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz \
    --set controller.podLabels.aadpodidbinding=$AAD_POD_IDENTITY_NAME \
    -f - <<EOF
controller:
  extraVolumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "azure-tls"
  extraVolumeMounts:
      - name: secrets-store-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
EOF

Verify the Kubernetes secret was created using the kubectl get secret command.

kubectl get secret -n $NAMESPACE

NAME                                             TYPE                                  DATA   AGE
ingress-tls-csi                                  kubernetes.io/tls                     2      1m34s

Deploy the application

Again, the instructions change slightly depending on your scenario. Follow the instructions corresponding to the scenario you selected.

Deploy the application using an application reference

Create a file named aks-helloworld-one.yaml with the following content.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aks-helloworld-one  
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aks-helloworld-one
  template:
    metadata:
      labels:
        app: aks-helloworld-one
    spec:
      containers:
      - name: aks-helloworld-one
        image: mcr.azk8s.cn/azuredocs/aks-helloworld:v1
        ports:
        - containerPort: 80
        env:
        - name: TITLE
          value: "Welcome to Azure Kubernetes Service (AKS)"
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "azure-tls"
---
apiVersion: v1
kind: Service
metadata:
  name: aks-helloworld-one  
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: aks-helloworld-one

Create a file named aks-helloworld-two.yaml with the following content.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aks-helloworld-two  
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aks-helloworld-two
  template:
    metadata:
      labels:
        app: aks-helloworld-two
    spec:
      containers:
      - name: aks-helloworld-two
        image: mcr.azk8s.cn/azuredocs/aks-helloworld:v1
        ports:
        - containerPort: 80
        env:
        - name: TITLE
          value: "AKS Ingress Demo"
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "azure-tls"
---
apiVersion: v1
kind: Service
metadata:
  name: aks-helloworld-two
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: aks-helloworld-two

Apply the YAML files to your cluster using the kubectl apply command.

kubectl apply -f aks-helloworld-one.yaml -n $NAMESPACE
kubectl apply -f aks-helloworld-two.yaml -n $NAMESPACE

Verify the Kubernetes secret was created using the kubectl get secret command.

kubectl get secret -n $NAMESPACE

NAME                                             TYPE                                  DATA   AGE
ingress-tls-csi                                  kubernetes.io/tls                     2      1m34s

Deploy the application using an ingress controller reference

Create a file named aks-helloworld-one.yaml with the following content.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aks-helloworld-one  
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aks-helloworld-one
  template:
    metadata:
      labels:
        app: aks-helloworld-one
    spec:
      containers:
      - name: aks-helloworld-one
        image: mcr.azk8s.cn/azuredocs/aks-helloworld:v1
        ports:
        - containerPort: 80
        env:
        - name: TITLE
          value: "Welcome to Azure Kubernetes Service (AKS)"
---
apiVersion: v1
kind: Service
metadata:
  name: aks-helloworld-one
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: aks-helloworld-one

Create a file named aks-helloworld-two.yaml with the following content.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aks-helloworld-two  
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aks-helloworld-two
  template:
    metadata:
      labels:
        app: aks-helloworld-two
    spec:
      containers:
      - name: aks-helloworld-two
        image: mcr.azk8s.cn/azuredocs/aks-helloworld:v1
        ports:
        - containerPort: 80
        env:
        - name: TITLE
          value: "AKS Ingress Demo"
---
apiVersion: v1
kind: Service
metadata:
  name: aks-helloworld-two  
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: aks-helloworld-two

Apply the YAML files to your cluster using the kubectl apply command.

kubectl apply -f aks-helloworld-one.yaml -n $NAMESPACE
kubectl apply -f aks-helloworld-two.yaml -n $NAMESPACE

Deploy an ingress resource referencing the secret

We can now deploy a Kubernetes ingress resource referencing the secret.

Create a file name hello-world-ingress.yaml with the following content.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-tls
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - demo.azure.com
    secretName: ingress-tls-csi
  rules:
  - host: demo.azure.com
    http:
      paths:
      - path: /hello-world-one(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: aks-helloworld-one
            port:
              number: 80
      - path: /hello-world-two(/|$)(.*)
        pathType: Prefix      
        backend:
          service:
            name: aks-helloworld-two
            port:
              number: 80
      - path: /(.*)
        pathType: Prefix      
        backend:
          service:
            name: aks-helloworld-one
            port:
              number: 80

Make note of the tls section referencing the secret created earlier and apply the file to your cluster using the kubectl apply command.
```
kubectl apply -f hello-world-ingress.yaml -n $NAMESPACE
```

Obtain the external IP address of the ingress controller

Get the external IP address for the ingress controller using the kubectl get service command.

kubectl get service --namespace $NAMESPACE --selector app.kubernetes.io/name=ingress-nginx

NAME                                       TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE
nginx-ingress-1588032400-controller        LoadBalancer   10.0.255.157   EXTERNAL_IP      80:31293/TCP,443:31265/TCP   19m
nginx-ingress-1588032400-default-backend   ClusterIP      10.0.223.214   <none>           80/TCP                       19m

Test ingress secured with TLS

Verify your ingress is properly configured with TLS using the following curl command. Make sure you use the external IP from the previous step.
```
curl -v -k --resolve demo.azure.com:443:EXTERNAL_IP https://demo.azure.com
```
Since another path wasn't provided with the address, the ingress controller defaults to the / route. The first demo application is returned, as shown in the following condensed example output:
```
[...]
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <link rel="stylesheet" type="text/css" href="/static/default.css">
    <title>Welcome to Azure Kubernetes Service (AKS)</title>
[...]
```
The -v parameter in the curl command outputs verbose information, including the TLS certificate received. Halfway through your curl output, you can verify your own TLS certificate was used. The -k parameter continues loading the page even though we're using a self-signed certificate. The following example shows the issuer: CN=demo.azure.com; O=aks-ingress-tls certificate was used:
```
[...]
 * Server certificate:
 *  subject: CN=demo.azure.com; O=aks-ingress-tls
 *  start date: Oct 22 22:13:54 2021 GMT
 *  expire date: Oct 22 22:13:54 2022 GMT
 *  issuer: CN=demo.azure.com; O=aks-ingress-tls
 *  SSL certificate verify result: self signed certificate (18), continuing anyway.
[...]
```

Add /hello-world-two path to the address, such as https://demo.azure.com/hello-world-two, and verify the second demo application is properly configured.

curl -v -k --resolve demo.azure.com:443:EXTERNAL_IP https://demo.azure.com/hello-world-two

The second demo application with the custom title is returned, as shown in the following condensed example output:

[...]
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <link rel="stylesheet" type="text/css" href="/static/default.css">
    <title>AKS Ingress Demo</title>
[...]