Send Prometheus data to Azure Monitor by using Microsoft Entra Workload ID authentication

Article
11/08/2024

This article describes how to set up remote write to send data from your Azure Monitor managed Prometheus cluster by using Microsoft Entra Workload ID authentication.

Prerequisites

Prometheus versions greater than v2.48 are required for Microsoft Entra ID application authentication.
Prometheus running in the cluster. This article assumes that the Prometheus cluster is set up by using the kube-prometheus stack, but you can set up Prometheus by using other methods.

Set up remote-write with Microsoft Entra Workload ID

The process to set up Prometheus remote write by using Microsoft Entra Workload ID authentication involves completing the following tasks:

Enable OpenID Connect and note the issuer URL.
Set up mutating admission webhook.
Set up the workload identity.
Create a Microsoft Entra application or user-assigned managed identity and grant permissions.
Assign the Monitoring Metrics Publisher role on the workspace data collection rule to the application.
Create or update your Kubernetes service account Prometheus pod.
Establish federated identity credentials between the identity and the service account issuer and subject.
Deploy a sidecar container to set up remote write.

The tasks are described in the following sections.

Enable OpenID Connect and query the Issuer

To enable OpenID Connect (OIDC) on an AKS cluster, follow the instructions in Create an OpenID Connect provider on AKS.

Once enabled, make a note of the SERVICE_ACCOUNT_ISSUER which is essentially the OIDC issuer URL. To get the OIDC issuer URL, run the az aks show command. Replace the default values for the cluster name and the resource group name.

az aks show --name myAKScluster --resource-group myResourceGroup --query "oidcIssuerProfile.issuerUrl" -o tsv

By default, the issuer is set to use the base URL https://{region}.oic.prod-aks.azure.com, where the value for {region} matches the location the AKS cluster is deployed in.

For other managed clusters (Amazon Elastic Kubernetes Service, and Google Kubernetes Engine), see Managed Clusters - Microsoft Entra Workload ID. For self-managed clusters, see Self-Managed Clusters - Microsoft Entra Workload ID.

Set up mutating admission webhook

Set up mutating admission webhook to keep federated credentials up to date. See Mutating Admission Webhook - Microsoft Entra Workload ID to set up.

Set up the workload identity

To set up the workload identity, export the following environment variables:

# [OPTIONAL] Set this if you're using a Microsoft Entra application
export APPLICATION_NAME="<your application name>"
    
# [OPTIONAL] Set this only if you're using a user-assigned managed identity
export USER_ASSIGNED_IDENTITY_NAME="<your user-assigned managed identity name>"
    
# Environment variables for the Kubernetes service account and federated identity credential
export SERVICE_ACCOUNT_NAMESPACE="<namespace where Prometheus pod is running>"
export SERVICE_ACCOUNT_NAME="<name of service account associated with Prometheus pod. See below for more details>"
export SERVICE_ACCOUNT_ISSUER="<your service account (or OIDC) issuer URL>"

For SERVICE_ACCOUNT_NAME, check to see whether a service account (separate from the default service account) is already associated with the Prometheus pod. Look for the value of serviceaccountName or serviceAccount (deprecated) in the spec of your Prometheus pod. Use this value if it exists. To find the service account associated with the Prometheus pod, run the below kubectl command:

kubectl get pods/<Promethuespodname> -o yaml

If serviceaccountName and serviceAccount don't exist, enter the name of the service account you want to associate with your Prometheus pod.

Create a Microsoft Entra application or user-assigned managed identity and grant permissions

Create a Microsoft Entra application or a user-assigned managed identity and grant permission to publish metrics to Azure Monitor workspace:

# create a Microsoft Entra application
az ad sp create-for-rbac --name "${APPLICATION_NAME}"

# create a user-assigned managed identity if you use a user-assigned managed identity for this article
az identity create --name "${USER_ASSIGNED_IDENTITY_NAME}" --resource-group "${RESOURCE_GROUP}"

Assign the Monitoring Metrics Publisher role on the workspace data collection rule to the application or managed identity

For information about assigning the role, see Assign the Monitoring Metrics Publisher role on the workspace data collection rule to the managed identity.

Create or update your Kubernetes service account Prometheus pod

Often, a Kubernetes service account is created and associated with the pod running the Prometheus container. If you're using the kube-prometheus stack, the code automatically creates the prometheus-kube-prometheus-prometheus service account.

If no Kubernetes service account except the default service account is associated with Prometheus, create a new service account specifically for the pod running Prometheus.

To create the service account, run the following kubectl command:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: service account
metadata:
  annotations:
    azure.workload.identity/client-id: ${APPLICATION_CLIENT_ID:-$USER_ASSIGNED_IDENTITY_CLIENT_ID}
  name: ${SERVICE_ACCOUNT_NAME}
  namespace: ${SERVICE_ACCOUNT_NAMESPACE}
EOF

If a Kubernetes service account other than default service account is associated with your pod, add the following annotation to your service account:

kubectl annotate sa ${SERVICE_ACCOUNT_NAME} -n ${SERVICE_ACCOUNT_NAMESPACE} azure.workload.identity/client-id="${APPLICATION_OR_USER_ASSIGNED_IDENTITY_CLIENT_ID}" -overwrite

If your Microsoft Entra application or user-assigned managed identity isn't in the same tenant as your cluster, add the following annotation to your service account:

kubectl annotate sa ${SERVICE_ACCOUNT_NAME} -n ${SERVICE_ACCOUNT_NAMESPACE} azure.workload.identity/tenant-id="${APPLICATION_OR_USER_ASSIGNED_IDENTITY_TENANT_ID}" -overwrite

Establish federated identity credentials between the identity and the service account issuer and subject

Create federated credentials by using the Azure CLI.

User-assigned managed identity

az identity federated-credential create \
   --name "kubernetes-federated-credential" \
   --identity-name "${USER_ASSIGNED_IDENTITY_NAME}" \
   --resource-group "${RESOURCE_GROUP}" \
   --issuer "${SERVICE_ACCOUNT_ISSUER}" \
   --subject "system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}"

Microsoft Entra application

# Get the ObjectID of the Microsoft Entra app.

export APPLICATION_OBJECT_ID="$(az ad app show --id ${APPLICATION_CLIENT_ID} --query id -otsv)"

# Add a federated identity credential.

cat <<EOF > params.json
{
  "name": "kubernetes-federated-credential",
  "issuer": "${SERVICE_ACCOUNT_ISSUER}",
  "subject": "system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}",
  "description": "Kubernetes service account federated credential",
  "audiences": [
    "api://AzureADTokenExchange"
  ]
}
EOF

az ad app federated-credential create --id ${APPLICATION_OBJECT_ID} --parameters @params.json

Deploy a sidecar container to set up remote write

Important

The Prometheus pod must have the following label: azure.workload.identity/use: "true"

The remote write sidecar container requires the following environment values:

INGESTION_URL: The metrics ingestion endpoint that's shown on the Overview page for the Azure Monitor workspace
LISTENING_PORT: 8081 (any port is supported)
IDENTITY_TYPE: workloadIdentity

Copy the following YAML and save it to a file. The YAML uses port 8081 as the listening port. If you use a different port, modify that value in the YAML.

prometheus:
  prometheusSpec:
    externalLabels:
          cluster: <AKS-CLUSTER-NAME>
    podMetadata:
        labels:
            azure.workload.identity/use: "true"
    ## https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write    
    remoteWrite:
    - url: 'http://localhost:8081/api/v1/write'

    containers:
    - name: prom-remotewrite
      image: <CONTAINER-IMAGE-VERSION>
      imagePullPolicy: Always
      ports:
        - name: rw-port
          containerPort: 8081
      env:
      - name: INGESTION_URL
        value: <INGESTION_URL>
      - name: LISTENING_PORT
        value: '8081'
      - name: IDENTITY_TYPE
        value: workloadIdentity

Replace the following values in the YAML:

Value	Description
`<CLUSTER-NAME>`	The name of your AKS cluster.
`<CONTAINER-IMAGE-VERSION>`	`mcr.microsoft.com/azuremonitor/containerinsights/ciprod/prometheus-remote-write/images:prom-remotewrite-20240617.1` The remote write container image version.
`<INGESTION-URL>`	The value for Metrics ingestion endpoint from the Overview page for the Azure Monitor workspace.

Use Helm to apply the YAML file and update your Prometheus configuration:

# set a context to your cluster 
az aks get-credentials -g <aks-rg-name> -n <aks-cluster-name> 

# use Helm to update your remote write config 
helm upgrade -f <YAML-FILENAME>.yml prometheus prometheus-community/kube-prometheus-stack -namespace <namespace where Prometheus pod resides>

Verification and troubleshooting

For verification and troubleshooting information, see Troubleshooting remote write and Azure Monitor managed service for Prometheus remote write.