Migrating Your Teleport Cloud Metrics Bot to WorkloadIdentity

Who this is for: Teleport Cloud customers who already scrape Teleport Cloud Metrics using the tbot + spiffe-svid setup.

Why: The legacy Workload Identity configuration (spec.allow.spiffe on roles and the spiffe-svid output in tbot) is deprecated and will be removed in Teleport v19.0.0. Migrate before your cluster upgrades to v19, or tbot will stop issuing the metrics client certificate and Prometheus scrapes will fail.

What changes:

The SPIFFE ID must remain exactly /_teleport-cloud/metrics-exporter/client; the metrics endpoint authorizes on it.

Step 1. Create the WorkloadIdentity resource

Save as metrics-workload-identity.yaml:

kind: workload_identity
version: v1
metadata:
  name: teleport-cloud-metrics-client
  labels:
    teleport-cloud-metrics: client
spec:
  spiffe:
    id: /_teleport-cloud/metrics-exporter/client

Run:

tctl create -f metrics-workload-identity.yaml

Step 2. Update the bot's role

Add the new permissions to the existing teleport-cloud-metrics-client role. Keep the legacy spiffe block for now so the running bot keeps working during the switch.

kind: role
version: v6
metadata:
  name: teleport-cloud-metrics-client
spec:
  allow:
    # New: grants access to the WorkloadIdentity created in Step 1
    workload_identity_labels:
      teleport-cloud-metrics: ["client"]
    rules:
    - resources: [workload_identity]
      verbs: [list, read]
    # Legacy: remove in Step 5
    spiffe:
    - path: /_teleport-cloud/metrics-exporter/client
  deny: {}

Update the role:

tctl create -f metrics-role.yaml --force

Step 3. Update the tbot output

In your tbot configuration (ConfigMap or Helm values), replace the spiffe-svid output with a workload-identity-x509 output. Keep the same secret name so Prometheus needs no changes.

Before:

outputs:
  - type: spiffe-svid
    svid:
      path: /_teleport-cloud/metrics-exporter/client
    destination:
      type: kubernetes_secret
      name: teleport-cloud-metrics-client

After:

outputs:
  - type: workload-identity-x509
    selector:
      name: teleport-cloud-metrics-client
    destination:
      type: kubernetes_secret
      name: teleport-cloud-metrics-client

Apply the change and restart tbot (e.g. kubectl rollout restart deployment/<tbot-deployment>).

Step 4. Verify

1. Check tbot logs for successful issuance and no permission errors.
2. Confirm the secret was refreshed and inspect its keys:

  kubectl get secret teleport-cloud-metrics-client -o jsonpath='{.data}' | jq 'keys'

Both the old and new outputs write svid.pem, svid_key.pem, and svid_bundle.pem. Make sure the key names in your Prometheus ScrapeConfig tlsConfig match what is actually in the secret.

3. Confirm the certificate's SPIFFE ID:

  kubectl get secret teleport-cloud-metrics-client -o jsonpath='{.data.svid\.pem}' \
     | base64 -d | openssl x509 -noout -ext subjectAltName

Expect URI:spiffe://<cluster name>/_teleport-cloud/metrics-exporter/client.

4. In Prometheus, confirm the teleport-cloud target is UP and teleport_cloud_last_updated_seconds is advancing.

Step 5. Remove the legacy permission

Once scrapes are healthy, delete the spiffe: block from the role and reapply:

tctl create -f metrics-role.yaml --force

Re-check the Prometheus target after the next certificate renewal.

Troubleshooting

• access denied / no WorkloadIdentity found: the labels on the workload_identity resource don't match workload_identity_labels on the role, or the read/list rule is missing.
• unknown output type workload-identity-x509: your tbot version predates the new configuration; upgrade tbot to match your cluster version.
• Target DOWN with TLS errors: secret key names don't match the ScrapeConfig, or Prometheus Operator lacks RBAC to read the secret.

References

• Teleport Cloud Metrics
• WorkloadIdentity Configuration Resource migration
• WorkloadIdentity Resource reference
• tbot configuration reference