Teleport Cloud Metrics

Teleport Cloud has an optional metrics service which can be enabled by contacting support. Once enabled, additional configuration will be needed for your Prometheus client to scrape metrics.

Metrics

Metrics exported by Teleport Cloud are derived by native Teleport Metrics and other metrics internal to Teleport Cloud.

teleport_cloud_registered_servers - Connected Teleport services by version. Derived from teleport_registered_servers

teleport_cloud_rx_bytes_2m - Per second rate of received bytes over the last two minutes. Derived from internal metrics.

teleport_cloud_tx_bytes_2m - Per second rate of transferred bytes over the last two minutes. Derived from internal metrics.

teleport_cloud_connected_resources - Connected Teleport resources by region and type. Derived from teleport_connected_resources

teleport_cloud_ssh_connections - Currently active connections from Teleport Proxies to Agents for SSH by region. Derived from proxy_ssh_sessions_total.

teleport_cloud_connect_to_agent_attempts_2m - Per second rate of connection attempts from Teleport Proxies to Agents in the last two minutes by region. Derived from teleport_connect_to_node_attempts_total.

teleport_cloud_connect_to_agent_failures_2m - Per second rate of connection failures from Teleport Proxies to agents in the last two minutes by region. Derived from failed_connect_to_node_attempts_total. 

teleport_cloud_reverse_tunnels - Count of Teleport reverse tunnels connected to Teleport Proxies by region. Derived from teleport_reverse_tunnels.

teleport_cloud_backend_health - Teleport Cloud backend health. Healthy backend reports 1. Derived from Teleport backend read/write metrics and internal metrics.

teleport_cloud_last_updated_seconds - Indicates the last time the metric was updated measured in seconds from Unix epoch.

Configuration

The metrics endpoint is protected via mTLS and will require configuration on three components for metrics to be scraped. This example configuration will assume a Kubernetes deployment for Machine ID and that you are using Prometheus Operator as your deployment for Prometheus in the cluster.

Teleport

You will need to create a a role, a bot, and a token in your Teleport cluster. This assumes that you are using IAM joining for your token and may need to be adjusted depending on your preferred join method.

---
kind: role
metadata:
  name: teleport-cloud-metrics-client
spec:
  allow:
    spiffe:
    - path: /_teleport-cloud/metrics-exporter/client
  deny: {}
version: v6
---
kind: bot
metadata:
  name: teleport-cloud-metrics-client
spec:
  roles:
  - teleport-cloud-metrics-client
version: v1
---
kind: token
metadata:
  name: teleport-cloud-metrics-client
spec:
  allow:
  - aws_account: "012345678901"
  bot_name: teleport-cloud-metrics-client
  join_method: iam
  roles:
  - Bot
version: v2

Machine ID

Follow the guide on deploying Machine ID on Kubernetes. You will need to adjust the outputs to generate a spiffe-svid.

Example output:

outputs:
  - type: spiffe-svid
    svid:
      path: /_teleport-cloud/metrics-exporter/client
    destination:
      type: kubernetes_secret
      name: teleport-cloud-metrics-client

Prometheus

Using a ScrapeConfig provided by Prometheus Operator, you can configure Prometheus to scrape Teleport Cloud and provide metrics for your cluster.

Example:

apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
  labels:
    # Configured label for prometheus scrapeConfigSelector
    release: monitoring
  name: teleport-cloud
spec:
  # Optional relabeling if they are using region externalLabels
  metricRelabelings:
  - action: labelmap
    regex: region
    replacement: teleport_cloud_region
  - action: labeldrop
    regex: region
  scheme: HTTPS
  # Teleport Cloud Metrics are updated every 60s
  scrapeInterval: 60s
  staticConfigs:
  - labels:
      job: teleport-cloud
    targets:
    - <cluster name>.metrics.teleport.sh:443
  # Points to secret created by tbot
  # Need to ensure prometheus operator has RBAC permissions to this secret
  tlsConfig:
    ca:
      secret:
        key: svid_bundle.pem
        name: teleport-cloud-metrics-client
    cert:
      secret:
        key: svid.pem
        name: teleport-cloud-metrics-client
    keySecret:
      key: svid_key.pem
      name: teleport-cloud-metrics-client