Troubleshooting Kubernetes Instability

Type:

Issue

Question/Problem:

Seeing intermittent disconnects and flapping behavior when querying k8s cluster resources through Teleport.

Symptoms:

Users who attempt to access k8s resources through teleport may see error messages indicating inability to connect to the resources. This will often be paired with observable pod restarts on the teleport agent pods. 

Logs:

Client Logs:

% k get deploy
Error from server: Teleport proxy failed to connect to "kube" agent "remote.kube.proxy.teleport.cluster.local" over reverse tunnel:

  no tunnel connection found: no kube reverse tunnel for 33388f13-b7a4-4257-bd05-3d6s9g362de15.example.teleport.sh found

This usually means that the agent is offline or has disconnected. Check the
agent logs and, if the issue persists, try restarting it or re-registering it
with the cluster.

Repro Steps:

1. Deploy teleport k8s access via teleport-kube-agentw/out persistent volumes
2. Force the teleport-kube-agentto bounce intermittently (either manually or via script)
3. Log into teleport via CLI and attempt to run kubectlcommands while agents are bouncing.
4. Observe intermittent inability to connect as noted above.

Solution:

The recommended solution is to redeploy the teleport-kube-agent with persistent volumes, or upgrade the proxy to 7.3.3 to take advantage of routing stability fixes. Ideally both should be utilized for maximum stability.