Seeing intermittent disconnects and flapping behavior when querying k8s cluster resources through Teleport.
Users who attempt to access k8s resources through teleport may see error messages indicating inability to connect to the resources. This will often be paired with observable pod restarts on the teleport agent pods.
% k get deploy Error from server: Teleport proxy failed to connect to "kube" agent "remote.kube.proxy.teleport.cluster.local" over reverse tunnel: no tunnel connection found: no kube reverse tunnel for 33388f13-b7a4-4257-bd05-3d6s9g362de15.example.teleport.sh found This usually means that the agent is offline or has disconnected. Check the agent logs and, if the issue persists, try restarting it or re-registering it with the cluster.
- Deploy teleport k8s access via
teleport-kube-agentw/out persistent volumes
- Force the
teleport-kube-agentto bounce intermittently (either manually or via script)
- Log into teleport via CLI and attempt to run
kubectlcommands while agents are bouncing.
- Observe intermittent inability to connect as noted above.
The recommended solution is to redeploy the
teleport-kube-agent with persistent volumes, or upgrade the proxy to 7.3.3 to take advantage of routing stability fixes. Ideally both should be utilized for maximum stability.