Type:
Issue
Question/Problem:
Seeing intermittent disconnects and flapping behavior when querying k8s cluster resources through Teleport.
Symptoms:
Users who attempt to access k8s resources through teleport may see error messages indicating inability to connect to the resources. This will often be paired with observable pod restarts on the teleport agent pods.
Logs:
Client Logs:
% k get deploy Error from server: Teleport proxy failed to connect to "kube" agent "remote.kube.proxy.teleport.cluster.local" over reverse tunnel: no tunnel connection found: no kube reverse tunnel for 33388f13-b7a4-4257-bd05-3d6s9g362de15.example.teleport.sh found This usually means that the agent is offline or has disconnected. Check the agent logs and, if the issue persists, try restarting it or re-registering it with the cluster.
Repro Steps:
- Deploy teleport k8s access via
teleport-kube-agent
w/out persistent volumes - Force the
teleport-kube-agent
to bounce intermittently (either manually or via script) - Log into teleport via CLI and attempt to run
kubectl
commands while agents are bouncing. - Observe intermittent inability to connect as noted above.
Solution:
The recommended solution is to redeploy the teleport-kube-agent
with persistent volumes, or upgrade the proxy to 7.3.3 to take advantage of routing stability fixes. Ideally both should be utilized for maximum stability.
Comments
0 comments
Please sign in to leave a comment.