Agent/Node unable to join cluster - the token does not allow \\\"Kube|Node|App|DB\\\" role

Type:

Issue

Question/Problem:

Agent/Node unable to join cluster due to disallowed Kube|Node|App|DB token role.

Symptoms:

Admins attempt to join a new ssh/kube/app/db agent to the cluster but agent fails to join. Logs on the agent suggest that it is not allowed to join due to the token.

Logs:

ERRO [PROC:1] "Kube failed to establish connection to cluster: {\n \"error\": {\n \"message\": \"node \\\"teleport-agents-5b4fc9c545-2gs94\\\" [0ff64123-0892-6900-c92d-683a422b8ba0] can not join the cluster, the token does not allow \\\"Kube\\\" role\"\n }\n}, invalid character '<' looking for beginning of value." time/sleep.go:148

Repro Steps:

1. Set up auth/proxy cluster with node/ssh/app/db token.
2. Configure a teleport agent to join the cluster but pass the incorrect value for the corresponding token key (i.e., if joining ssh node pass the value of the kube token instead in the agent config).
3. Attempt to join node and observe the error above.

Solution:

The error observed above will generally appears when an agent attempts to join the cluster with the incorrect token value but a valid token key. For example, if an ssh agent config is passed the node token key, but the value of a valid kube token, it will attempt to join but the auth server will reject the attempt due to token value not being allowed for the incorrectly-specified use.

Recommended solution is to confirm correct token key/value pair by copying the pair configured on the auth server (if statically configured) or generating a new key/value pair (if configuring dynamically) into the agent. Once the agent config is updated stop the teleport service on the agent, remove the /var/lib/teleport directory on the agent, and then restart service. 

The agent should now be able to join assuming all other join parameters and requirements are met.