Node failed to establish connection to cluster: key /ids/node/replacement is not found.

Type:

issue

Question/Problem:

A node is unable to connect to a cluster after it had previously joined with a token successfully.

Symptoms:

When a teleport process starts up and reads its existing identity information in /var/lib/teleport, it then connects to the cluster. When this issue is happening, the teleport agent logs may include entries similar to the following:

DEBU [PROC:1]    Connected state: rotating servers (mode: manual, started: Jan  5 08:49:26 UTC, ending: Jan  6 14:49:26 UTC). service/connect.go:103
ERRO [PROC:1]    Node failed to establish connection to cluster: key /ids/node/replacement is not found. service/connect.go:68
DEBU [PROC:1]    Connected state: rotating servers (mode: manual, started: Jan  5 08:49:26 UTC, ending: Jan  6 14:49:26 UTC). service/connect.go:103
ERRO [PROC:1]    Node failed to establish connection to cluster: key /ids/node/replacement is not found. service/connect.go:68

The agent then exits altogether.

Repro Steps:

Start a manual teleport certificate authority on a teleport cluster, but do not finish it.

Join a new node with a token

Stop the new node and remove the token from the config file or command line

Start the node again, and observe the described symptoms

Solution:

Complete the teleport certificate authority rotation procedure.