Where are Teleport Logs and How do I Share Them with Support? – Teleport

This guide describes the log output a Teleport process produces, and provides guidance on locating those log outputs for sharing as part of the support issue troubleshooting workflow. If you are unsure about your Teleport logs, this guide will help you to identify and share them with Technical Support Engineers when working on issues.

What are the Teleport Operational Logs and Where Can I Find Them?

Technical Support Engineers will often ask for Teleport operational logs as a troubleshooting step, so it is extremely helpful to us if you provide these logs whenever relevant and especially when opening a new support issue.

Understanding Teleport Operational Logs (Auth, Proxy, teleport-agent)

The teleport.log configuration block in the teleport.yaml configuration file decides how logging is done for teleport process. See Teleport Configuration Reference on the docs page for full context.

teleport:
  # Logging configuration. Possible output values to disk via
  # '/var/lib/teleport/teleport.log',
  # 'stdout', 'stderr' and 'syslog'. Possible severity values are DEBUG, INFO (default), WARN,
  # and ERROR.
  log:
    output: /var/lib/teleport/teleport.log
    severity: INFO
  # Log format configuration
  # Possible output values are 'json' and 'text' (default).
  # Possible extra_fields values include: timestamp, component, caller,
  # and level.
  # All extra fields are included by default.
    format:
      output: text
      extra_fields: [level, timestamp, component, caller]

Often times, it is necessary to collect debug level logs. You can set the log teleport.log.severity in the teleport config file to DEBUG and restart service to have it take effect. See also How To Debug Teleport.

teleport:
  log:
    severity: DEBUG

The teleport.log.output of your logging configuration determines where the debug logs end up. If unset, then log output goes to stdout. This allows teleport to work with several logging setups that capture process stdout as logs. This includes containers and journald.

Journald logs

On a Linux system with systemd, Teleport logs should be configured to go to stdout and you can access them via the journalctl interface.

You can access a Teleport server and issue a quick command to find only the Teleport process logs entries from the system journal. Presuming your Teleport service is named teleport, use a command like this to retrieve only those log entries:

$ journalctl -b --no-pager -u teleport

The output should go back to the system boot time and will sometimes also include restarts of Teleport process and it can be shared in the support ticket. A command to send these logs to a compressed file would look as below:

$ journalctl -b --no-pager -u teleport | gzip -9 > /tmp/"$(hostname)-$(date +%Y-%m-%dT%H-%M-%SZ)-teleport.log.gz"

which result in a compressed log file in the /tmp directory named like this:

/tmp/ip-10-0-0-42.ap-southeast-2.compute.internal-2022-07-11T01-11-44Z-teleport.log.gz

If your Teleport systemd service is not named “teleport” and you are unsure of the service name, then a more generic command can be used:

$ journalctl -b | awk '$5 ~ "teleport"'

Please share compressed log file as above from all teleport servers (auth, proxy) and teleport-agent node.

Static File Logging

If teleport.log.output is configured to set logs to a static file, i.e.

output:
  /file/path/filename.log

, teleport logs will be directed to this file.

Below configuration would send teleport logs to file /var/lib/teleport/teleport.log.

teleport:
    log:
      output: /var/lib/teleport/teleport.log
      severity: DEBUG

Other System Log Locations

Teleport can log to syslog if teleport.log.output is set to syslog. Other logging systems are syslog compatible, so the exact location as to where the logs end up can vary. Check with your OS's syslog configuration file to confirm where teleport syslog messages will end up.

Common places to check for syslog and other various logging systems include:

/var/log/messages
/var/log/syslog

You can execute a quick grep command to check those locations:

$ sudo grep 'teleport\[' /var/log/messages
$ sudo grep 'teleport\[' /var/log/syslog

Docker

Logs from Teleport Docker containers can be retrieved with the docker logs command, adjusting for the name of the container in question:

$ docker logs teleport_teleport_1
2022-07-04T23:48:51Z INFO Generating new host UUID: 2728bab0-1dd9-42fc-b9e1-0ff6c07f34a3. service/service.go:7162022-07-04T23:48:51Z DEBU [SQLITE] Connected to: file:%2Fvar%2Flib%2Fteleport%2Fproc%2Fsqlite.db?_busy_timeout=10000&_sync=FULL&_txlock=immediate, poll stream period: 1s lite/lite.go:246
2022-07-04T23:48:51Z DEBU [SQLITE] Connected to: file:%2Fvar%2Flib%2Fteleport%2Fproc%2Fsqlite.db?_busy_timeout=10000&_sync=FULL&_txlock=immediate, poll stream period: 1s lite/lite.go:246
2022-07-04T23:48:51Z DEBU [SQLITE] journal_mode=delete, synchronous=2, busy_timeout=10000 lite/lite.go:293
2022-07-04T23:48:55Z DEBU [PROC:1] Adding service to supervisor. service:readyz.monitor service/supervisor.go:207
2022-07-04T23:48:55Z INFO [PROC:1] Service diag is creating new listener on 0.0.0.0:3000. service/signals.go:212
2022-07-04T23:48:55Z INFO [DIAG:1] Starting diagnostic service on 0.0.0.0:3000. service/service.go:2438
2022-07-04T23:48:55Z DEBU [PROC:1] Adding service to supervisor. service:diagnostic.service service/supervisor.go:207
2022-07-04T23:48:55Z DEBU [PROC:1] Adding service to supervisor. service:diagnostic.shutdown service/supervisor.go:207
2022-07-04T23:48:55Z DEBU [KEYGEN] SSH cert authority started with no keys pre-compute. native/native.go:106

Kubernetes

Logs from Teleport Kubernetes pods can be retrieved with the kubectl logs command as in this example:

% kubectl -n teleport-agent logs teleport-agent-699d6b8f6c-x29dx
2022-07-04T23:48:51Z INFO Generating new host UUID: 2728bab0-1dd9-42fc-b9e1-0ff6c07f34a3. service/service.go:716
2022-07-04T23:48:51Z DEBU [SQLITE] Connected to: file:%2Fvar%2Flib%2Fteleport%2Fproc%2Fsqlite.db?_busy_timeout=10000&_sync=FULL&_txlock=immediate, poll stream period: 1s lite/lite.go:246
2022-07-04T23:48:51Z DEBU [SQLITE] journal_mode=delete, synchronous=2, busy_timeout=10000 lite/lite.go:293
2022-07-04T23:48:55Z DEBU [PROC:1] Adding service to supervisor. service:readyz.monitor service/supervisor.go:207
2022-07-04T23:48:55Z INFO [PROC:1] Service diag is creating new listener on 0.0.0.0:3000. service/signals.go:212
2022-07-04T23:48:55Z INFO [DIAG:1] Starting diagnostic service on 0.0.0.0:3000. service/service.go:2438
2022-07-04T23:48:55Z DEBU [PROC:1] Adding service to supervisor. service:diagnostic.service service/supervisor.go:207
2022-07-04T23:48:55Z DEBU [PROC:1] Adding service to supervisor. service:diagnostic.shutdown service/supervisor.go:207
2022-07-04T23:48:55Z DEBU [KEYGEN] SSH cert authority started with no keys pre-compute. native/native.go:106