How to collect golang profiling data from teleport

Type:

How-To

Question/Problem:

How do I collect golang profiling data from a running teleport process?

This may be needed to help determine low level teleport behaviors especially when troubleshooting things related to cpu usage, memory usage, or other performance type issues.

Solution:

Make sure Teleport is running with the `--diag-addr` and the `-d` arguments. For example:

teleport start --diag-addr=127.0.0.1:3000 -d

The --diag-addr argument enables the http prometheus metrics endpoint, and golang profiling data is available at /debug/pprof/ on that server when teleport is in debug mode, specified by the -d flag.

With that running, you can collect the performance information as follows:

• curl -o cpu.profile http://127.0.0.1:3000/debug/pprof/profile 
• curl -o goroutine.profile http://127.0.0.1:3000/debug/pprof/goroutine 
• curl -o heap.profile http://127.0.0.1:3000/debug/pprof/heap 

The resulting files can be analyzed using standard golang pprof tools.