Teleport and its various components may be configured to utilize AWS or AWS compatible services for various things. When this happens, teleport needs AWS credentials to access those services.
If you get an error about not being able to load credentials, you'll need to double check that they are in place. It usually specifically mentions NoCredentialProviders, but the exact error message/stack trace can vary depending on the teleport component and the aws service it is accessing.
Some possible error messages in the teleport logs might look like one of the following:
User Message: NoCredentialProviders: no valid providers in chain. Deprecated
Resolving this Error
When the teleport process starts up and reads its configuration file, it will try to contact any configured AWS services. Teleport uses the AWS SDK for Go to communicate with AWS services. We use the default credential chain, so the behavior that teleport will have should be in line with using any other AWS tool that uses the same AWS standard credential format.
Details in the AWS docs about the order which credentials can be found are here: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials
You should refer to the AWS SDK documentation as the authoritative source of truth for how credentials are loaded rather than this document. The commentary here is supposed to serve as a guide for dealing with this sort of issue in the context of teleport.
Shared Credential File
Make sure that ~/.aws/config and/or ~/.aws/credentials, have the correct format, and are readable by teleport. Make sure that they are in the home directory of the teleport user. In containerization, you may end up with a user without a home directory due to the container being forced to start with a specific uid. Make sure that the file is accessible.
For example, if the teleport process is running as the user `teleport`, then run the following:
ls -lahd ~teleport
ls -lah ~teleport/.aws/
The ~teleport directory, ~teleport/.aws directory, and the config and/or credentials files should all exist and have appropriate permissions.
Environment Variables
It is generally recommended to not put credentials into an environment variable, but since the AWS sdk supports it, it does work.
Double check that you have correctly set each variable in the context that teleport is running.
AWS_ACCESS_KEY_ID=YOUR_AKID
AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY
If you are running teleport via systemd, you will need to make sure it is set in the systemd unit file. Make sure you run `systemctl daemon-reload` before restarting teleport after making any changes to systemd units.
If you are running teleport in a container, you'll need to make sure your container has those environment variables set. Every software deploy method should have a provision for setting an environment variable.
IAM Role
If you are running teleport in an environment where an IAM role will be applied, that is also a possible method for teleport to get credentials. Usually if it is running on an ec2 instance, it will at least get the node IAM role. Access issues that happen after the credentials are loaded would result in an error other than the NoCredentialProviders error that this article covers.
Testing Credentials using the AWS cli interface
If you need to test your credential loading without starting teleport itself for some reason, you can make use of the aws cli tool to try out various AWS operations: https://aws.amazon.com/cli/
For example, try running the following command in the same environment where you are trying to run teleport:
aws --region us-east-2 --debug dynamodb describe-table --table-name teleport-events
Make sure that you launch this aws command in the exact same way that you would launch teleport. You can drop this command into the systemd unit or run it inside a teleport container. You may need to install the awscli package to get it to run.
If the aws cli tool does not find any credentials, it will output an error
Unable to locate credentials. You can configure credentials by running "aws configure".
Appendix of places this may appear
While not necessarily exhaustive, here are several examples of what this error can look like depending on what teleport is doing when it encounters it.
S3
Teleport attempting to create an S3 at startup may give this message:
2022-01-14T22:05:43Z ERRO [S3] "Failed to ensure that bucket \"teleport-bucket\" exists (NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors). S3 session uploads may fail. If you've set up the bucket already and gave Teleport write-only access, feel free to ignore this error." s3sessions/s3handler.go:303
Teleport attempting to upload a session to S3:
2022-01-14T22:45:46Z WARN [AUTH:COMP] Failed to check uploads. error:[
ERROR REPORT:
Original Error: *trace.BadParameterError NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Stack Trace:
/go/src/github.com/gravitational/teleport/lib/events/s3sessions/s3handler.go:365 github.com/gravitational/teleport/lib/events/s3sessions.ConvertS3Error
/go/src/github.com/gravitational/teleport/lib/events/s3sessions/s3stream.go:169 github.com/gravitational/teleport/lib/events/s3sessions.(*Handler).ListUploads
/go/src/github.com/gravitational/teleport/lib/events/complete.go:125 github.com/gravitational/teleport/lib/events.(*UploadCompleter).CheckUploads
/go/src/github.com/gravitational/teleport/lib/events/complete.go:113 github.com/gravitational/teleport/lib/events.(*UploadCompleter).run
/opt/go/src/runtime/asm_amd64.s:1371 runtime.goexit
User Message: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors] events/complete.go:114
Dynamodb
Teleport cluster storage:
2022-01-14T20:04:10Z INFO [DYNAMODB] Initializing backend. Table: "teleport_storage", poll streams every 0s. dynamo/dynamodbbk.go:207
ERROR: initialization failed
NoCredentialProviders: no valid providers in chain. Deprecated.
"\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
Teleport event storage:
2022-01-14T20:07:40Z INFO [DYNAMODB] Initializing event backend. dynamoevents/dynamoevents.go:248
ERROR: initialization failed
NoCredentialProviders: no valid providers in chain. Deprecated.
"\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
Database Access (RDS)
Teleport Database agent accessing an RDS database:
2022-01-14T22:42:05Z ERRO [WATCHER:C] rdsFetcher(Region=us-east-2, Labels=map[*:[*]]) failed. error:[
ERROR REPORT:
Original Error: *awserr.baseError NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Stack Trace:
/go/src/github.com/gravitational/teleport/lib/srv/db/cloud/watchers/rds.go:84 github.com/gravitational/teleport/lib/srv/db/cloud/watchers.(*rdsFetcher).Get
/go/src/github.com/gravitational/teleport/lib/srv/db/cloud/watchers/watcher.go:120 github.com/gravitational/teleport/lib/srv/db/cloud/watchers.(*Watcher).fetchAndSend
/go/src/github.com/gravitational/teleport/lib/srv/db/cloud/watchers/watcher.go:104 github.com/gravitational/teleport/lib/srv/db/cloud/watchers.(*Watcher).Start
/opt/go/src/runtime/asm_amd64.s:1581 runtime.goexit
User Message: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors] watchers/watcher.go:122
2022-01-14T22:42:05Z INFO [DB:SERVIC] Failed to ping web proxy "teleport.example.com:11246" addr: Get "https://teleport.example.com:11246/webapi/find": tls: first record does not look like a TLS handshake leaseID:1 target:teleport.example.com:11246 reversetunnel/agent.go:269
2022-01-14T22:42:05Z INFO [DB:SERVIC] Connected. addr:198.51.100.86:58392 remote-addr:203.0.113.80:11246 leaseID:1
Another place it can show up is if you attempt to access an RDS postgres database through teleport, and the database agent doesn't have the credential, it can echo back to the end user trying to run tsh db connect dbname:
psql: error: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
You'll still need to perform the troubleshooting steps on the actual database agent instance rather than where the tsh client is running.
Comments
0 comments
Article is closed for comments.