Hardening Infrastructure Security Against SSO Identity Provider Compromise
Jul 11
Virtual
Register Today
Support
LOG IN
Teleport logoTry For Free
Contact SalesTry for Free
Support
LOG IN
Fork me on GitHub

Teleport

AWS Database Discovery

Version preview

Teleport can be configured to discover AWS databases automatically and register them with your Teleport cluster.

Teleport Database auto-discovery involves two components.

The first, the Discovery Service, is responsible for watching your cloud provider and checking if there are any new databases or if there have been any modifications to previously discovered databases. It dynamically registers each discovered database as a db resource in your Teleport cluster. The second, the Database Service, monitors the dynamic db resources registered by the Discovery Service. It proxies communications between users and the database.

Tip

This guide presents the Discovery Service and Database Service running in the same process, however both can run independently and on different machines.

For example, you can run an instance of the Database Service in the same private network as the databases you want to register with your Teleport cluster, and an instance of the Discovery Service in any network you wish.

Prerequisites

  • A running Teleport cluster version 17.0.0-dev or above. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.

  • The tctl admin tool and tsh client tool.

    Visit Installation for instructions on downloading tctl and tsh.

  • An AWS account with permissions to create and attach IAM policies.
  • One or more database servers hosted by AWS database services.
  • A host to run the Teleport Discovery Service.

Step 1/4. Generate a join token

The Discovery Service requires a valid join token to join your Teleport cluster. Run the following tctl command and save the token output in /tmp/token on the server that will run the Discovery Service:

tctl tokens add --type=discovery --format=text
abcd123-insecure-do-not-use-this

For users with a lot of infrastructure in AWS, or who might create or recreate many instances, consider alternative methods for joining new EC2 instances running Teleport:

Step 2/4. Configure the Discovery service

Enabling AWS database discovery requires that the discovery_service.aws section includes at least one entry and that discovery_service.aws.types includes one of database types listed in the sample YAML below.

Create a teleport.yaml file similar to the following on the host that will run the Discovery Service:

version: v3
teleport:
  join_params:
    token_name: "/tmp/token"
    method: token
  proxy_server: "teleport.example.com:443"
auth_service:
  enabled: off
proxy_service:
  enabled: off
ssh_service:
  enabled: off
discovery_service:
  enabled: "yes"
  discovery_group: "aws-prod"
  aws:
    # Database types. Valid options are:
    # 'rds' - discovers AWS RDS and Aurora databases.
    # 'rdsproxy' - discovers AWS RDS Proxy databases.
    # 'redshift' - discovers AWS Redshift databases.
    # 'redshift-serverless' - discovers AWS Redshift Serverless databases.
    # 'elasticache' - discovers AWS ElastiCache Redis databases.
    # 'memorydb' - discovers AWS MemoryDB Redis databases.
    # 'opensearch' - discovers AWS OpenSearch Redis databases.
  - types: ["rds"]
    regions: ["us-east-1"]
    tags:
      "env": "prod" # Match database resource tags where tag:env=prod

  - types: ["redshift", "redshift-serverless"]
    regions: ["us-west-1"]
    tags:
      "env": "prod"
    # Optional AWS role that the Discovery Service will assume to discover
    # AWS-hosted databases. The IAM identity assigned to the host must be able
    # to assume this role.
    assume_role_arn: "arn:aws:iam::123456789012:role/example-role-name"
    # Optional AWS external ID that the Database Service will use to assume
    # a role in an external AWS account.
    external_id: "example-external-id"

Adjust the keys under discovery_service.aws to match your AWS databases.

Discovery Service exposes a configuration parameter - discovery_service.discovery_group - that allows you to group discovered resources into different sets. This parameter is used to prevent Discovery Agents watching different sets of cloud resources from colliding against each other and deleting resources created by another services.

When running multiple Discovery Services, you must ensure that each service is configured with the same discovery_group value if they are watching the same cloud resources or a different value if they are watching different cloud resources.

It is possible to run a mix of configurations in the same Teleport cluster meaning that some Discovery Services can be configured to watch the same cloud resources while others watch different resources. As an example, a 4-agent high availability configuration analyzing data from two different cloud accounts would run with the following configuration.

  • 2 Discovery Services configured with discovery_group: "prod" polling data from Production account.
  • 2 Discovery Services configured with discovery_group: "staging" polling data from Staging account.

Step 3/4. Bootstrap IAM permissions

Create an IAM role and attach it to the host that will run the Discovery Service.

Teleport can bootstrap IAM permissions for the Discovery Service using the teleport discovery bootstrap command. You can use this command in automatic or manual mode:

  • In automatic mode, Teleport attempts to create the appropriate IAM policies and attach them to the specified IAM roles. This requires IAM permissions to create and attach IAM policies.
  • In manual mode, Teleport prints the required IAM policies. You can then create and attach them manually using the AWS management console.

Either temporarily give IAM admin permissions to the host of the Discovery Service or copy the service YAML configuration file to your desktop where you have the IAM admin permissions.

Use this command to bootstrap the permissions automatically with YAML configuration file of the Discovery Service:

teleport discovery bootstrap \  --attach-to-role arn:aws:iam::aws-account-id:role/iam-role-name \  --policy-name TeleportDatabaseDiscovery \  -c /etc/teleport.yaml

Use the following command to display the required IAM policies that you need to create in your AWS console:

teleport discovery bootstrap --manual \  --attach-to-role arn:aws:iam::aws-account-id:role/iam-role-name \  --policy-name TeleportDatabaseDiscovery \  -c /etc/teleport.yaml

When assume_role_arn is configured for AWS matchers, teleport discovery bootstrap command determines the permissions required for the bootstrap target AWS IAM identity using the following logic:

  • When the target does not match assume_role_arn in any AWS matcher in the configuration file, the target is assumed to be the Teleport Discovery Service's AWS IAM identity and permissions are bootstrapped for all the AWS matchers without assume_role_arn.
  • When an --attach-to-role target matches an assume_role_arn setting for AWS matchers in the configuration file, permissions are bootstrapped only for those AWS matchers.

You will need to run the bootstrap command once with the Teleport Discovery Service's IAM identity as the policy attachment target, and once for each AWS IAM role that is used for assume_role_arn.

Step 4/4. Use the Discovery Service

Start the Discovery Service

Configure the Discovery Service to start automatically when the host boots up by creating a systemd service for it. The instructions depend on how you installed the Discovery Service.

On the host where you will run the Discovery Service, enable and start Teleport:

sudo systemctl enable teleport
sudo systemctl start teleport

On the host where you will run the Discovery Service, create a systemd service configuration for Teleport, enable the Teleport service, and start Teleport:

sudo teleport install systemd -o /etc/systemd/system/teleport.service
sudo systemctl enable teleport
sudo systemctl start teleport

You can check the status of the Discovery Service with systemctl status teleport and view its logs with journalctl -fu teleport.

After the Discovery Service starts, database servers matching the tags and regions specified in the AWS section are added to the Teleport cluster automatically.

List registered databases

You can list them with the tctl get db command or check a specific database with tctl get db/<database-name>.

The default name of the discovered database includes the AWS resource ID, endpoint and matcher types, AWS region and account ID to ensure uniqueness, for example my-postgres-rds-aurora-us-east-1-123456789012.

A discovered database with the default name also has a shorter display name that consists only the AWS resource ID and the endpoint type, for example my-postgres, my-postgres-reader. Either the full name or the display name can be used for tctl and tsh commands.

You can override the default name by applying the TeleportDatabaseName AWS tag to the AWS database resource.

Next

Troubleshooting

Discovery Service

To check if the Discovery Service is working correctly, you can check if any databases have been discovered. To do this, you can use the tctl get db command and check if the expected databases have already been registered with your Teleport cluster.

If some databases do not appear in the list, check if the Discovery Service selector labels match the missing database tags or look into the Discovery Service logs for permission errors.

If you are running multiple Discovery Services, you must ensure that each service is configured with the same discovery_group value if they are watching the same cloud databases or a different value if they are watching different cloud databases. If this is not configured correctly, a typical symptom is db resources being intermittently deleted from your Teleport cluster's registry.

Database Service

If the tctl get db command returns the discovered databases you expect, but the tctl db ls command does not include them, check that you have set the db_service.resources section correctly, for example:

db_service:
  enabled: "yes"
  resources:
  - labels:
      "env": "prod"

If the section is correctly configured, but databases still do not appear, check that you have the correct permissions to list databases in Teleport.

If there are connection errors when you try to connect to a database, then first check if there are multiple db_server heartbeat resources for the target database: tctl get db_server/yourDatabaseName. If there are, it means that multiple Teleport Database Services are proxying the database - this is an HA setup that will complicate troubleshooting. Teleport will choose one of those Database Services at random to proxy the connection and if one of them can't reach the database endpoint or lacks permissions, then you will see random connection errors.

Even if connection errors are consistent, you should scale down or reconfigure your Teleport Database Services such that only one matches the target db while you are troubleshooting errors. Verify that there is only one db_server with tctl get db_server/yourDatabaseName and then try the connection again.

Check the Teleport Database Service logs with DEBUG level logging enabled and look for network or permissions errors.

There may be more specifc troubleshooting steps for a particular type of database in the database access guides.