Skip to main content
Version: 14.x

Google Cloud KMS

This guide will show you how to set up your Teleport Cluster to use the Google Cloud Key Management Service (KMS) to store and handle the CA private key material used to sign all certificates issued by your Teleport cluster.

Teleport generates private key material for its internal Certificate Authorities (CAs) during the first Auth Server's initial startup. These CAs are used to sign all certificates issued to clients and hosts in the Teleport cluster. When configured to use Google Cloud KMS, all private key material for these CAs will be generated, stored, and used for signing inside of Google Cloud KMS. Instead of the actual private key, Teleport will only store the ID of the KMS key. In short, private key material will never leave Google Cloud KMS.

If launching a new Teleport cluster this will all be handled during initial startup with no specific interventions required after configuration. For existing Teleport clusters that already have private key material on disk, a CA rotation must be performed. This ensures that the private key material only ever exists in KMS when this feature is enabled. Read on to migrating an existing cluster to learn more.

tip

Teleport Enterprise Cloud takes care of this setup for you so you can provide secure access to your infrastructure right away.

Get started with a free trial of Teleport Enterprise Cloud.

Prerequisites​

The features documented on this page are available in Teleport 11.1.0 and higher.

  • A running Teleport Enterprise cluster. For details on how to set this up, see the Enterprise Getting Started guide.

  • The Enterprise tctl admin tool and tsh client tool version >= 14.3.33. You can download these tools by visiting your Teleport account. You can verify the tools you have installed by running the following commands:

    $ tctl version
    # Teleport Enterprise v14.3.33 go1.21

    $ tsh version
    # Teleport v14.3.33 go1.21
  • To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. tctl is supported on macOS and Linux machines. For example: 
    $ tsh login --proxy=teleport.example.com --user=email@example.com
    $ tctl status
    # Cluster  teleport.example.com
    # Version  14.3.33
    # CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
    If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.
  • A Google Cloud account.

Step 1/5. Create a key ring in GCP​

Each Teleport Auth Server will need to be configured to use a GCP key ring which will hold all keys generated and used by that Auth Server. If running a High-Availability Teleport cluster with two or more Auth Servers, every Auth Server can be configured to use the same key ring, or if desired each can be configured to use a unique key ring in a different region (for redundancy or to decrease latency).

It is recommended to create a dedicated key ring for use by Teleport to logically separate it from any other keys in your cloud account. Choose a supported KMS location for the key ring which is geographically near to your Teleport Auth Servers.

You can create a key ring from the Google Cloud Console or from the gcloud CLI tool. Follow this guide or run the following command if you have the gcloud CLI configured:

$ gcloud kms keyrings create "teleport-keyring" --location location

Step 2/5. Create a GCP service account​

Teleport needs permissions to create, list, destroy, sign with, and view KMS keys in your key ring. Start by creating the following custom IAM role.

# teleport_kms_role.yaml
title: teleport_kms_role
description: 'Teleport permissions for using KMS keys'
stage: ALPHA
includedPermissions:
- cloudkms.cryptoKeys.create
- cloudkms.cryptoKeys.list
- cloudkms.cryptoKeyVersions.create
- cloudkms.cryptoKeyVersions.destroy
- cloudkms.cryptoKeyVersions.useToSign
- cloudkms.cryptoKeyVersions.viewPublicKey
$ gcloud iam roles create teleport_kms_role \
    --project GCP-Project-ID \
    --file teleport_kms_role.yaml \
    --format yaml

Note the name field in the output which is the fully qualified name for the custom role and must be used in later steps.

$ export IAM_ROLE=<role name output from above>

If you don't already have a GCP service account for your Teleport Auth Server you can create one with the following command, otherwise use your existing service account.

$ gcloud iam service-accounts create teleport-auth-server \
    --description="Service account for Teleport Auth Server" \
    --display-name="Teleport Auth Server" \
    --format=yaml

Note the email field in the output, this must be used as the identifier for the service account.

$ export SERVICE_ACCOUNT=<email output from above command>

Create the IAM policy binding to grant the role to the service account for this keyring.

$ gcloud kms keyrings add-iam-policy-binding teleport-keyring \
    --location location \
    --member "serviceAccount:${SERVICE_ACCOUNT}" \
    --role "${IAM_ROLE}"

Step 3/5. Provide the service account credentials to your Auth Server​

The Teleport Auth Server will use Application Default Credentials to make requests to the GCP KMS service. Provide credentials for the teleport-auth-server service account created in step 2 to the Application Default Credentials of the environment you are running your Teleport Auth Server in. Supported environments include GCE VMs, GKE pods, and others.

See the GCP docs for Application Default Credentials to learn how to provide them for your preferred environment.

Manually checking permissions

To make sure the credentials have been configured correctly, you can run the gcloud CLI tool from your Teleport Auth Server's environment. Some example commands you could use to debug are listed here:

$ gcloud kms keys list --location location --keyring "teleport-keyring"
Listed 0 items.
$ gcloud kms keys create --location location --keyring "teleport-keyring" \
    --purpose asymmetric-signing \
    --default-algorithm rsa-sign-pkcs1-4096-sha512 \
    test-key
$ gcloud kms keys list --location location --keyring "teleport-keyring"
NAME                                                                                   PURPOSE          ALGORITHM                   PROTECTION_LEVEL
projects/my-gcp-account/locations/global/keyRings/teleport-keyring/cryptoKeys/test-key ASYMMETRIC_SIGN  RSA_SIGN_PKCS1_4096_SHA512  SOFTWARE
$ echo hello > /tmp/hello.txt
$ gcloud kms asymmetric-sign --keyring "teleport-keyring" --location location \
    --key "test-key" --version 1 \
    --input-file /tmp/hello.txt --signature-file /tmp/hello.sig
$ gcloud kms keys versions destroy --keyring "teleport-keyring" --location location --key "test-key" 1

Step 4/5. Configure your Auth Server to use KMS keys​

CA key parameters are statically configured in the teleport.yaml configuration file of the Teleport Auth Server(s) in your cluster.

Find the fully qualified name of the KMS key ring you created in step 1 in the GCP Console or by running:

$ gcloud kms keyrings list --location location
Choosing a protection level

Supported KMS protection levels are SOFTWARE and HSM. If you choose SOFTWARE, GCP KMS will performs all cryptographic operations in software (Teleport performs no cryptographic operations). If you choose HSM, GCP KMS will perform all cryptographic operations in a Hardware Security Module.

Both protection levels are considered secure by Google and Teleport, you should evaluate your own organization's requirements and security policies when making your decision.

One relevant difference is the usage quotas available to keys of each protection level. At the time of writing, software keys have a project-wide quota of 60k cryptographic operations per minute, while asymmetric HSM keys have a quota of 3k operations per minute. See the KMS docs for updated numbers. If your cluster will have many thousands of hosts or active users, the higher quota software keys have may help to avoid any potential throttling, especially during CA rotations where many new certificates must be signed.

Include the following ca_key_params configuration in the auth_service section.

# /etc/teleport.yaml
auth_service:
  # ...
  ca_key_params:
    gcp_kms:
      keyring: "projects/<your-gcp-project>/locations/<location>/keyRing/<your-teleport-keyring>"
      protection_level: "SOFTWARE"

If configuring this before the first start of a new Teleport cluster, the initial CA keys will be generated in GCP and no additional steps are necessary. If you wish to migrate an existing Teleport cluster from software keys to GCP KMS keys, read on to migrating an existing cluster.

Step 5/5. Make sure everything is working​

After starting up your Auth Server with the gcp_kms configuration, you can confirm that Teleport has generated keys in your keyring in the GCP Console or by running

$ gcloud kms keys list --keyring "teleport-keyring" --location location

Try logging in to the cluster with a Teleport user to make sure that new certificates can be signed without error.

Migrating an existing cluster​

If you have an existing Teleport cluster it will have already created CA keys during its first start. Those existing CA keys will have been used to sign all existing user and host certificates, and will be trusted by all other services in your cluster.

When an Auth Server starts up with a gcp_kms keyring configured in its ca_key_params, it will refuse to sign any certificates with any existing software keys in the CA. This will prevent any new user logins or new hosts from joining your cluster if their requests are directed to that Auth Server and effectively cause downtime for that server until a CA rotation is completed.

If some downtime until you can complete a CA rotation is acceptable, the migration can be performed in three steps:

  1. Configure all Auth Servers ca_key_params to use your desired KMS keyring, as described in Step 4.
  2. Restart all Auth Servers.
  3. Perform a full CA rotation.

To avoid any downtime while migrating your cluster, do the following procedure instead:

  1. Start a new Auth Server with an identical backend configuration to your existing Auth Servers and with ca_key_params configured to use your KMS key ring. Make sure no requests are routed to this new, temporary, Auth Server by not adding it to your load balancer. You can run this anywhere with access to your existing backend and new KMS key ring, one option would be to run it locally on an existing Auth Server host (make sure to give it its own teleport.yaml and unique data_dir).
  2. Perform a full CA rotation. The temporary Auth Server will generate new KMS keys and include their names in the backend CA state.
  3. Stop/remove/delete the temporary Auth Server as it is no longer necessary.
  4. Configure all other existing Auth Servers with identical ca_key_params and reload/restart them, one by one. They will now use the KMS keys generated by the temporary Auth Server.
  5. Perform one more full CA rotation to evict all now-unused software keys from the CA backend state so that hosts will no longer trust them.