Version: 14.x

Set up Automatic Agent Upgrades

Teleport supports automatic agent upgrades for systemd-based Linux distributions using apt, yum, and zypper package managers, as well as Kubernetes clusters.

Teleport agents run an upgrader that queries a version server to determine whether they are out of date. This guide describes how to set up your infrastructure to support automatic upgrades. If you are a Teleport Cloud user or run a version server already, return to the Upgrading menu for the appropriate next steps to upgrade Teleport.

The Automatic Update Architecture guide explains how automatic agent upgrades work in more detail.

warning

Systemd agents enrolled into automatic upgrades can only install versions present in their package repositories. As Teleport 14 won't be published to stable/v13, those agents will require manual intervention to be upgraded to the next major version (adding a new APT/YUM/zypper repo for stable/v14). Alternatively, you can use the stable/rolling channel, which contains Teleport v13.3.2 forward, including future major releases.

Prerequisites

Familiarity with the Upgrading Compatibility Overview guide, which describes the sequence in which to upgrade components of your cluster.
Self-hosted Teleport cluster v13.0 or higher.

The tctl and tsh client tools version >= 14.3.33.

$ tctl version
# Teleport v14.3.33 go1.21

$ tsh version
# Teleport v14.3.33 go1.21

To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. tctl is supported on macOS and Linux machines. For example:
```
$ tsh login --proxy=teleport.example.com --user=email@example.com
$ tctl status
# Cluster  teleport.example.com
# Version  14.3.33
# CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
```
If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

The version server must be hosted on a webserver with trusted TLS certificates and reachable by all agents. You must have either:

A public Amazon S3 or Google Cloud Storage bucket.
A web server accessible from all agents with valid TLS certificates.

If you do not have an S3 or GCS bucket for your version server, this guide provides an example of how to create one using Terraform. You can install Terraform to spin up a demo version server along with this guide so you can get started with automatic agent upgrades. Terraform is not required to set up a version server.

To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. tctl is supported on macOS and Linux machines. For example:
```
$ tsh login --proxy=teleport.example.com --user=email@example.com
$ tctl status
# Cluster  teleport.example.com
# Version  14.3.33
# CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
```
If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

Step 1/5. Create release channel files

A release channel contains two pieces of information: the targeted version and if the upgrade is critical. Updaters subscribe to a release channel and will upgrade to the provided version during a maintenance window (which we will configure later in this guide). If the upgrade is critical, updaters will ignore the maintenance schedule and upgrade as soon as possible.

The version server is a static file server that responds to the following queries:

$ curl https://<hosting-domain-and-path>/current/version
14.3.33

$ curl https://<hosting-domain-and-path>/current/critical
no

Create a project directory for the files we create in this guide:
```
$ mkdir version-server
$ cd version-server
```
Create a directory for the new release channel current.
```
$ mkdir current/
```
Make the current release channel target the version 14.3.33:
```
$ echo -n "14.3.33" > current/version
```
And mark the upgrade as not critical:
```
$ echo -n "no" > current/critical
```

Step 2/5. Create a Terraform configuration

In the project directory you created in the previous section, create a file called main.tf and populate it with the following content, depending on whether you will use Amazon S3 or Google Cloud Storage:

Amazon S3
Google Cloud Storage

In the configuration below, replace REGION with the name of the AWS region where you will deploy the version server:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "REGION"
}

resource "aws_s3_bucket" "version_server" {
  // Replace the bucket name to ensure it is unique
  bucket = "version-server"
}

resource "aws_s3_object" "version" {
  bucket = aws_s3_bucket.version_server.id
  key = "current/version"
  source = "${path.root}/current/version"
}

resource "aws_s3_object" "critical" {
  bucket = aws_s3_bucket.version_server.id
  key = "current/critical"
  source = "${path.root}/current/critical"
}

resource "aws_s3_bucket_policy" "version_server" {
  depends_on = [aws_s3_bucket_public_access_block.version_server]
  bucket = aws_s3_bucket.version_server.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid       = "GrantAnonymousReadPermissions"
        Principal = "*"
        Action = [
          "s3:GetObject",
        ]
        Effect   = "Allow"
        Resource = ["${aws_s3_bucket.version_server.arn}/*"]
      },
    ]
  })
}

resource "aws_s3_bucket_public_access_block" "version_server" {
  bucket = aws_s3_bucket.version_server.id

  block_public_acls       = false
  block_public_policy     = false
  ignore_public_acls      = false
  restrict_public_buckets = false
}

In the configuration below, replace PROJECT and REGION with the Google Cloud project and region where you will deploy the version server:

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
  }
}

provider "google" {
  project     = "PROJECT"
  region      = "REGION"
}

resource "google_storage_bucket" "version_server" {
  // Change the location as needed. See:
  // https://cloud.google.com/storage/docs/locations
  location = "US-EAST1"
  // Replace this to ensure it is unique
  name = "version-server"
}

resource "google_storage_bucket_object" "version" {
  name   = "current/version"
  source = "${path.root}/current/version"
  bucket = google_storage_bucket.version_server.name
}

resource "google_storage_bucket_object" "critical" {
  name   = "current/critical"
  source = "${path.root}/current/critical"
  bucket = google_storage_bucket.version_server.name
}

data "google_iam_policy" "viewer" {
  binding {
    role = "roles/storage.objectViewer"
    members = [
        "allUsers",
    ] 
  }
}

resource "google_storage_bucket_iam_policy" "version_server" {
  bucket = google_storage_bucket.version_server.name
  policy_data = data.google_iam_policy.viewer.policy_data
}

Make your cloud provider credentials available to Terraform. The method to use depends on your organization, e.g., pasting environment variables into your terminal.
Apply the configuration:

$ terraform init
$ terraform apply

Step 3/5. Configure the maintenance schedule

At this point the updaters can be configured to pull the version from the release channel and upgrade the agents. In this step you'll configure a maintenance schedule for the Teleport cluster that agents will use to determine when to check for upgrades.

Create a Teleport role

Create a Teleport role that can manage cluster maintenance configurations through the cluster_maintenance_config dynamic resource. No preset Teleport roles provide this ability, so you will need to create one.

Create a file called cmc-editor.yaml with the following content:

kind: role
version: v7
metadata:
  name: cmc-editor
spec:
  allow:
    rules:
    - resources: ['cluster_maintenance_config']
      verbs: ['create', 'read', 'update', 'delete']

Create the role resource:
```
$ tctl create cmd-editor.yaml
```
Add the role to your Teleport user:

Assign the cmc-editor role to your Teleport user by running the appropriate commands for your authentication provider:

Local User
GitHub
SAML
OIDC

Retrieve your local user's configuration resource:

$ tctl get users/$(tsh status -f json | jq -r '.active.username') > out.yaml

Edit out.yaml, adding cmc-editor to the list of existing roles:

  roles:
   - access
   - auditor
   - editor
+  - cmc-editor 

Apply your changes:
```
$ tctl create -f out.yaml
```
Sign out of the Teleport cluster and sign in again to assume the new role.

Retrieve your github authentication connector:
```
$ tctl get github/github --with-secrets > github.yaml
```
Note that the --with-secrets flag adds the value of spec.signing_key_pair.private_key to the github.yaml file. Because this key contains a sensitive value, you should remove the github.yaml file immediately after updating the resource.
Edit github.yaml, adding cmc-editor to the teams_to_roles section.

The team you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the team must include your user account and should be the smallest team possible within your organization.

Here is an example:
```
  teams_to_roles:
    - organization: octocats
      team: admins
      roles:
        - access
+       - cmc-editor
```
Apply your changes:
```
$ tctl create -f github.yaml
```
Sign out of the Teleport cluster and sign in again to assume the new role.

Retrieve your saml configuration resource:
```
$ tctl get --with-secrets saml/mysaml > saml.yaml
```
Note that the --with-secrets flag adds the value of spec.signing_key_pair.private_key to the saml.yaml file. Because this key contains a sensitive value, you should remove the saml.yaml file immediately after updating the resource.
Edit saml.yaml, adding cmc-editor to the attributes_to_roles section.

The attribute you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization.

Here is an example:
```
  attributes_to_roles:
    - name: "groups"
      value: "my-group"
      roles:
        - access
+       - cmc-editor
```
Apply your changes:
```
$ tctl create -f saml.yaml
```
Sign out of the Teleport cluster and sign in again to assume the new role.

Retrieve your oidc configuration resource:
```
$ tctl get oidc/myoidc --with-secrets > oidc.yaml
```
Note that the --with-secrets flag adds the value of spec.signing_key_pair.private_key to the oidc.yaml file. Because this key contains a sensitive value, you should remove the oidc.yaml file immediately after updating the resource.
Edit oidc.yaml, adding cmc-editor to the claims_to_roles section.

The claim you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization.

Here is an example:
```
  claims_to_roles:
    - name: "groups"
      value: "my-group"
      roles:
        - access
+       - cmc-editor
```
Apply your changes:
```
$ tctl create -f oidc.yaml
```
Sign out of the Teleport cluster and sign in again to assume the new role.

Create a cluster maintenance configuration

Create the following cmc.yaml manifest allowing maintenance on Monday, Wednesday and Friday between 02:00 and 03:00 UTC.

kind: cluster_maintenance_config
spec:
  agent_upgrades:
    # Maintenance window start hour in UTC.
    # The maintenance window lasts 1 hour.
    utc_start_hour: 2
    # Week days when maintenance is allowed
    # Possible values are:
    # - Short names: Sun, Mon, Tue, Wed, Thu, Fri, Sat
    # - Long names: Sunday, Monday, Tuesday, Wednesday, Thursday, Friday, Saturday
    weekdays:
      - Mon
      - Wed
      - Fri

Finally, apply the manifest using tctl:

$ tctl create cmc.yaml
maintenance window has been updated

Step 4/5. Enroll Kubernetes agents in automatic upgrades

Now that you have deployed a version server, you can enroll agents in automatic upgrades. This guide begins with agents deployed on Kubernetes. If all of your agents run on Linux servers, you can skip to Step 5.

Install the agent upgrader Helm chart

This section assumes that the name of your teleport-kube-agent release is teleport-agent, and that you have installed it in the teleport namespace.

Confirm you are using the Teleport Enterprise edition of the teleport-kube-agent chart. You should see the following when you query your teleport-kube-agent release:
```
$ helm -n teleport get values -n teleport-agent -o json | jq '.enterprise'
true   
```
Add the following chart values to the values file for the teleport-kube-agent chart:
```
updater:
  enabled: true
  versionServer: https://version.example.com
  releaseChannel: release-channel
```
Change release-channel to the name of the release channel you would like the agent to query for upgrades. For the current version, choose current.

Update the Helm chart release with the new values:

$ helm -n teleport  upgrade teleport-agent teleport/teleport-kube-agent \
--values=values.yaml \
--version=14.3.33

Verify that the upgrader is working properly

You can validate the updater is running properly by checking if its pod is ready:

$ kubectl -n teleport-agent get pods
NAME                               READY   STATUS    RESTARTS   AGE
<your-agent-release>-0                         1/1     Running   0          14m
<your-agent-release>-1                         1/1     Running   0          14m
<your-agent-release>-2                         1/1     Running   0          14m
<your-agent-release>-updater-d9f97f5dd-v57g9   1/1     Running   0          16m

Check for any deployment issues by checking the updater logs:

$ kubectl -n teleport logs deployment/teleport-agent-updater
2023-04-28T13:13:30Z	INFO	StatefulSet is already up-to-date, not updating.	{"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"my-agent","namespace":"agent"}, "namespace": "agent", "name": "my-agent", "reconcileID": "10419f20-a4c9-45d4-a16f-406866b7fc05", "namespacedname": "agent/my-agent", "kind": "StatefulSet", "err": "no new version (current: \"v12.2.3\", next: \"v12.2.3\")"}

Troubleshooting automatic agent upgrades on Kubernetes

The updater is a controller that periodically reconciles expected Kubernetes resources with those in the cluster. The updater executes a reconciliation loop every 30 minutes or in response to a Kubernetes event. If you don't want to wait until the next reconciliation, you can trigger an event.

Any deployment update will send an event, so you can trigger the upgrader by annotating the resource:
```
$ kubectl -n teleport annotate statefulset/teleport-agent 'debug.teleport.dev/trigger-event=1'
```
To suspend automatic upgrades for an agent, annotate the agent deployment with teleport.dev/skipreconcile: "true", either by setting the annotations.deployment value in Helm, or by patching the deployment directly with kubectl.

Step 5/5. Enroll Linux agents in automatic upgrades

This section shows you how to enroll Teleport agents running on Linux virtual or bare-metal machines into automatic upgrades.

Follow these instructions on each of your Teleport agents.

Install the agent upgrader

Ensure the Teleport repository is added and Teleport Enterprise is installed.

To verify if the Teleport repository was added to the system, check if either of the follow files exist:
```
$ ls /etc/apt/sources.list.d/teleport.list
# or
$ ls /etc/yum.repos.d/teleport.repo
```
The upgrader checks the repository for available releases, so make sure that it is up to date.
If the repository was added, make sure the Teleport binary installed on the agent can run the automatic upgrader:
```
$ which teleport-upgrade || echo "Install the upgrader"
Install the upgrader
```

If the Teleport repository is not found, or the Teleport binary you installed does not include the upgrader, add the appropriate repository and reinstall Teleport.

# Download Teleport's PGP public key
$ sudo curl https://apt.releases.teleport.dev/gpg \
-o /usr/share/keyrings/teleport-archive-keyring.asc
# Source variables about OS version
$ source /etc/os-release
# Add the Teleport APT repository for v14. You'll need to update this
# file for each major release of Teleport.
$ echo "deb [signed-by=/usr/share/keyrings/teleport-archive-keyring.asc] \
https://apt.releases.teleport.dev/${ID?} ${VERSION_CODENAME?} stable/v14" \
| sudo tee /etc/apt/sources.list.d/teleport.list > /dev/null

$ sudo apt-get update
$ sudo apt-get install teleport-ent-updater