Upgrade Teleport Cloud Agents on Kubernetes

This guide explains how to upgrade Teleport Cloud agents running on Kubernetes. On Teleport Cloud, Auth Service and Proxy Service upgrades are managed for you. To keep agents up to date, you can either enroll them in automatic upgrades or upgrade them manually using the method you used to install Teleport.

Prerequisites

• Familiarity with the Upgrading Compatibility Overview guide, which describes the sequence in which to upgrade components of your cluster.

• Teleport Enterprise Cloud. You can determine the current version of these services by running the following command, where mytenant is the name of your Teleport Enterprise Cloud tenant. This requires the jq CLI tool:

  $ curl -s https://mytenant.teleport.sh/webapi/ping | jq '.server_version'

• The tctl and tsh client tools version >= 16.4.7.

  The tctl and tsh client tools version >= 16.4.7. To install these tools, see the Installation page.

  $ tctl version
  # Teleport Enterprise v16.4.7 go1.21
  
  $ tsh version
  # Teleport v16.4.7 go1.21

• To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. tctl is supported on macOS and Linux machines.

  For example:

  $ tsh login --proxy=teleport.example.com --user=email@example.com
  $ tctl status
  # Cluster  teleport.example.com
  # Version  14.3.33
  # CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

  If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

Teleport supports automatic agent updates for the teleport-kube-agent Helm chart. The Automatic Update Architecture guide describes how agent updating works. Automatic agent upgrades require:

• A Teleport Cloud account that supports automatic upgrades. To determine if your account supports automatic upgrades, run the following command, replacing mytenant.teleport.sh with the address of your Teleport Cloud account:

  $ curl -s https://mytenant.teleport.sh/webapi/ping | jq '.automatic_upgrades'
  true

• At least one Teleport Enterprise agent deployed via the teleport-kube-agent Helm chart.

Step 1/2. Determine whether any agents require automatic upgrades

To determine if any agents in your cluster are not configured for automatic upgrades, run the following command. This prints a list of all Teleport services with no upgrader, filtering these to ignore the Auth Service and Proxy Service (which are maintained for you in Teleport Cloud):

$ tctl inventory ls \
  --services=node,kube,db,app,windows_desktop,discovery,okta \
  --upgrader=none
Server ID                            Hostname                        Services Version Upgrader
------------------------------------ ------------------------------- -------- ------- --------
00000000-0000-0000-0000-000000000000 teleport-agent-0                Kube     v13.4.3 none

If the list is nonempty, proceed to the next step to enable automatic upgrades.

Step 2/2. Enroll agents in automatic upgrades

This section assumes that the name of your teleport-kube-agent release is teleport-agent, and that you have installed it in the teleport namespace.

1. Confirm you are using the Teleport Enterprise edition of the teleport-kube-agent chart. You should see the following when you query your teleport-kube-agent release:

   $ helm -n teleport get values teleport-agent -o json | jq '.enterprise'
   true   

2. Add the following chart values to your existing agent values.yaml to enable the automatic updater:

   updater:
     enabled: true

3. Upgrade the Helm chart release with the new values by running helm upgrade. The command should resemble the following:

   $  helm -n teleport upgrade teleport-agent teleport/teleport-kube-agent  \
   --values=values.yaml \
   --version=16.4.7

4. Validate the updater is running properly by checking if its pod is ready:

   $ kubectl -n teleport get pods
   NAME                               READY   STATUS    RESTARTS   AGE
   my-agent-0                         1/1     Running   0          14m
   my-agent-1                         1/1     Running   0          14m
   my-agent-2                         1/1     Running   0          14m
   my-agent-updater-d9f97f5dd-v57g9   1/1     Running   0          16m

   And by consulting its logs:

   $ kubectl -n teleport logs deployment/teleport-agent-updater
   2023-04-28T13:13:30Z	INFO	StatefulSet is already up-to-date, not updating.	{"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"my-agent","namespace":"agent"}, "namespace": "agent", "name": "my-agent", "reconcileID": "10419f20-a4c9-45d4-a16f-406866b7fc05", "namespacedname": "agent/my-agent", "kind": "StatefulSet", "err": "no new version (current: \"v12.2.3\", next: \"v12.2.3\")"}

The updater is a controller that periodically reconciles expected Kubernetes resources with those in the cluster. The updater executes a reconciliation loop every 30 minutes or in response to a Kubernetes event. If you don't want to wait until the next reconciliation, you can trigger an event. Any deployment update will send an event, so the updater can be triggered by annotating the resource:

$ kubectl -n teleport annotate statefulset/teleport-agent 'debug.teleport.dev/trigger-event=1'

To suspend automatic upgrades for an agent, annotate the agent deployment with teleport.dev/skipreconcile: "true", either by setting the annotations.deployment value in Helm, or by patching the deployment directly with kubectl.

GitOps

Automatic updates are incompatible with some GitOps tools used for continuous deployment. The teleport-kube-agent Helm chart owns the version of the teleport-agent resource, so when the teleport-agent-updater modifies the image version of the teleport-agent resource, the GitOps tool will detect a drift or a diff in the teleport-agent resource.

ArgoCD deployments

After an automatic update, ArgoCD reports the teleport-agent resource as OutOfSync. As a workaround to this problem use a Diff Customization to ignore the difference in image version. Here is an example deployment using the name teleport-agent and namespace teleport.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: teleport-agent
  namespace: teleport
spec
  ignoreDifferences:
  - group: apps
    kind: StatefulSet
    name: teleport-agent
    namespace: teleport
    jqPathExpressions:
    - .spec.template.spec.containers[] | select(.name == "teleport").image
...

FluxCD deployments

After an automatic update, FluxCD reports a DriftDetected event. As a workaround to this problem modify the drift detection configuration to ignore the difference in image version. Here is an example deployment using the name teleport-agent and namespace teleport.

apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
  name: teleport-agent
  namespace: teleport
spec
  driftDetection:
    mode: enabled
    ignore:
    - paths: ["/spec/template/spec/containers/0/image"]
      target:
        kind: StatefulSet
        name: teleport-agent
        namespace: teleport
...

Manually upgrading agents

Run the following commands to upgrade Teleport agents running on Kubernetes.

1. Update the Teleport Helm chart repository so you can install the latest version of the teleport-kube-agent chart:

   Set up the Teleport Helm repository.

   Allow Helm to install charts that are hosted in the Teleport Helm repository:

   $ helm repo add teleport https://charts.releases.teleport.dev

   Update the cache of charts from the remote repository so you can upgrade to all available releases:

   $ helm repo update

2. Upgrade the Helm release:

   $ helm -n teleport upgrade teleport-agent teleport/teleport-kube-agent \
   --values=values.yaml \
   --version=16.4.7