Managed Updates (v2) for Teleport Agents

Report an issue with this page

warning

This document describes Managed Updates for Agents (v2), which is gradually being rolled out to Cloud.

For Managed Updates v1 instructions, see Managed Updates for Agents (v1).

In Managed Updates v2, a binary called teleport-update is distributed in all Teleport packages, alongside the teleport binary. Admins configure updates by managing the autoupdate_version and autoupdate_config dynamic resources.

This document covers how to use teleport-update and the autoupdate_* resources to manage your agent updates from Teleport. It describes:

• The agent architecture
• How to enroll existing agents
• How to enroll new agents
• How to configure Managed Updates v2 ( when updates happen and for self-hosted users, which version to update to)
• How to migrate to Managed Updates v2

teleport-update supports:

• Both Teleport Enterprise and Teleport Community Edition
• Both cloud and self-hosted Teleport Enterprise deployments
• Regular and FIPS variants of Teleport
• amd64 and arm64 CPU architectures
• systemd-based operating systems, regardless of the package manager used

Compatibility between Managed Updates v1 and v2

The Managed Updates v2 teleport-update binary is backwards-compatible with the cluster_maintenance_config resource. The Managed Updates v1 teleport-upgrade script is forwards-compatible with the autoupdate_config and autoupdate_version resources. Agents connected to the same cluster will all update to the same version.

If the autoupdate_config resource is configured, it takes precedence over cluster_maintenance_config. This allows for a safe, non-breaking, incremental migration between Managed Updates v1 and v2. If autoupdate_config is not present and autoupdate_version is present, the autoupdate_config settings are implicitly derived from cluster_maintenance_config.

Users of cloud-hosted Teleport Enterprise will be migrated to Managed Updates v2 in the first half of 2025 and should plan to migrate their agents to teleport-update.

How it works

When Managed Updates are enabled, a Teleport updater is installed alongside each new Teleport Agent. The updater communicates with the Teleport Proxy Service to determine when an update is available and if it should perform the update now.

Each agent belongs to an update group. The update schedule specifies when each group is updated. The schedule is stored in the autoupdate_config resource and can be edited via tctl.

For Linux server-based installations, teleport-update command configures Managed Updates locally on the server.

For Kubernetes-based installations, the teleport-kube-agent Helm chart deploys a controller that automatically updates the main Teleport container.

Existing agents must be manually enrolled into Managed Updates.

Prerequisites

• Familiarity with the Upgrading Compatibility Overview guide, which describes the sequence in which to upgrade components in your cluster.
• Teleport Agents that are not yet enrolled in Managed Updates.

• • A running Teleport cluster version 17.5.2 or above. If you do not have one, read Get Started with Teleport.

  • The tctl and tsh clients.

    Details

    Installing tctl and tsh clients

    Mac

    Download the signed macOS .pkg installer for Teleport, which includes the tctl and tsh clients:

    curl -O https://cdn.teleport.dev/teleport-17.5.2.pkg

    In Finder double-click the pkg file to begin installation.

    danger

    Using Homebrew to install Teleport is not supported. The Teleport package in Homebrew is not maintained by Teleport and we can't guarantee its reliability or security.

    Windows - Powershell

    curl.exe -O https://cdn.teleport.dev/teleport-v17.5.2-windows-amd64-bin.zip
    Unzip the archive and move the `tctl` and `tsh` clients to your %PATH%
    NOTE: Do not place the `tctl` and `tsh` clients in the System32 directory, as this can cause issues when using WinSCP.
    Use %SystemRoot% (C:\Windows) or %USERPROFILE% (C:\Users\<username>) instead.

    Linux

    All of the Teleport binaries in Linux installations include the tctl and tsh clients. For more options (including RPM/DEB packages and downloads for i386/ARM/ARM64) see our installation page.

    curl -O https://cdn.teleport.dev/teleport-v17.5.2-linux-amd64-bin.tar.gz
    tar -xzf teleport-v17.5.2-linux-amd64-bin.tar.gz
    cd teleport
    sudo ./install
    Teleport binaries have been copied to /usr/local/bin

    The tctl and tsh clients must be at most one major version behind your Teleport cluster version. Send a GET request to the Proxy Service at /v1/webapi/ping and use a JSON query tool to obtain your cluster version:

    curl https://example.teleport.sh/v1/webapi/ping | jq -r '.server_version'
    17.5.2

• To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. For example, run the following command, assigning teleport.example.com to the domain name of the Teleport Proxy Service in your cluster and email@example.com to your Teleport username:

  tsh login --proxy=teleport.example.com --user=email@example.com
  tctl status
  Cluster  teleport.example.com
  Version  17.5.2
  CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

  If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

Quick setup for existing connected Linux servers

Users can enable Managed Updates v2 on Linux servers that are already running a Teleport Agent by running the following command on every server:

sudo teleport-update enable

note

If this command is not available, update the teleport package to the latest version that is supported by your cluster.

The teleport-update enable command will disable (but not remove) the v1 updater if present. No other action is necessary.

If everything is working, the v1 updater package can be removed:

sudo apt remove teleport-ent-updater

If the v2 updater does not work, your installation can be reverted back to manual updates or the v1 updater (if it has not been removed):

sudo teleport-update uninstall

If Teleport was installed via the apt or yum package, teleport-update uninstall will revert the running version of Teleport back to the version provided by the package.

Quick setup for new Linux servers

The Install Script is the fastest way to onboard new Linux servers. However, you may also use teleport-update by itself to set up a Teleport Agent manually.

Users can create a new installation of Teleport using any version of the teleport-update binary. First, download copy of the Teleport tarball from the downloads page. Next, invoke teleport-update to install the correct version for your cluster.

tar xf teleport-[version].tgz
cd teleport-[version]
sudo ./teleport-update enable --proxy example.teleport.sh

After Teleport is installed, you can create /etc/teleport.yaml, either manually or using teleport configure. After, the Teleport Agent can be enabled and started via the systemctl command:

sudo systemctl enable teleport --now

Configuring managed agent updates

Managed agent updates are configured via two Teleport resources:

• autoupdate_config controls the update schedule
• autoupdate_version controls the desired version

Self-hosted Teleport users must configure both autoupdate_config and autoupdate_version.

Cloud-hosted Teleport Enterprise users can configure the autoupdate_config, while the autoupdate_version is managed by Teleport Cloud. Updates will roll out automatically during the first chosen maintenance window that is at least 36 hours after the cluster version is updated.

To configure Managed Updates in your cluster, you must have access to the autoupdate_config and autoupdate_version resources. By default, the editor role can modify both resources.

Configuring the schedule

For both cloud-hosted and self-hosted editions of Teleport, an update schedule may be set with the autoupdate_config resource. The default resource looks like this:

kind: autoupdate_config
metadata:
  name: autoupdate-config
spec:
  agents:
    mode: enabled
    strategy: halt-on-error
    schedules:
      regular:
        - name: default
          days: [ "Mon", "Tue", "Wed", "Thu" ]
          # start_hour is in UTC
          start_hour: 16

For example, a Teleport user with staging and production environments might create a custom schedule that looks like this:

kind: autoupdate_config
metadata:
  name: autoupdate-config
spec:
  agents:
    mode: enabled
    strategy: halt-on-error
    schedules:
      regular:
        - name: staging
          days: [ "Mon", "Tue", "Wed", "Thu" ]
          start_hour: 4
        - name: production
          days: [ "Mon", "Tue", "Wed", "Thu" ]
          start_hour: 5
          wait_hours: 24

This schedule would update agents in the staging group at 4 UTC, and then update the production group at 5 UTC the next day. The production group will not execute update until the staging group has updated. The wait_hours field sets a minimum duration between groups, ensuring that production happens the day after staging, and not one hour after.

warning

While failed installations will revert automatically on the client-side, server-side healthchecks are still in development. To prevent the production group above from updating after staging has failed, you must manually suspend the schedule by setting the spec.agents.mode to suspended.

Two update rollout strategies are available:

• The halt-on-error strategy provides predictable, sequential updates across environments. It's ideal for traditional development pipelines where you want to ensure that development environments are successfully updated before proceeding to staging and production.
• The time-based strategy is designed for environments where update groups are independent of each other, such as geographical regions or different teams. It allows updates to occur whenever the specified maintenance window is active for a group, regardless of the status of other groups. This strategy does not provide ordering guarantees across groups.

You can find more information in the Managed Updates v2 resource reference

Except for autoupdate_config.agents.mode, changes to autoupdate_config fields take effect during the next version rollout. A new rollout happens when autoupdate_version is changed and targets a new version. Version is automatically updated for Cloud-hosted Teleport clusters; for self-hosted ones you have to update the version manually, see the dedicated guide section.

Setting the version (self-hosted only)

For cloud-hosted Teleport Enterprise, Managed Updates are enabled by default. The autoupdate_version resource is managed for you and cannot be edited. This ensures your agents are always up-to-date and running the best version for your Teleport cluster.

important

Self-hosted Teleport users must specify which version their agents should update to via the autoupdate_version resource. If the resource does not exist, agents will not update.

Create a file called autoupdate_version.yaml containing:

kind: autoupdate_version
metadata:
  name: autoupdate-version
spec:
  agents:
    start_version: 17.2.0
    target_version: 17.2.1
    schedule: regular
    mode: enabled

This resource is used to deploy new versions of Teleport to your agents. The cluster will update agents from start_version to target_version according to the update schedule specified in the autoupdate_config.

Run the following command to create or update the resource:

tctl create -f autoupdate_version.yaml

Changes to autoupdate_version can take up to a minute to create a new rollout. You can observe the current rollout state with the command:

tctl autoupdate agents status
Agent autoupdate mode: enabledRollout creation date: 2025-03-10 15:01:45Start version: 1.2.3Target version: 1.2.4Rollout state: ActiveStrategy: halt-on-error
Group Name State     Start Time          State Reason---------- --------- ------------------- ------------------------dev        Active    2025-03-11 12:00:10 can_startstage      Unstarted                     previous_groups_not_doneprod       Unstarted                     previous_groups_not_done

Migrating agents on Linux servers to Managed Updates

Finding unmanaged agents

Use the tctl inventory ls command to list connected agents along with their current version. Use the --upgrader=none flag to list agents that are not enrolled in managed updates.

tctl inventory ls --upgrader=none
Server ID                            Hostname      Services Version Upgrader------------------------------------ ------------- -------- ------- --------00000000-0000-0000-0000-000000000000 ip-10-1-6-130 Node     v14.4.5 none...

Use the --upgrader=unit flag to list agents that are using Managed Updates v1 and should be updated to Managed Updates v2:

tctl inventory ls --upgrader=unit
Server ID                            Hostname      Services Version Upgrader------------------------------------ ------------- -------- ------- --------00000000-0000-0000-0000-000000000000 ip-10-1-6-131 Node     v14.4.5 unit...

Agents enrolled into Managed Updates v2 can be queried with the --upgrader=binary flag.

Enrolling unmanaged agents

1. For each agent ID returned by the tctl inventory ls command, copy the ID and run the following tctl command to access the host via tsh:

   HOST=00000000-0000-0000-0000-000000000000
   USER=root
   tsh ssh "${USER?}@${HOST?}"

2. Run teleport-update enable on each agent you would like to enroll into Managed Updates v2:

   sudo teleport-update enable

3. Confirm that the version of the teleport binary is the one you expect:

   teleport version

4. Remove the Managed Updates v1 updater if present:

   DEB

   sudo apt remove teleport-ent-updater

   RPM

   sudo yum remove teleport-ent-updater

Running the agent as a non-root user

If you changed the agent user to run as non-root, create /etc/teleport-upgrade.d/schedule and grant ownership to your Teleport user:

sudo mkdir -p /etc/teleport-upgrade.d/
sudo touch /etc/teleport-upgrade.d/schedule
sudo chown your-teleport-user /etc/teleport-upgrade.d/schedule

While teleport-update does not read this file, teleport will warn if it cannot disable the Managed Update v1 updater using this file.

Enroll Kubernetes agents in Managed Updates

This section assumes that the name of your teleport-kube-agent release is teleport-agent, and that you have installed it in the teleport namespace.

1. Add the following chart values to the values file for the teleport-kube-agent chart:

   updater:
     enabled: true
   

2. Update the Teleport Helm repository to include any new versions of the teleport-kube-agent chart:

   helm repo update teleport

3. Update the Helm chart release with the new values:

   Cloud-Hosted

   helm -n teleport  upgrade teleport-agent teleport/teleport-kube-agent \--values=values.yaml \--version="17.5.2"

   Self-Hosted

   helm -n teleport  upgrade teleport-agent teleport/teleport-kube-agent \--values=values.yaml \--version="17.5.2"

4. You can validate the updater is running properly by checking if its pod is ready:

   kubectl -n teleport-agent get pods
   NAME                               READY   STATUS    RESTARTS   AGE<your-agent-release>-0                         1/1     Running   0          14m<your-agent-release>-1                         1/1     Running   0          14m<your-agent-release>-2                         1/1     Running   0          14m<your-agent-release>-updater-d9f97f5dd-v57g9   1/1     Running   0          16m

5. Check for any deployment issues by checking the updater logs:

   kubectl -n teleport logs deployment/teleport-agent-updater
   2023-04-28T13:13:30Z	INFO	StatefulSet is already up-to-date, not updating.	{"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"my-agent","namespace":"agent"}, "namespace": "agent", "name": "my-agent", "reconcileID": "10419f20-a4c9-45d4-a16f-406866b7fc05", "namespacedname": "agent/my-agent", "kind": "StatefulSet", "err": "no new version (current: \"v12.2.3\", next: \"v12.2.3\")"}

Troubleshooting

You can inspect the current agent autoupdate status by running:

tctl autoupdate agents status

Agent autoupdate mode: enabledRollout creation date: 2025-02-24 16:01:44Start version: 17.2.0Target version: 17.2.1Rollout state: UnstartedStrategy: time-based
Group Name State     Start Time State Reason---------- --------- ---------- --------------default    Unstarted            outside_window

This rollout state is computed by each Auth Service instance every minute. An autoupdate_config or autoupdate_version change might take up to a minute to be reflected and applied.

Teleport Agents are not updated immediately when a new version of Teleport is released, and agent updates can lag behind the cluster by a few days.

If the Teleport Agent has not been automatically updating for several weeks, you can consult the updater logs to help troubleshoot the problem:

Troubleshooting managed agent upgrades on Kubernetes

The updater is a controller that periodically reconciles expected Kubernetes resources with those in the cluster. The updater executes a reconciliation loop every 30 minutes or in response to a Kubernetes event. If you don't want to wait until the next reconciliation, you can trigger an event.

1. Any deployment update will send an event, so you can trigger the upgrader by annotating the resource:

   kubectl -n teleport annotate statefulset/teleport-agent 'debug.teleport.dev/trigger-event=1'

2. To suspend Managed Updates for an agent, annotate the agent deployment with teleport.dev/skipreconcile: "true", either by setting the annotations.deployment value in Helm, or by patching the deployment directly with kubectl.

Troubleshooting managed agent upgrades on Linux

1. You can query the updater status by running:

   teleport-update status
   proxy: teleport.example.com:443path: /usr/local/binbase_url: https://cdn.teleport.devenabled: truepinned: falseactive:    version: 17.2.0    flags: [Enterprise]target:    version: 17.2.1    flags: [Enterprise]in_window: falsejitter: 1m0s

   Here, the local active version is 17.2.0. The cluster's target version is 17.2.1, but we are not in an update window, so the agent is not immediately updated.

journalctl -u teleport-update

1. If an agent is not automatically updated, you can invoke the updater manually and look at its logs:

   sudo teleport-update update --now

Using a different CDN URL

If your agents cannot reach the default Teleport CDN URL (cdn.teleport.dev), they will be unable to download updates.

Here are a couple of potential solutions to this issue:

Use an HTTP CONNECT proxy

If you configure the HTTPS_PROXY variable in the teleport-update process's environment, it will use this proxy to pull updates.

The easiest way to configure a proxy with a default install is to add this variable to /etc/systemd/system/teleport-update.service.d/override.conf:

$ sudo mkdir -p /etc/systemd/system/teleport-update.service.d
$ sudo tee -a /etc/systemd/system/teleport-update.service.d/override.conf > /dev/null <<'EOF'
[Service]
Environment=HTTPS_PROXY=http://proxy-url:3128
EOF

You can view the teleport-update process logs with sudo journalctl -u teleport-update.service.

Mirror the Teleport tarball packages and change the base-url

If you can mirror the Teleport tarball installers somewhere that your agents are able to access, you can change the base-url used by Teleport updaters so they can pull them directly.

To change the base-url, you should add the -b or --base-url flag to the teleport-update enable command:

$ sudo teleport-update enable --base-url https://teleport.artifactory.company.local

It is safe to re-run sudo teleport-update enable to modify the base URL. Existing updater settings will be preserved if not explicitly overridden by flags.

More information about flags that can be used with teleport-update enable can be found here