Skip to main content

Diagnostics Service

The tbot process can optionally expose a diagnostics service. This is disabled by default, but once enabled, allows useful information about the running tbot process to be queried via HTTP.

Configuration

To enable the diagnostics service, you must specify an address and port for it to listen on.

For security reasons, you should ensure that access to this listener is restricted. In most cases, the most secure thing to do is to bind the listener to 127.0.0.1, which will only allow access from the local machine.

You can configure the diagnostics service using the --diag-addr CLI parameter:

tbot start -c my-config.yaml --diag-addr 127.0.0.1:3001

Or directly within the configuration file using diag_addr:

diag_addr: 127.0.0.1:3001

Endpoints

The diagnostics service exposes the following HTTP endpoints.

/livez

The /livez endpoint always returns with a 200 status code. This can be used to determine if the tbot process is running and has not crashed or hung.

If deploying to Kubernetes, we recommend this endpoint is used for your Liveness Probe.

/readyz and /readyz/{service}

The /readyz endpoint returns the overall health of tbot, including all of its internal and user-defined services. If all services are healthy, it will respond with a 200 status code. If any service is unhealthy, it will respond with a 503 status code.

curl -v http://127.0.0.1:3001/readyz

HTTP/1.1 503 Service UnavailableContent-Type: application/json
{ "status": "unhealthy", "services": { "ca-rotation": { "status": "healthy" }, "heartbeat": { "status": "healthy" }, "identity": { "status": "healthy" }, "aws-roles-anywhere": { "status": "unhealthy", "reason": "access denied to perform action \"read\" on \"workload_identity\"" } }}

If deploying to Kubernetes, we recommend this endpoint is used for your Readiness Probe.

You can also use the /readyz/{service} endpoint to query the health of a specific service.

curl -v http://127.0.0.1:3001/readyz/aws-roles-anywhere

HTTP/1.1 200 OKContent-Type: application/json
{ "status": "healthy"}

By default, tbot generates service names based on their configuration such as the output destination. You can override this by providing your own name in the tbot configuration file.

services:
  - type: identity
    name: my-service-123

/metrics

The /metrics endpoint returns a Prometheus-compatible metrics snapshot.

The metrics provided by the Go runtime are exposed and these can be used for monitoring the overall health of the tbot process. In addition, certain outputs and services configured in tbot will produce metrics.

ssh-multiplexer

The SSH multiplexer service exposes metrics about the number of active SSH connections:

  • tbot_ssh_multiplexer_requests_started_total: the total number of SSH connections that have been started.
  • tbot_ssh_multiplexer_requests_handled_total: the total number of SSH connections that have been handled. This has a status label with the following values: OK or ERROR.
  • tbot_ssh_multiplexer_requests_in_flight: the number of SSH connections currently in progress.

/debug/pprof

These endpoints allow the collection of pprof profiles for debugging purposes. You may be asked by a Teleport engineer to collect these if you are experiencing performance issues.

They will only be enabled if the -d/--debug flag is provided when starting tbot. This is known as debug mode.