Diagnostics Service
The tbot
process can optionally expose a diagnostics service. This is
disabled by default, but once enabled, allows useful information about the
running tbot
process to be queried via HTTP.
Configuration
To enable the diagnostics service, you must specify an address and port for it to listen on.
For security reasons, you should ensure that access to this listener is
restricted. In most cases, the most secure thing to do is to bind the listener
to 127.0.0.1
, which will only allow access from the local machine.
You can configure the diagnostics service using the --diag-addr
CLI parameter:
tbot start -c my-config.yaml --diag-addr 127.0.0.1:3001
Or directly within the configuration file using diag_addr
:
diag_addr: 127.0.0.1:3001
Endpoints
The diagnostics service exposes the following HTTP endpoints.
/livez
The /livez
endpoint always returns with a 200 status code. This can be used
to determine if the tbot
process is running and has not crashed or hung.
If deploying to Kubernetes, we recommend this endpoint is used for your Liveness Probe.
/readyz
and /readyz/{service}
The /readyz
endpoint returns the overall health of tbot
, including all of
its internal and user-defined services. If all services are healthy, it will
respond with a 200 status code. If any service is unhealthy, it will respond
with a 503 status code.
curl -v http://127.0.0.1:3001/readyz
HTTP/1.1 503 Service UnavailableContent-Type: application/json
{ "status": "unhealthy", "services": { "ca-rotation": { "status": "healthy" }, "heartbeat": { "status": "healthy" }, "identity": { "status": "healthy" }, "aws-roles-anywhere": { "status": "unhealthy", "reason": "access denied to perform action \"read\" on \"workload_identity\"" } }}
If deploying to Kubernetes, we recommend this endpoint is used for your Readiness Probe.
You can also use the /readyz/{service}
endpoint to query the health of a
specific service.
curl -v http://127.0.0.1:3001/readyz/aws-roles-anywhere
HTTP/1.1 200 OKContent-Type: application/json
{ "status": "healthy"}
By default, tbot
generates service names based on their configuration such as
the output destination. You can override this by providing your own name in the
tbot
configuration file.
services:
- type: identity
name: my-service-123
/metrics
The /metrics
endpoint returns a Prometheus-compatible metrics snapshot.
The metrics provided by the Go runtime are exposed and these can be used for
monitoring the overall health of the tbot
process. In addition, certain
outputs and services configured in tbot
will produce metrics.
ssh-multiplexer
The SSH multiplexer service exposes metrics about the number of active SSH connections:
tbot_ssh_multiplexer_requests_started_total
: the total number of SSH connections that have been started.tbot_ssh_multiplexer_requests_handled_total
: the total number of SSH connections that have been handled. This has astatus
label with the following values:OK
orERROR
.tbot_ssh_multiplexer_requests_in_flight
: the number of SSH connections currently in progress.
/debug/pprof
These endpoints allow the collection of pprof profiles for debugging purposes. You may be asked by a Teleport engineer to collect these if you are experiencing performance issues.
They will only be enabled if the -d
/--debug
flag is provided when starting
tbot
. This is known as debug mode.