Workload Identity API & Workload Attestation
The Workload Identity API service (workload-identity-api) is a configurable
tbot service that allows workloads to request JWT and X509 workload identity
credentials on-the-fly.
It's a more secure alternative to writing credentials to disk and supports performing a process known as workload attestation to determine attributes of the workload before issuing credentials.
The Workload Identity API is compatible with two standards:
In addition to issuing credentials to workloads, the Workload Identity API can also provide the trust bundle necessary for workloads to validate the credentials of other workloads.
Configuration
# type specifies the type of the service. For the Workload Identity API service,
# this will always be `workload-identity-api`.
type: workload-identity-api
# listen specifies the address that the service should listen on.
#
# Two types of listener are supported:
# - TCP: `tcp://<address>:<port>`
# - Unix socket: `unix:///<path>`
listen: unix:///opt/machine-id/workload.sock
# attestors allows Workload Attestation to be configured for this Workload
# API.
attestors:
# kubernetes is configuration for the Kubernetes Workload Attestor. See
# the Kubernetes Workload Attestor section for more information.
kubernetes:
# enabled specifies whether the Kubernetes Workload Attestor should be
# enabled. If unspecified, this defaults to false.
enabled: true
# kubelet holds configuration relevant to the Kubernetes Workload Attestors
# interaction with the Kubelet API.
kubelet:
# read_only_port is the port on which the Kubelet API is exposed for
# read-only operations. Since Kubernetes 1.16, the read-only port is
# typically disabled by default and secure_port should be used instead.
read_only_port: 10255
# secure_port is the port on which the attestor should connect to the
# Kubelet secure API. If unspecified, this defaults to `10250`. This is
# mutually exclusive with ReadOnlyPort.
secure_port: 10250
# token_path is the path to the token file that the Kubelet API client
# should use to authenticate with the Kubelet API. If unspecified, this
# defaults to `/var/run/secrets/kubernetes.io/serviceaccount/token`.
token_path: "/var/run/secrets/kubernetes.io/serviceaccount/token"
# ca_path is the path to the CA file that the Kubelet API client should
# use to validate the Kubelet API server's certificate. If unspecified,
# this defaults to `/var/run/secrets/kubernetes.io/serviceaccount/ca.crt`.
ca_path: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
# skip_verify is used to disable verification of the Kubelet API server's
# certificate. If unspecified, this defaults to false.
#
# If specified, the value specified in ca_path is ignored.
#
# This is useful in cases where the Kubelet API server has not been issued
# with a certificate signed by the Kubernetes cluster's CA. This is fairly
# common with a number of Kubernetes distributions.
skip_verify: true
# anonymous is used to disable authentication with the Kubelet API. If
# unspecified, this defaults to false. If set, the token_path field is
# ignored.
anonymous: false
# Selector is used to control which WorkloadIdentity resource will be used to
# issue the workload identity credential. The selector can either be the name of
# a specific WorkloadIdentity resource or a label selector that can match
# multiple WorkloadIdentity resources.
#
# The selector must be set to either a name or labels, but not both.
selector:
# Name is used to select a specific WorkloadIdentity resource by its name.
name: foo
# Labels is used to select multiple WorkloadIdentity resources by their labels.
labels:
app: [foo, bar]
SPIFFE Workload API
The Workload Identity API implements the SPIFFE Workload API, a standardized API for workloads to request workload identity credentials and trust bundles.
Via this API, both JWT and X509 workload identity credentials can be issued.
Workload Attestation
Workload Attestation is the process completed by tbot to assert the identity
of a workload that has connected to the Workload API and requested credentials.
Workload Attestors are the individual components that perform this attestation. They use the process ID of the workload to gather information about the workload from platform-specific APIs. For example, the Kubernetes Workload Attestor queries the local Kubelet API to determine which Kubernetes pod the process belongs to.
The result of this attestation process is known as attestation metadata. This attestation metadata can be included in the rules or templates you configure as part of a WorkloadIdentity resource.
Unix
The Unix Workload Attestor is the most basic attestor and allows you to restrict the issuance of workload identities to specific Unix processes based on a range of criteria.
Support for non-standard procfs mounting
To resolve information about a process from the PID, the Unix Workload Attestor
reads information from the procfs filesystem. By default, it expects procfs to
be mounted at /proc.
If procfs is mounted at a different location, you must configure the Unix
Workload Attestor to read from that alternative location by setting the
HOST_PROC environment variable.
This is a sensitive configuration option, and you should ensure that it is set correctly or not set at all. If misconfigured, an attacker could provide falsified information about processes, and this could lead to the issuance of SVIDs to unauthorized workloads.
Kubernetes
The Kubernetes Workload Attestor allows you to restrict the issuance of workload identities to specific Kubernetes workloads based on a range of criteria.
It works by first determining the pod ID for a given process ID and then by querying the local kubelet API for details about that pod.
Deployment Guidance
To use Kubernetes Workload Attestation, tbot must be deployed as a daemon
set. This is because the unix domain socket can only be accessed by pods on the
same node as the agent. Additionally, the daemon set must have the hostPID
property set to true to allow the agent to access information about
processes within other containers.
The daemon set must also have a service account assigned that allows it to query the Kubelet API. This is an example role with the required RBAC:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: tbot
rules:
- resources: ["pods","nodes","nodes/proxy"]
apiGroups: [""]
verbs: ["get"]
Mapping the Workload API Unix domain socket into the containers of workloads can be done in two ways:
- Directly configuring a hostPath volume for the
tbotdaemonset and workloads which will need to connect to it. - Using spiffe-csi-driver.
Example manifests for required Kubernetes resources:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: tbot
rules:
- resources: ["pods","nodes","nodes/proxy"]
apiGroups: [""]
verbs: ["get"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: tbot
subjects:
- kind: ServiceAccount
name: tbot
namespace: default
roleRef:
kind: ClusterRole
name: tbot
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: tbot
namespace: default
---
apiVersion: v1
kind: ConfigMap
metadata:
name: tbot-config
namespace: default
data:
tbot.yaml: |
version: v2
onboarding:
join_method: kubernetes
# replace with the name of a join token you have created.
token: example-token
storage:
type: memory
# ensure this is configured to the address of your Teleport Proxy Service.
proxy_server: example.teleport.sh:443
services:
- type: workload-identity-api
listen: unix:///run/tbot/sockets/workload.sock
attestor:
kubernetes:
enabled: true
kubelet:
# skip verification of the Kubelet API certificate as this is not
# usually issued by the cluster CA.
skip_verify: true
selector:
name: example-workload-identity
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: tbot
spec:
selector:
matchLabels:
app: tbot
template:
metadata:
labels:
app: tbot
spec:
securityContext:
runAsUser: 0
runAsGroup: 0
hostPID: true
containers:
- name: tbot
image: public.ecr.aws/gravitational/tbot-distroless:17.0.0-dev
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
args:
- start
- -c
- /config/tbot.yaml
- --log-format
- json
volumeMounts:
- mountPath: /config
name: config
- mountPath: /var/run/secrets/tokens
name: join-sa-token
- name: tbot-sockets
mountPath: /run/tbot/sockets
readOnly: false
env:
- name: TELEPORT_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: KUBERNETES_TOKEN_PATH
value: /var/run/secrets/tokens/join-sa-token
serviceAccountName: tbot
volumes:
- name: tbot-sockets
hostPath:
path: /run/tbot/sockets
type: DirectoryOrCreate
- name: config
configMap:
name: tbot-config
- name: join-sa-token
projected:
sources:
- serviceAccountToken:
path: join-sa-token
# 600 seconds is the minimum that Kubernetes supports. We
# recommend this value is used.
expirationSeconds: 600
# `example.teleport.sh` must be replaced with the name of
# your Teleport cluster.
audience: example.teleport.sh
Envoy SDS
The workload-identity-api service endpoint also implements the Envoy SDS API.
This allows it to act as a source of certificates and certificate authorities
for the Envoy proxy.
As a forward proxy, Envoy can be used to attach an X.509 SVID to an outgoing connection from a workload that is not SPIFFE-enabled.
As a reverse proxy, Envoy can be used to terminate mTLS connections from SPIFFE-enabled clients. Envoy can validate that the client has presented a valid X.509 SVID and perform enforcement of authorization policies based on the SPIFFE ID contained within the SVID.
When acting as a reverse proxy for certain protocols, Envoy can be configured to attach a header indicating the identity of the client to a request before forwarding it to the service. This can then be used by the service to make authorization decisions based on the client's identity.
When configuring Envoy to use the SDS API exposed by the workload-identity-api
service, three additional special names can be used to aid configuration:
default:tbotwill return the default SVID for the workload.ROOTCA:tbotwill return the trust bundle for the trust domain that the workload is a member of.ALL:tbotwill return the trust bundle for the trust domain that the workload is a member of, as well as the trust bundles of any trust domain that the trust domain is federated with.
The following is an example Envoy configuration that sources a certificate
and trust bundle from the workload-identity-api service listening on
unix:///opt/machine-id/workload.sock. It requires that a connecting client
presents a valid SPIFFE SVID and forwards this information to the backend
service in the x-forwarded-client-cert header.
node:
id: "my-envoy-proxy"
cluster: "my-cluster"
static_resources:
listeners:
- name: test_listener
enable_reuse_port: false
address:
socket_address:
address: 0.0.0.0
port_value: 8080
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
common_http_protocol_options:
idle_timeout: 1s
forward_client_cert_details: sanitize_set
set_current_client_cert_details:
uri: true
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: my_service
domains: ["*"]
routes:
- match:
prefix: "/"
route:
cluster: my_service
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
# configure the certificate that the reverse proxy should present.
tls_certificate_sds_secret_configs:
# `name` can be replaced with the desired SPIFFE ID if multiple
# SVIDs are available.
- name: "default"
sds_config:
resource_api_version: V3
api_config_source:
api_type: GRPC
transport_api_version: V3
grpc_services:
envoy_grpc:
cluster_name: tbot_agent
# combined validation context "melds" two validation contexts
# together. This is handy for extending the validation context
# from the SDS source.
combined_validation_context:
default_validation_context:
# You can use match_typed_subject_alt_names to configure
# rules that only allow connections from specific SPIFFE IDs.
match_typed_subject_alt_names: []
validation_context_sds_secret_config:
name: "ALL" # This can also be replaced with the trust domain name
sds_config:
resource_api_version: V3
api_config_source:
api_type: GRPC
transport_api_version: V3
grpc_services:
envoy_grpc:
cluster_name: tbot_agent
clusters:
# my_service is the example service that Envoy will forward traffic to.
- name: my_service
type: strict_dns
load_assignment:
cluster_name: my_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8090
- name: tbot_agent
http2_protocol_options: {}
load_assignment:
cluster_name: tbot_agent
endpoints:
- lb_endpoints:
- endpoint:
address:
pipe:
# Configure the path to the socket that `tbot` is
# listening on.
path: /opt/machine-id/workload.sock