Protect Kubernetes secrets

$ run as a normal user# run as root (or with sudo)

Skills in this chapter

Manage Kubernetes secrets and encrypt them at rest

See the problem

Without encryption, a secret is readable in etcd in plain text.

$kubectl create secret generic db -n shop --from-literal=password=S3cret
$kubectl get secret db -n shop -o jsonpath='{.data.password}' | base64 -d
S3cret
#ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get /registry/secrets/shop/db | hexdump -C | head

6.2

Turn on encryption at rest

An EncryptionConfiguration tells the API server to encrypt secrets before they go to etcd. The first provider in the list is used to write.

$head -c 32 /dev/urandom | base64

/etc/kubernetes/enc/enc.yaml

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: [secrets]
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: BASE64_32_BYTE_KEY
      - identity: {}

Then in /etc/kubernetes/manifests/kube-apiserver.yaml:

add --encryption-provider-config=/etc/kubernetes/enc/enc.yaml,
add a volume (hostPath /etc/kubernetes/enc) and a volumeMount with the same path.

$kubectl get secrets -A -o json | kubectl replace -f -
#ETCDCTL_API=3 etcdctl ... get /registry/secrets/shop/db | hexdump -C | head -3
00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 73 68 6f 70 2f 64  62 0a 6b 38 73 3a 65 6e  |s/shop/db.k8s:en|
00000020  63 3a 61 65 73 63 62 63  3a 76 31 3a 6b 65 79 31  |c:aescbc:v1:key1|

Remember

Existing secrets stay unencrypted until they are written again — that is what kubectl replace does. The value in etcd starts with k8s:enc:aescbc:v1:.

Watch out

identity first in the list = no encryption. Keep identity last, only to read old data.

6.3

Limit who can read secrets

get, list and watch on secrets are as strong as the secrets themselves.

$kubectl auth can-i get secrets -n shop --as system:serviceaccount:shop:default
$kubectl get rolebindings,clusterrolebindings -A -o wide | grep -i secret

6.4

Use secrets as files

Environment variables leak easily (crash dumps, env in debug output, child processes). Mounted files can be read-only and updated without a restart.

pod (secret as file)

spec:
  containers:
    - name: api
      image: ghcr.io/s0sta/api:2.0
      volumeMounts:
        - { name: db, mountPath: /run/secrets/db, readOnly: true }
  volumes:
    - name: db
      secret: { secretName: db, defaultMode: 0400 }

6.5

Use an external secret store

Keep the source of truth in a real secret manager; sync or mount secrets into the cluster.

Tool	How
External Secrets Operator	Syncs from Vault, Azure Key Vault, AWS Secrets Manager into Kubernetes Secrets
Secrets Store CSI Driver	Mounts secrets from the store as files, no Kubernetes Secret needed
Vault Agent Injector	A sidecar writes secrets into the pod
KMS provider	etcd encryption with a key held in a cloud KMS

All commands in this chapter