Detect threats with audit logs and Falco

$ run as a normal user# run as root (or with sudo)

Skills in this chapter

Audit logs, Falco, immutable containers and threat detection

Turn on audit logging

An audit policy says which requests to log and in how much detail. The API server writes them to a file.

/etc/kubernetes/audit/policy.yaml

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: [RequestReceived]
rules:
  - level: None
    resources: [{ group: "", resources: [events] }]
  - level: Metadata
    resources: [{ group: "", resources: [secrets, configmaps] }]
  - level: RequestResponse
    namespaces: [shop]
    resources: [{ group: "", resources: [pods] }]
  - level: Metadata

In kube-apiserver.yaml:

kube-apiserver flags

--audit-policy-file=/etc/kubernetes/audit/policy.yaml
--audit-log-path=/var/log/kubernetes/audit/audit.log
--audit-log-maxage=7
--audit-log-maxbackup=3
--audit-log-maxsize=100

Add hostPath volumes and mounts for the policy file and the log folder.

Level	Logs
None	Nothing
Metadata	Who, what, when — no bodies (use it for secrets!)
Request	Plus the request body
RequestResponse	Plus the response body

#tail -n 2 /var/log/kubernetes/audit/audit.log | jq '{user: .user.username, verb, uri: .requestURI}'
#grep '"resource":"secrets"' /var/log/kubernetes/audit/audit.log | jq -r '.user.username' | sort | uniq -c

Remember

Rules are checked in order; the first match wins. Put specific rules before the catch-all.

9.2

Detect behaviour with Falco

Falco reads system calls (eBPF) and checks them against rules. A match writes an alert (stdout, file, syslog, webhook).

$helm install falco falcosecurity/falco -n falco --create-namespace
#journalctl -u falco -f
$kubectl exec -it web-7d9f -n shop -- sh
#grep "Terminal shell in container" /var/log/syslog | tail -1

/etc/falco/falco_rules.local.yaml

- rule: Write below etc in container
  desc: A process in a container writes to /etc
  condition: open_write and container and fd.name startswith /etc
  output: "File below /etc opened for writing (user=%user.name command=%proc.cmdline file=%fd.name container=%container.name image=%container.image.repository)"
  priority: WARNING
  tags: [container, filesystem]

#systemctl restart falco
#falco --list | grep -i shell

Exam tip

A typical task: change an existing Falco rule's output format or priority, find which pod triggered it, and stop that pod. Local overrides go in falco_rules.local.yaml.

9.3

Make containers immutable

A container should not change at run time: read-only root, no shell tools, no package manager, no privilege.

immutable (container securityContext)

securityContext:
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  runAsNonRoot: true

$kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{" ro="}{.spec.containers[*].securityContext.readOnlyRootFilesystem}{"\n"}{end}' | grep -v "ro=true"

9.4

Investigate a suspicious pod

Contain first, then collect evidence, then remove.

Isolate: label the pod and apply a deny-all NetworkPolicy for that label.
Collect: kubectl logs, kubectl describe, events, audit log entries, Falco alerts, crictl inspect.
Find the source: which image, which service account, which node.
Remove: delete the pod; fix the image or the policy; rotate secrets it could read.

$kubectl label pod web-7d9f -n shop quarantine=true
$kubectl get pod web-7d9f -n shop -o yaml > evidence-pod.yaml
#crictl inspect $(crictl ps --name nginx -q) > evidence-container.json

All commands in this chapter