9 commands to triage a failing pod

Quick reference

What is the phase?

# phase, restarts and node
$kubectl get pod api-7d9f -n prod -o wide
# the Events block at the end
$kubectl describe pod api-7d9f -n prod
# spec against status
$kubectl get pod api-7d9f -n prod -o yaml

What did it say?

# only a restarted container has logs
$kubectl logs api-7d9f -n prod --previous
# one container, thirty lines
$kubectl logs api-7d9f -n prod -c app --tail=30
# every pod of one label
$kubectl logs -n prod -l app=api --tail=20

Is the node healthy?

# conditions and allocatable
$kubectl describe node ip-10-0-12-44
# usage against allocatable
$kubectl top node ip-10-0-12-44
# evictions across the cluster
$kubectl get events -A --field-selector reason=Evicted

Explained

Look

Start with the phase. The phase is the first fact because it decides which tab can help. kubectl get pod -o wide adds restarts and the node; describe pod ends with the Events block, which names FailedScheduling, a probe failure or a backoff. kubectl get pod -o yaml shows the spec against the status, where a missing claim or a wrong image appears.

Logs

Logs only exist once a container has run. --previous reads the container that died; -c selects one container; -l app=api merges every pod of a label. If the container never started, the log command returns a waiting error; that is a clue, not a broken command.

Node

If the pod looks fine, check the node. describe node lists conditions and allocatable; top node shows usage; get events --field-selector reason=Evicted finds the pods that were pushed out. The field selector keeps the list short; -o wide adds the source node, which is the one to inspect. If the node is healthy, the pod is the problem.

In one line

Try it

$kubectl describe pod api-7d9f -n prod | tail -15