The GPU and Kubernetes cheat sheet

Quick reference

GPUs

$nvidia-smi
$nvidia-smi -L
$nvidia-smi topo -m
$nvidia-smi --query-gpu=name,memory.used
$watch -n 2 nvidia-smi

Nodes

$kubectl describe node gpu-0
$kubectl describe node gpu-0 | grep -i gpu
$kubectl get node gpu-0 -o yaml
$kubectl label node gpu-0 pool=a100
$kubectl describe node gpu-0 | tail -20

Pods

$kubectl get pods -A -o wide
$kubectl exec -it gpu-pod -- nvidia-smi
$kubectl logs -f gpu-pod
$kubectl describe pod gpu-pod
$kubectl get pod gpu-pod -o yaml

Checks

$kubectl get events -A
$kubectl get events -A --watch
$kubectl get pods -A | grep Pending
$kubectl top pod -A --sort-by=memory
$kubectl get events -A -o wide

Explained

GPUs

Start at the card. nvidia-smi shows utilisation, memory and temperature; nvidia-smi -L lists the devices; nvidia-smi topo -m prints the topology, which tells you whether two GPUs talk over NVLink or PCIe. The query form, nvidia-smi --query-gpu=name,memory.used, is the one to remember.

Nodes

Then the node. kubectl describe node gpu-0 shows capacity and allocatable, including nvidia.com/gpu, and the events at the bottom explain why a card is not schedulable. kubectl get node gpu-0 -o yaml is the full record, and kubectl label node gpu-0 pool=a100 is how you pin a workload to a class of card.

Pods

Then the workload. kubectl get pods -A -o wide shows where it landed. kubectl exec -it gpu-pod -- nvidia-smi shows what the container can actually see, and kubectl logs -f gpu-pod or kubectl describe pod gpu-pod explains a crash or an eviction.

Checks

Then the cluster. kubectl get events -A is the history, and kubectl get pods -A | grep Pending finds the jobs waiting for a card. kubectl top pod -A --sort-by=memory needs metrics-server, and it catches the pod that is running but not working.

In one line

Start at the card

$kubectl exec -it gpu-pod -- nvidia-smi