$ run as a normal user# run as root (or with sudo)
Give pods GPUs
The NVIDIA device plugin (or the GPU Operator) makes GPUs a schedulable resource: nvidia.com/gpu.
helm install gpu-operator nvidia/gpu-operator -n gpu-operator --create-namespace
kubectl describe node gpu1 | grep nvidia.com/gpugpu-job.yaml
apiVersion: batch/v1
kind: Job
metadata: { name: train-gpu, namespace: ml }
spec:
template:
spec:
restartPolicy: Never
tolerations: [{ key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }]
containers:
- name: train
image: ghcr.io/s0sta/train:1.2
resources:
limits: { nvidia.com/gpu: 1, memory: 16Gi }At work
Taint GPU nodes so only GPU jobs land there — GPUs are expensive to leave idle or to fill with web pods.
Know Kubeflow
Kubeflow is a set of ML tools on Kubernetes.
| Part | Job |
|---|---|
| Notebooks | Jupyter servers per user, with GPUs |
| Pipelines | Training pipelines as containers |
| Katib | Hyperparameter tuning |
| Training Operator | Distributed training (PyTorch, TensorFlow) |
| KServe | Model serving |
Share the cluster fairly
Many teams, few GPUs: use namespaces, quotas and queues.
- One namespace per team with a
ResourceQuotaonrequests.nvidia.com/gpu. - Priority classes: production inference above training above experiments.
- Batch queues (Kueue) start jobs when GPUs are free.
gpu-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata: { name: gpu, namespace: team-a }
spec:
hard:
requests.nvidia.com/gpu: "2"Secure the ML platform
The same controls as any platform, plus data protection.
- Pod Security
restrictedfor notebooks and jobs where possible; no privileged GPU pods for users. - NetworkPolicies: training jobs reach only the data store and the registry.
- Workload identity (no keys in notebooks) to read data from cloud storage.
- Signed images and models; scan large base images (CUDA images have many packages).
- Audit who downloaded which model and dataset.
Plan your first MLOps project
The year-2 portfolio project (see the path).
- A small model trained by a nightly pipeline, tracked in MLflow.
- A FastAPI model server image, scanned and signed in CI.
- Deployed on the home cluster with Argo CD; canary for new versions.
- Prometheus metrics, a Grafana dashboard and a drift report.