Serve models as APIs

Dit deel is nog niet vertaald. Je ziet de Engelse tekst.

$ als gewone gebruiker# als root (of met sudo)

Build a model API

Load the model once at start, validate input, return predictions.

app.py

from fastapi import FastAPI
from pydantic import BaseModel
import joblib
 
app = FastAPI()
model = joblib.load("model.joblib")
 
 
class Flower(BaseModel):
    features: list[float]
 
 
@app.get("/health")
def health():
    return {"status": "ok"}
 
 
@app.post("/predict")
def predict(f: Flower):
    return {"class": int(model.predict([f.features])[0])}

$pip install fastapi uvicorn scikit-learn joblib
$uvicorn app:app --host 0.0.0.0 --port 8080
$curl -s -X POST localhost:8080/predict -H 'Content-Type: application/json' -d '{"features": [5.1, 3.5, 1.4, 0.2]}'
{"class":0}

5.2

Package it in a container

The same rules as any image: small base, non-root, pinned packages, health check.

Dockerfile

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py model.joblib ./
USER 1001
EXPOSE 8080
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"]

5.3

Choose online or batch

Not every model needs a live API.

Mode	When	How
Online (real time)	An app needs an answer now	API behind a load balancer, autoscaling
Batch	Score many rows every night	A job (CronJob, pipeline step) writes results to a table
Streaming	Events one by one	A consumer on Kafka / Event Hubs

5.4

Serve on Kubernetes with KServe

An InferenceService object gives a model an endpoint with autoscaling (also to zero) and canary rollout.

inference.yaml

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: iris
  namespace: models
spec:
  predictor:
    model:
      modelFormat: { name: sklearn }
      storageUri: "s3://models/iris/v3"
      resources:
        limits: { cpu: "1", memory: 1Gi }

$kubectl apply -f inference.yaml
$kubectl get inferenceservice iris -n models

5.5

Release models safely

Use the same release patterns as for apps.

Pattern	Use
Shadow	The new model gets copies of real requests; answers are only logged
Canary	5 – 10 % of traffic goes to the new model, then more
A/B test	Two models compared on a business metric
Rollback	Switch the alias / traffic back to the old version

Alle commando's in dit hoofdstuk