Dit deel is nog niet vertaald. Je ziet de Engelse tekst.
$ als gewone gebruiker# als root (of met sudo)
Watch the service
Export metrics in Prometheus format and alert like for any API.
metrics.py
from prometheus_client import Counter, Histogram, make_asgi_app
PREDICTIONS = Counter("predictions_total", "Predictions served", ["model_version", "result"])
LATENCY = Histogram("prediction_seconds", "Time per prediction")
# in the /predict handler: with LATENCY.time(): ... PREDICTIONS.labels("v3", "ok").inc()
metrics_app = make_asgi_app()| Metric | Alert when |
|---|---|
| Error rate | Above 1 % for 5 minutes |
| p95 latency | Above the agreed limit |
| Requests per second | Sudden drop (clients broken) or spike (abuse) |
| GPU / CPU use, cost | Over budget |
Detect data drift
Drift = the live inputs (or outputs) no longer look like the training data. The model may still answer, but worse.
| Drift | Example |
|---|---|
| Data drift | Average order value doubled after a price change |
| Concept drift | Fraud patterns changed; old rules learned by the model no longer hold |
| Prediction drift | The model suddenly says "fraud" twice as often |
drift_report.py
import pandas as pd
from evidently import Report
from evidently.presets import DataDriftPreset
reference = pd.read_parquet("training_sample.parquet")
current = pd.read_parquet("last_24h_inputs.parquet")
report = Report([DataDriftPreset()])
result = report.run(current_data=current, reference_data=reference)
result.save_html("drift.html")Measure real quality
When the true answers arrive later (was it really fraud?), join them with the logged predictions and compute the real metrics.
- Log every prediction with an ID, the model version and the inputs (without sensitive data where possible).
- Store the true labels when they come in.
- A daily job computes precision/recall per model version and alerts on a drop.
Close the loop
Monitoring triggers action.
| Signal | Action |
|---|---|
| Drift above the limit | Start the training pipeline with fresh data |
| Quality drop | Roll back to the previous champion |
| Latency / cost problem | Scale, use a smaller model, batch requests |
Onthoud
Monitoring without an owner and an action is just a dashboard. Decide who reacts to every alert.