Monitor models in production — MLOps Chapter 6

$ run as a normal user# run as root (or with sudo)

Watch the service

Export metrics in Prometheus format and alert like for any API.

metrics.py

from prometheus_client import Counter, Histogram, make_asgi_app
 
PREDICTIONS = Counter("predictions_total", "Predictions served", ["model_version", "result"])
LATENCY = Histogram("prediction_seconds", "Time per prediction")
# in the /predict handler:  with LATENCY.time(): ...   PREDICTIONS.labels("v3", "ok").inc()
metrics_app = make_asgi_app()

Metric	Alert when
Error rate	Above 1 % for 5 minutes
p95 latency	Above the agreed limit
Requests per second	Sudden drop (clients broken) or spike (abuse)
GPU / CPU use, cost	Over budget

6.2

Detect data drift

Drift = the live inputs (or outputs) no longer look like the training data. The model may still answer, but worse.

Drift	Example
Data drift	Average order value doubled after a price change
Concept drift	Fraud patterns changed; old rules learned by the model no longer hold
Prediction drift	The model suddenly says "fraud" twice as often

drift_report.py

import pandas as pd
from evidently import Report
from evidently.presets import DataDriftPreset
 
reference = pd.read_parquet("training_sample.parquet")
current = pd.read_parquet("last_24h_inputs.parquet")
report = Report([DataDriftPreset()])
result = report.run(current_data=current, reference_data=reference)
result.save_html("drift.html")

6.3

Measure real quality

When the true answers arrive later (was it really fraud?), join them with the logged predictions and compute the real metrics.

Log every prediction with an ID, the model version and the inputs (without sensitive data where possible).
Store the true labels when they come in.
A daily job computes precision/recall per model version and alerts on a drop.

6.4

Close the loop

Monitoring triggers action.

Signal	Action
Drift above the limit	Start the training pipeline with fresh data
Quality drop	Roll back to the previous champion
Latency / cost problem	Scale, use a smaller model, batch requests

Remember

Monitoring without an owner and an action is just a dashboard. Decide who reacts to every alert.