Detecting Model Degradation: Drift, Feedback Loops & Golden Evals
Models don’t crash like servers do — they rot. Three ways a deployed model quietly degrades, and the three monitors that catch it: distribution checks, golden evals, and user-behavior proxies.
Explain it like I’m five
Imagine a friend with perfect restaurant taste. In 2022, every recommendation was gold. By 2024, half their picks are duds. Your friend didn’t forget how to judge food — the city changed around them. Old favorites closed, new neighborhoods opened, and their mental map of the city is frozen in 2022.
That frozen mental map is a deployed model. It can go stale three ways. First, the city changed but the rules of good taste didn’t — your friend keeps recommending downtown when everyone moved uptown. Second, your taste changed — the same restaurants they always picked no longer count as “great,” and no city map can fix that. Third, they only ever eat at places they already recommended, so their picture of the city narrows until it’s just their own echo. And here’s the scary part: your friend never gets an error message. They feel confident the whole way down. Detecting model degradation is learning to notice the rot while everything still looks fine.
Intuition: the distinguishing test
Your fraud model aced offline validation. Three months later, chargebacks are climbing and nobody can say when the slide began. The question interviewers actually ask isn’t “define drift” — it’s “the world moved or the truth moved: how do you tell the difference when ground-truth labels arrive weeks late?”
Start with the formal split, because everything else is a procedure for checking it. Data drift (covariate shift) means the input distribution P(X) changed while the mapping from inputs to answers P(y|X) stayed put. The model is still “right” where it was trained — it’s just operating in a neighborhood of feature space it never visited. Concept drift means the mapping itself changed: P(y|X) moved, so the right answer for a given input is different now. The data looks normal. The truth moved.
Here is the crisp distinguishing test, and it’s worth saying aloud in an interview. Hold the model fixed. Compare live input distributions against the training baseline: if P(X) shifted and the errors concentrate in the new region, it’s data drift — the model is extrapolating. If the inputs look identical to training but the error rate on a fresh labeled probe has climbed anyway, it’s concept drift — you need new labels, and no amount of input monitoring will find it. The trap interviewers set is asking which one you can detect without labels: data drift, always; concept drift, never — you can only proxy it until ground truth lands.
Then there is the third failure mode, the sneaky one. A feedback loop happens when the model’s own outputs reshape the data it later trains on — the recommender trains on what it recommended, the hiring filter only ever sees candidates it let through. The model eats its own exhaust, and the bias compounds with each retraining cycle. Sculley et al. named these “hidden feedback loops” in the systems literature: models that act on the world while learning from it degrade in ways that are especially hard to catch in quick offline experiments, because every experiment runs inside the loop.
Models don’t rot on a schedule — detect degradation with distribution monitors, golden evals, and user-behavior proxies. Then alert and roll back.
How the three monitors work
No single monitor catches all three failure modes, because each mode breaks a different assumption. Distribution watches catch the world moving. Golden evals catch the truth moving. User-behavior proxies catch everything the other two miss — including the failure modes you never thought to name.
Monitor 1 — watch the distributions
Compare live feature histograms against the training baseline, bin by bin. The workhorse metric is the Population Stability Index: for each bin, take the difference in percentage points between the live and reference proportions, multiply by the log of their ratio, and sum across bins — PSI = Σ (Actual% − Expected%) · ln(Actual% / Expected%). A PSI of zero means identical distributions; it grows as mass moves into bins that were rare in training. The classic rule of thumb from credit-scorecard practice: below 0.1 the population is stable, 0.1–0.25 means watch it closely, above 0.25 the shift is material. Treat those bands as a heuristic, not a theorem — bin choices and sample sizes dominate interpretation, and some teams draw the action line at 0.2.
Monitor the model’s score distribution too, not just features. A shift in P(ŷ) — prediction drift — with stable inputs is suspicious: it suggests the model is behaving differently for reasons the feature monitors can’t see, and it’s often the first thing to move. When you do have labels, the label distribution itself is worth a chi-square check.
For the metric itself, PSI and KL divergence are the usual suspects, and the tradeoff is worth knowing cold. KL divergence, KL(P‖Q), is asymmetric — it changes when you swap the reference and the live window — and unbounded, which makes absolute alert thresholds awkward. PSI is symmetric in role assignment and stays in a comfortable band for dashboards, but it is entirely a function of your bin definitions: change the bins, change the verdict. Engineers who want the best of both often reach for the Jensen–Shannon divergence — symmetric like PSI, bounded between 0 and 1 — because a bounded metric makes threshold-setting sane. Whichever you pick, bin with quantiles from the reference window so each bin had real mass in training, smooth empty bins with a small epsilon, and recompute on a schedule, not on vibes.
Monitor 2 — golden evals on a schedule
A golden eval is a fixed, held-out probe set scored on a schedule — daily, hourly, every deploy. Not your training benchmarks: the model may have seen those, and even if it hasn’t, benchmarks freeze the world at collection time while the real world keeps moving. A golden set is a canary: a small, carefully maintained sample that reflects what the model should get right today, graded in production, compared against a baseline with an alert threshold — say, accuracy dropping more than two points below baseline, or below an absolute floor.
This is the only monitor that catches concept drift directly, because it’s the only one with ground truth. Distribution monitors can stay perfectly green while the truth moves — the inputs look normal, after all. The golden set is also your deploy gate: new models go out as a canary deploy to a fraction of traffic, and you compare golden scores and user proxies before full rollout. When an alert fires, the default move is a one-click rollback — the blast radius of a bad model should be a config change, not an incident.
Golden evals are evals with a job — the design playbook for building a set that actually discriminates (held-out items, human-verified answers, versioning) is the same one in the LLM evals guide. What changes here is the cadence: the set is fixed, and the scoreboard is a time series you watch, not a one-time leaderboard.
Monitor 3 — watch the users
The cheapest signal of all: user behavior. For an LLM assistant, a climbing regeneration rate or thumbs-down rate. For a recommender, click-through on recommended items sagging. For a fraud system, dispute and chargeback rates creeping up. Sessions dying early, support tickets with a new flavor, usage quietly declining — users feel degradation before any dashboard does, because they’re the ground truth with the lowest latency.
The honest caveat is confounding. A thumbs-down spike can come from a UI change, a regen-rate bump from a prompt tweak, a chargeback wave from a merchant change upstream. Treat proxies as smoke alarms, not thermometers: they tell you to look, not what’s burning. Their real power is triage — a proxy alarm with clean distributions and a flat golden eval points at instrumentation or product, while a proxy alarm that agrees with another monitor is degradation you can act on.
Three independent signals — if two of them agree, the degradation is real.
When an alert fires
Thresholds need severities, or everything becomes noise. A practical ladder: PSI drifting into 0.1–0.25 opens a ticket for investigation; PSI above 0.25 or a golden-eval drop below baseline pages whoever owns the model; a proxy alarm alone pages nobody — it enriches whichever of the other two fired. The runbook then branches on the distinguishing test. Inputs shifted → it’s data drift: hunt upstream first, because the most common root cause is a pipeline change — a new traffic source, a schema tweak, a partner feed that changed format — and retraining on poisoned or shifted data without fixing the pipe just re-bakes the problem. Inputs stable but the golden eval slid → concept drift: the truth moved, and the fix is fresh labels and a retrain, possibly a feature rebuild if the old signals no longer discriminate. And always audit for feedback loops: if model outputs feed anywhere near the training pipeline, the “drift” you’re measuring may be the model measuring itself.
Retraining triggers are policy, not heroics. Typical triggers: drift beyond threshold plus golden-eval decline, a scheduled refresh cadence that matches how fast your world moves, and a hard freshness rule — no model serves stale past its review date. The canary deploy is what makes all of this safe: every model change ships to a slice of traffic first, and the same three monitors that caught the rot confirm the fix.
Three ways models rot in the wild
Every offline-validated model eventually meets the same three failure shapes — and in each case the model that passed every lab test fails silently, gradually, for reasons invisible to training-time metrics.
Data drift. A fraud model trains on mostly domestic transactions, where the learned patterns hold. Summer travel season hits and suddenly a large share of traffic is foreign card swipes — a region of feature space the model barely visited in training. Precision tanks on the unfamiliar slice while aggregate metrics look tolerable. This is the canonical data-drift failure: the mapping is still valid, the inputs moved. Per-feature PSI alerts fire weeks before the chargeback metrics move.
Feedback loops. A recommender trains on engagement data — clicks on items it showed. But it only showed items it already believed in, so the training labels are the model’s own past decisions. Each retraining cycle narrows the catalog the model is willing to try, clicks concentrate on the shown subset, and the bias compounds. New or niche items never get the impressions that would let them prove themselves. The systems fix, articulated by Sculley et al., is to detect feedback loops and remove them wherever possible — in practice that means randomized exploration traffic, a small slice of impressions allocated by chance rather than by the model, so training data includes items the model would never have chosen.
Concept drift. A demand model learns the office-lunch rush: weekday midday spikes, dead weekends. Then remote work reshapes the week — Tuesdays start looking like Saturdays — and the inputs look perfectly normal while the right answer for those inputs has moved. PSI stays green. Only a golden eval on fresh labels catches it, because concept drift is invisible to every monitor that doesn’t carry ground truth. Retraining here isn’t a refresh; the old features no longer discriminate, so the feature set itself needs a rebuild.
Source: Arize AI, “What Is Population Stability Index (PSI)” — the <0.1 / 0.1–0.25 / >0.25 rule-of-thumb bands ↗
Three different failure shapes, one pattern: the lab-passed model degrades silently, gradually, and for reasons its training metrics can’t see — which is why distribution monitors are the early warning and golden evals are the truth check.
Notice what each example teaches about detection, not just failure. Fraud shows why you monitor features and scores — the score distribution moved first. The recommender shows why offline experiments can’t find loop-driven rot — every experiment runs inside the loop. The demand forecast shows why golden evals are non-negotiable — they’re the only monitor that sees the truth move.
PSI by hand: five buckets, one verdict
One fraud-model feature — transaction amount. The training baseline (expected) says most traffic is small; this week’s live window (actual) says otherwise. Expected and actual are per-bucket proportions; each bucket contributes (Actual − Expected) · ln(Actual / Expected).
Two things to take from the arithmetic. First, the biggest contributions came from the rarest training bins — the model is now seeing two-and-a-half times as many $2K+ transactions as it trained on, exactly the extrapolation region where it’s weakest. Second, the thresholds are a rule of thumb from credit-scorecard practice, not a theorem: some teams draw the action line at 0.2 instead of 0.25, and bin choices dominate the number. In production you run this per top feature plus the model’s score distribution — the feature that’s drifting usually names the upstream cause.
A drift-and-evals harness in fifty lines
import math
from statistics import mean
def psi(expected, actual, eps=1e-4):
# Population Stability Index between two binned distributions.
# expected: training-window proportions; actual: live-window proportions.
# Rule of thumb: <0.1 stable, 0.1-0.25 watch, >0.25 investigate.
total = 0.0
for e, a in zip(expected, actual):
e, a = max(e, eps), max(a, eps) # epsilon keeps empty bins from blowing up the log
total += (a - e) * math.log(a / e)
return total
def run_golden_eval(score_fn, canary_set, baseline, drop_threshold=0.02):
# Score a fixed canary set; alert if the score slides vs the baseline.
# Never reuse training benchmarks here — the model may have seen them.
scores = [score_fn(x, y) for x, y in canary_set]
current = mean(scores)
drop = baseline - current
return {
"score": round(current, 4),
"drop_vs_baseline": round(drop, 4),
"alert": drop >= drop_threshold,
}
def check_user_proxies(live, baseline, rel_threshold=0.10):
# Compare cheap behavior signals: regen rate, thumbs-down, session churn.
# Proxies move first; they are smoke alarms, not thermometers.
rel_moves = {}
for name, cur in live.items():
base = baseline.get(name, 0) or 1e-9
rel_moves[name] = round((cur - base) / base, 3)
tripped = [n for n, rel in rel_moves.items() if rel >= rel_threshold]
return {"relative_moves": rel_moves, "tripped": tripped}
if __name__ == "__main__":
exp = [0.50, 0.25, 0.15, 0.07, 0.03] # training buckets
act = [0.35, 0.25, 0.20, 0.12, 0.08] # this week's live traffic
print("PSI:", round(psi(exp, act), 4)) # -> 0.1439: moderate, investigateThe shape of the harness is the real lesson: three independent functions, three independent signals, one schedule. In production, psi() runs per top feature and on the score distribution, run_golden_eval() runs after every deploy and on a cron, and check_user_proxies() runs continuously — and the alerting rule is that any two agreeing means the degradation is real.
Six questions that test the real understanding
What to say (≈90 sec): “Data drift is a change in the input distribution, P(X), with the mapping P(y|X) intact — the model is right where it was trained, it’s just seeing a new neighborhood. Concept drift is a change in the mapping itself — the right answer for a given input moved. The distinguishing test: hold the model fixed and compare live inputs to the training baseline. If P(X) shifted and errors concentrate in the new region, it’s data drift. If inputs look identical to training but the error rate on a fresh labeled probe climbed anyway, it’s concept drift. And that split answers the second half: data drift is always detectable without labels, concept drift never is — you can only proxy it with user-behavior signals until ground truth lands.”
Likely follow-up: “So how do you detect concept drift at all?” → With a golden eval — a fixed, held-out canary set scored on a schedule. It’s the only monitor that carries ground truth, which is exactly why it’s the only one that sees the truth move.
The answer that sinks you: “You detect both with PSI on the features.” Why it fails: PSI compares input distributions — it is definitionally blind to a world where the inputs didn’t move and only the answers did. That’s the concept-drift trap.
What to say (≈90 sec): “Because they corrupt the measuring stick. A recommender trains on engagement data — clicks on items it showed — but it only showed items it already believed in, so the training labels are the model’s own past decisions wearing a costume. Each retraining cycle narrows the catalog, clicks concentrate on the shown subset, and the bias compounds. The sneaky part is that every offline experiment runs inside the loop, so the model looks great on its own historical data — it’s being graded on an exam it wrote. Sculley et al. flagged these as hidden feedback loops precisely because they degrade in ways that are hard to notice in quick experiments. The fix is structural: detect feedback loops and remove them wherever possible — in practice, randomized exploration traffic, a slice of impressions allocated by chance, so the training data includes items the model would never have chosen.”
Likely follow-up: “Give another example beyond recommenders.” → A hiring filter that only ever sees candidates it let through: it never learns what a good rejected candidate looked like, so it can’t improve its rejections — and its training data says its accept rate is justified by construction.
The answer that sinks you: “Just retrain more often.” Why it fails: retraining inside a feedback loop doesn’t break the loop — it amplifies it. More retraining cycles just compound the bias faster.
What to say (≈90 sec): “The formula difference first: PSI is the symmetric sum over bins of (actual − expected) times the log ratio, while KL divergence is asymmetric — KL(P‖Q) is not KL(Q‖P) — and unbounded, which makes absolute alert thresholds awkward. I default to PSI for feature and score monitoring because it’s symmetric in role assignment and lives in a dashboard-friendly band, with the classic rule of thumb: under 0.1 stable, 0.1 to 0.25 watch closely, above 0.25 investigate. But I’m honest about the weakness: PSI is entirely a function of the bin definitions — change the bins, change the verdict — so I bin with quantiles from the reference window, smooth empty bins with an epsilon, and never let anyone treat the bands as a theorem. If I need a principled bounded metric for threshold-setting, I reach for Jensen–Shannon — symmetric like PSI, bounded between 0 and 1 — and if I need a statistical test rather than a drift index, the two-sample Kolmogorov–Smirnov test on the CDFs.”
Likely follow-up: “Why not just use KL everywhere?” → The asymmetry is the problem: swapping which window you call the reference changes the number, and the unbounded range means ‘alert at KL > x’ has no portable meaning across features.
The answer that sinks you: “They’re basically the same thing.” Why it fails: it erases exactly the tradeoff the question probes — symmetry and boundedness, and the binning dependence that dominates PSI interpretation.
What to say (≈90 sec): “A golden eval is a fixed, held-out probe set that reflects what the model should get right today, scored on a schedule — daily, hourly, and on every deploy — with the score tracked as a time series against a baseline and an alert threshold, say two points of degradation. Two reasons you can’t reuse training benchmarks. First, contamination: the model may have trained on them, in which case they measure memory, not current correctness — a benchmark the model has seen is a test it took home. Second, staleness: benchmarks freeze the world at collection time while the real world keeps moving, and concept drift is exactly the failure where the world moved. So I maintain a small canary set separately from all training data, refresh its items as the product evolves, human-verify the answers, version it, and grade it in production. And it doubles as the deploy gate: new models ship as a canary to a fraction of traffic, and the golden score plus user proxies decide whether the rollout continues — with a one-click rollback as the default response to any alert.”
Likely follow-up: “How do you keep the golden set itself from going stale?” → By refreshing items on the same cadence the world moves — dated items get retired and replaced, and the set’s difficulty is recalibrated so a perfect score stays meaningful.
The answer that sinks you: “I’d evaluate on the training data to save labeling cost.” Why it fails: it maximizes the contamination problem — you’re testing the model on data it learned, which measures fitting, not degradation.
What to say (≈90 sec): “You treat proxies as smoke alarms, not thermometers. Regeneration rate climbing, thumbs-down ticking up, sessions dying early — users feel degradation before any dashboard does, which is why the signal is worth having. But it’s confounded: a thumbs-down spike can come from a UI change, a regen bump from a prompt tweak, a chargeback wave from an upstream merchant change. So I never page on a proxy alone. The rule is agreement: a proxy alarm with clean distributions and a flat golden eval points at instrumentation or product, not the model; a proxy alarm that agrees with a distribution alert or a golden-eval drop is degradation I act on. In the harness that means the proxy check returns relative moves against baseline, and the alerting policy requires two of the three monitors to fire before anyone gets paged. The proxies get you the fastest detection; the other two monitors give you the confirmation.”
Likely follow-up: “What proxies would you pick for a fraud model?” → Dispute and chargeback rates, investigator override rates, and the share of transactions sent to manual review — each with the same baseline-and-relative-move treatment.
The answer that sinks you: “More proxies is always better.” Why it fails: every uncurated proxy is a new source of false alarms — without baselines, thresholds, and the agreement rule, you get alert fatigue and the team starts ignoring the monitors that matter.
What to say (≈90 sec): “First, don’t touch the model — contain the blast radius. Check whether this is a canary-deployed change: if a new model version is mid-rollout, one-click rollback immediately and let the monitors confirm the recovery. If no deploy is in flight, run the distinguishing test: the PSI alert says inputs shifted, so hunt upstream first — new traffic source, schema change, partner feed format, anything that altered the input mix — because retraining on shifted or poisoned data without fixing the pipe just re-bakes the problem. If upstream is clean but the golden eval dropped anyway, you may be looking at concept drift riding along: fresh labels and a retrain, possibly a feature rebuild. And either way, audit for feedback loops before you retrain — if model outputs feed anywhere near the training pipeline, the drift you’re measuring may be the model measuring itself. The hour ends with a written record: what fired, what the root cause was, and which monitor caught it first. Retraining triggers are policy — drift beyond threshold plus golden decline — not 2am heroics.”
Likely follow-up: “What if the PSI alert fired but the golden eval is flat?” → Then it’s data drift the model is surviving — monitor, investigate upstream, lower urgency. The model extrapolating well today doesn’t mean it will tomorrow, but there’s no 2am action.
The answer that sinks you: “Retrain immediately on the latest data.” Why it fails: you haven’t diagnosed anything — if the shift is an upstream bug or a feedback loop, you just trained the model to memorize the incident.
Key takeaways
- Data drift is P(X) moving, concept drift is P(y|X) moving — hold the model fixed and compare inputs to tell them apart.
- Feedback loops are the sneakiest failure mode because every offline experiment runs inside the loop; break them with randomized exploration traffic.
- PSI is the workhorse for distribution watches — symmetric and dashboard-friendly, but the 0.1/0.25 bands are a rule of thumb, not a theorem.
- Golden evals are the only monitor with ground truth, which makes them the only monitor that sees concept drift; never reuse training benchmarks as the canary.
- User-behavior proxies are the cheapest, fastest signal — and the noisiest. Require two of the three monitors to agree before you act.
- When an alert fires: contain with a canary rollback, diagnose with the distinguishing test, audit for feedback loops — then retrain as policy, not heroics.