I hit the wall when one noisy signal kept dragging candidate selection around like a shopping cart with one bad wheel. On paper, the composite score looked stable. Then I watched different weight sets compete on real samples, and it was obvious that I had been tuning a judgment function by instinct, hoping the surface underneath was kind.
The idea behind the calibrator
What changed my approach was treating the weights as a search space rather than a set of sacred constants. The reward mixer already exposes five signals (visualDrift, colorHarmony, motionContinuity, compositionStability, and narrativeCoherence), and the calibrator sits above that layer to ask a different question: which combination of weights best matches the benchmark data?
That's a very different job from adjusting a threshold. A threshold says accept or reject at this line. A calibrator asks what the system should believe more or less strongly, given these signals. The distinction matters because the calibrator is searching the parameter surface of the judgment function itself. Moving a gate is one thing. This reshapes the scorer the gate depends on.
The reward mixer starts with explicit, typed signals, and that structure is what makes calibration possible in the first place.
export interface RewardSignals {
/** CLIP embedding cosine similarity to source frame (0-1) */
visualDrift: number | null
/** Color palette consistency score (0-1) */
colorHarmony: number | null
/** Motion direction alignment score (0-1) */
motionContinuity: number | null
/** Spatial composition stability score (0-1) */
compositionStability: number | null
/** LLM-derived narrative coherence score (0-1) */
narrativeCoherence: number | null
}
What I like about this shape is that every signal is visible, named, and separable. That makes the failure mode obvious too: if one signal is noisy, it doesn't deserve to dominate the whole score just because it arrived with confidence.
How I combine the signals before calibration
The non-obvious part is that the composite score can't assume every signal is present. In the calibrator, null values get skipped and the weights are renormalized over whatever remains. That guardrail keeps a missing or unavailable signal from pretending it should still count.
Here's the core scoring helper from the calibrator.
function computeComposite(
weights: RewardWeights,
signals: RewardSignals
): number {
let totalWeight = 0
let weightedSum = 0
for (const key of SIGNAL_KEYS) {
const signal = signals[key]
if (signal !== null && signal !== undefined) {
const w = weights[key]
weightedSum += w * signal
totalWeight += w
}
}
return totalWeight > 0 ? weightedSum / totalWeight : 0
}
That renormalization does more work than its handful of lines suggests. Multiply every signal by its weight and sum the result, and a missing one behaves like a silent penalty. Instead, the score comes from the evidence that actually exists, and the denominator follows it.
The calibrator's own config spells out the bounds the search runs under.
const DEFAULT_CONFIG: Required<CalibratorConfig> = {
gridSteps: 5,
refinementIterations: 10,
refinementStep: 0.02,
minWeight: 0.05,
maxWeight: 0.50,
normalizeWeights: true,
holdoutFraction: 0.2,
}
Those bounds keep any one signal from swallowing the rest during the search. Explore, yes. Wander into a regime where a single weight becomes the whole story, no.
What the search actually does
Two strategies, in order: a grid search, then coordinate descent. The grid is broad and exhaustive over a discretized weight space. Coordinate descent picks up from the best grid point and nudges one dimension at a time until the score stops improving.
I trust that more than a single clever optimizer, because it leaves me with a baseline and a refinement pass. I can see whether the coarse search found a decent region before I ask the local search to polish it.
I wanted the calibrator to feel like a lab bench rather than a black box. Try a weight set, score it against the samples, keep what wins, move on. That makes the search legible when I inspect it later, which is exactly when these systems usually try to become mysterious.
The calibrator's result type shows that I care about more than the final weights.
export interface CalibrationResult {
/** Optimized weights */
weights: RewardWeights
/** Accuracy on holdout set (if k-fold used) */
accuracy: number
/** Correlation between predicted composite and actual quality */
correlation: number
/** Number of samples used */
sampleCount: number
/** Number of weight configurations evaluated */
configurationsEvaluated: number
/** Phase 1 vs Phase 2 improvement */
gridBestAccuracy: number
refinedAccuracy: number
}
gridBestAccuracy and refinedAccuracy are the before-and-after story. Both hold the combined objective the calibrator actually optimizes, which weights accuracy at 70% and Pearson correlation with the quality score at 30%. If the two numbers land close together, refinement didn't earn its keep. If they diverge, coordinate descent found a ridge the grid stepped past. accuracy and correlation on their own are the holdout-set read of the final weight set, computed after the train/validation split. configurationsEvaluated is less glamorous but load-bearing for debugging: at gridSteps: 5, Phase 1 alone walks 3,125 weight vectors, so the count tells me whether Phase 2 contributed fifty more candidates or five hundred. The difference between a run that bounced off its bounds immediately and one that actually polished the grid point. sampleCount is the denominator everything else is calibrated against, and below ten the calibrator short-circuits to defaults.
Why this isn't hand-tuning
Hand-tuning starts with a belief and tries to make the numbers agree with it. Calibration starts with the data and asks which belief survives contact with the benchmark set.
The calibrator operates on logged samples from real pipeline runs. Each sample carries the raw reward signals, a quality score, and whether the candidate was accepted. So the search has something to push against: examples where the pipeline already revealed something about what "good" looked like.
export interface CalibrationSample {
/** Raw reward signals for this candidate */
signals: RewardSignals
/** Human or automated quality assessment (0-1) */
qualityScore: number
/** Was this candidate selected/accepted? */
accepted: boolean
/** Optional metadata for filtering */
sceneType?: string
timestamp?: number
}
The limitation worth stating plainly is that calibration is only as honest as the samples feeding it. If the benchmark data is narrow, the search can still find a polished lie. None of this makes the calibrator a substitute for judgment. What it does is make the judgment function answer to evidence instead of habit.
What Phase 1 already told me
The weights in production today (visualDrift: 0.30, colorHarmony: 0.25, motionContinuity: 0.15, compositionStability: 0.15, narrativeCoherence: 0.15) came from the Phase 1 baseline run on 2026-02-21 across 100 contracts, and formal hyperparameter search was deferred in that phase while the HITL gate was calibrated first. But Phase 1 surfaced exactly the kind of signal evidence the calibrator is built to reconcile. On the 69 GPU-rendered contracts, visualDrift averaged 0.529 and drove zero HITL flags, while narrativeCoherence averaged 0.317 and drove 65.2% of them. motionContinuity came in at 0.387 with another 17.4%. The two signals sitting at weight 0.15 together accounted for over 80% of the flagging, while the signal carrying the largest weight at 0.30 passed everything.
That's the distribution the calibrator exists to rebalance. A weight vector that puts its largest share on the signal doing no discrimination isn't a tuned instrument. It's a ritual. The correlation field is exactly the statistic that would catch it: Pearson correlation between the composite score and the logged quality score, which punishes weight vectors that let a flat signal dominate.
The baseline comparison is the control
The sharpest part of the design is that before-and-after comparison. The calibrator doesn't stop at reporting a weight vector and calling it done. It compares the refined result against the grid-search baseline, which makes the improvement legible instead of assumed.
That matters because search algorithms can easily produce movement without progress. A local refinement step can feel productive while it circles the same hill. Keeping the baseline in view, I can tell whether the search surface rewarded the new weights or merely entertained the optimizer.
The code comments are explicit about the two phases: coarse grid search first, then coordinate descent around the best grid point. That sequence isn't an accident. The grid gives me coverage, the refinement gives me precision. Skip the grid and I'm polishing a guess. Skip the refinement and I'm leaving useful accuracy on the table.
The guardrails are doing real work here. The minimum and maximum weight bounds keep the search from overcommitting to one signal. The normalization option keeps the total scale stable. And the null-skipping composite score means missing evidence doesn't masquerade as a negative vote.
Architecture Overview
Where the calibrator fits in the larger scoring stack
The reward calibrator extends the evaluation layer that already handles quality-predictor weights, but it targets the reward weights used by the multi-signal reward mixer. That separation is what keeps the architecture clean: the mixer defines how scores are combined, and the calibrator searches for better parameters for that combination.
In practice, I can improve the judgment function without rewriting the signals themselves. If the system learns that composition stability should matter a little more than motion continuity on the benchmark corpus, the calibrator can discover that. If the opposite is true, it can discover that too. I don't need to hard-code a philosophical stance into the scorer.
That's the real trick. A threshold decides where a line sits. The calibrator is learning how much each of its own senses should count.
What comes out is a scoring layer that reads less like a fixed rulebook and more like a tuned instrument: bounded, testable, and willing to admit when one note has been playing too loudly for too long.