Skip to content
← Back to Blog
ml-systemsrankingnormalizationtypescriptprompt-engineering

MR‑GRPO in Practice: The Reward Mixer That Stops CLIP From Lying to Your Scene Compiler

Daniel Anthony Romitelli Jr. · March 4, 2026

CLIP is a single thermometer taped to the outside of a house. It tells you something about the temperature. It can't tell you whether the furnace is on, whether a window is open, or whether the kitchen is on fire. In my pipeline, CLIP-only ranking produced the same kind of failure: candidates that "looked similar" in embedding space while violating constraints that mattered for continuity. (See Radford et al. for the original CLIP formulation and caveats about what an embedding captures.)

I found this out the hard way. I built Scenematic to compile scenes rather than prompts alone, and that means I needed a scoring system that could survive reality. Reality looks like this. I generate multiple candidate prompts for a scene, and I need to pick one that will preserve continuity constraints across a directed scene graph: character identity, camera style, color grade. My first scoring approach was the obvious one, CLIP similarity to a source frame. It worked right up until it didn't.

So I replaced it with what the code calls a Multi‑Signal Reward Mixer (MR‑GRPO inspired), a weighted multi-head scoring system with GRPO-style group-relative normalization.

The feature in this post is the mixer itself. Not the downstream generation, not the prompt compiler, not the post-processing. Just the algorithmic core that turns messy, partly missing signal outputs into a single decision.

Key insight: normalize within the candidate group, not against a fantasy global scale

The non-obvious part of this system isn't "use more signals." Everyone says that.

The trick is refusing to trust raw signal magnitudes across candidates.

Even when a signal is documented as "0–1", its effective distribution can move around depending on scene category, prompt structure, or the quirks of whatever analyzer produced it. Combine raw values naively and one of them can dominate, either because its variance is higher or because it saturates near 1.0 most of the time.

So the mixer stays deliberately local. It evaluates each candidate across multiple independent reward signals, then, for each signal head, computes population statistics across the candidate group: mean and standard deviation. Each candidate's head score becomes a normalized value through z-score normalization. The code is explicit about this, calling it "Per-head z-score normalization (Phase 2 calibration fix)". Weights compose the final score only after that.

Production pipelines are messy, so two more things matter more than they sound. When a candidate lacks a signal, value null, that head is ignored entirely for that candidate. And when some heads are missing, the weights get re-normalized over the ones that remain, so the score stays comparable. That pairing is what makes the mixer feel "stable" in practice: it doesn't hallucinate certainty, and it doesn't punish candidates for missing data.

How it works under the hood

The implementation lives in the reward mixer module.

The file header is blunt about what changed. It replaces single-dimensional CLIP-only candidate scoring, evaluates candidates across five independent reward signals, and scores using GRPO-style group-relative normalization.

The type that anchors everything is RewardSignals:

/** Individual reward signal scores for a single candidate */
export interface RewardSignals {
  /** CLIP embedding cosine similarity to source frame (0-1) */
  visualDrift: number | null
  /** Color palette consistency score (0-1) */
  colorHarmony: number | null
  /** Motion direction alignment score (0-1) */
  motionContinuity: number | null
}

What surprised me once this was wired up was how fast null became the "normal" case. Analyzers fail in production. They time out, or they simply can't produce a score for a given candidate, and treating that as a first-class input made the whole selection step dramatically less brittle.

Per-head normalization (mean/std with epsilon)

Phase 2 introduced a specific fix: per-head population statistics for z-score normalization.

Those stats are explicitly modeled:

/** Per-head population statistics for z-score normalization */
export interface HeadStats {
  mean: number
  std: number
}

The intent shows up here ("Per-head z-score normalization (Phase 2 calibration fix)"), and so does the stats structure, but I am not reproducing the exact epsilon constant or the full body of the normalization function. In my codebase those live as functions exported from the reward mixer module, tested in the reward mixer.test module. The internals stay in the repo.

So rather than walk through the math, here's the actual callable surface the tests import: the real names, the real exports, plus a runnable example that documents the input and output shapes.

// example/rewardMixerSurface.ts
// This file is intentionally limited to the exported surface.
// The real implementations live in the reward mixer module.

import {
  computeCompositeScore,
  rankCandidates,
  identifyWeakSignals,
  identifyWeakSignalsPerHead,
  normalizeHeadScore,
  mapSignalToTriggerHead,
  computeSubReason,
  DEFAULT_REWARD_WEIGHTS,
  type RewardSignals,
  type PerHeadHITLConfig,
} from "../lib/reward-mixer"

export type Candidate = {
  id: string
  signals: RewardSignals
}

// A tiny harness that demonstrates how the mixer API is used.
// Note: the internal math is omitted here.
export function scoreAndRank(candidates: Candidate[], hitl?: PerHeadHITLConfig) {
  const scored = candidates.map((c) => {
    // normalizeHeadScore(...) is imported by the tests; body omitted here.
    // computeCompositeScore(...) is imported by the tests; body omitted here.
    const composite = computeCompositeScore(c.signals, DEFAULT_REWARD_WEIGHTS)
    const weak = identifyWeakSignals(c.signals)

    return { ...c, composite, weak }
  })

  // rankCandidates(...) is imported by the tests; body omitted here.
  const ranked = rankCandidates(scored.map(({ id, signals }) => ({ id, signals })))

  return { scored, ranked }
}

The non-obvious detail: I force myself, and future me, to touch the mixer through exported functions that have tests. That's how reward logic avoids turning into a pile of ad-hoc if-statements scattered across the pipeline.

Mapping signals to heads (and why it matters)

The tests also import mapSignalToTriggerHead.

That name is doing a lot of work. It implies the mixer produces structured "why" metadata alongside the scalar score, the kind of metadata you can use to trigger HITL (human-in-the-loop) gates or diagnostics.

Again, that function's body is omitted here. It exists, it's part of the public surface, and the test suite uses it next to identifyWeakSignalsPerHead and computeSubReason.

Here's a runnable example of how I thread that mapping through:

// example/rewardReasons.ts
import {
  mapSignalToTriggerHead,
  computeSubReason,
  type RewardSignals,
} from "../lib/reward-mixer"

export function explainWeakness(signals: RewardSignals) {
  // identifyWeakSignalsPerHead(...) exists; its body is omitted here.
  // Here we demonstrate how the exported mapping helpers would be used.

  const entries = Object.entries(signals) as Array<[keyof RewardSignals, number | null]>

  return entries
    .filter(([, v]) => v !== null)
    .map(([k, v]) => {
      const head = mapSignalToTriggerHead(k)
      const sub = computeSubReason(k, v as number)
      return { signal: k, value: v, head, subReason: sub }
    })
}

What broke for me early on was bolting "explanations" onto the score after the fact. By baking the mapping into the mixer layer, I can keep the scoring decision and the debugging story consistent.

The candidate scoring pipeline: sources → normalizer → mixer → reranker

The mixer is only one box in the pipeline, but it's the box that decides what survives.

Here's the architecture at the level that matters for this post:

I think of this like mixing audio tracks: you don't want the loudest microphone to dominate the whole song just because it's loud. You normalize each track, then mix with intent.

Null-signal skip + weight re-normalization (the "production reality" feature)

In the RewardSignals type, every head is number | null.

That's not a TypeScript nicety. It's a design decision: the mixer must be able to score candidates even when one or more analyzers fail.

The file header explicitly says candidates are evaluated across independent signals, and the Phase 2 work added normalization and per-head handling. The test suite imports the functions behind those behaviors: normalizeHeadScore for per-head normalization, computeCompositeScore for final mixing, and identifyWeakSignals alongside identifyWeakSignalsPerHead for diagnostics.

I am not reproducing the exact weight re-normalization loop here. What I will show is how I structure inputs so that the mixer can do that correctly:

// example/nullSignals.ts
import { computeCompositeScore, DEFAULT_REWARD_WEIGHTS, type RewardSignals } from "../lib/reward-mixer"

const a: RewardSignals = {
  visualDrift: 0.82,
  colorHarmony: null, // analyzer failed or not applicable
  motionContinuity: 0.41,
}

const b: RewardSignals = {
  visualDrift: 0.77,
  colorHarmony: 0.66,
  motionContinuity: null,
}

// computeCompositeScore(...) is responsible for handling nulls.
// The internal logic (skip + re-normalize) is omitted here.
console.log({ a: computeCompositeScore(a, DEFAULT_REWARD_WEIGHTS) })
console.log({ b: computeCompositeScore(b, DEFAULT_REWARD_WEIGHTS) })

The operational point: once you allow null, you stop writing brittle "if missing then score=0" hacks. That one change removed an entire class of silent ranking bugs for me.

Practical extensions

The natural extensions here are per-category reward normalizers, reward clipping, and lightweight calibration.

One of them already has a precedent in the codebase. Per-category calibration exists in the broader system. In the ood detector module, Phase 2 introduced "Per-category thresholds derived from threshold sweep analysis." That's OOD rather than rewards, but it proves the system already supports category-conditioned calibration logic. And the reward mixer carries its own "Phase 2 calibration fix" for per-head normalization.

The rest are not built yet. There is no reward clipping implementation in the mixer: no constants, no function names, no tests. There is no weight-fitting or calibration routine trained from a small human-labeled set either, no file, no function, no artifact. So the extensions below stay limited to the patterns the system already supports, and each one notes where it would need code that does not exist yet.

Extension 1: category-conditioned normalizers

The OOD detector already does category-conditioned thresholding ("Phase 2: Per-category thresholds…"). The same pattern can apply to reward normalization: keep HeadStats per category instead of globally. No code from me on this one, because there's no per-category reward stats store or lookup function in the mixer today.

Extension 2: clamp/diagnose weak heads

identifyWeakSignals and identifyWeakSignalsPerHead are both imported by the tests, which tells you the mixer supports head-level diagnostics. That's the foundation you need for any "don't let one head dominate" policy: you can detect which head is weak or strong, then react. There is no clipping function in the mixer yet, though, so no policy ships with it.

Extension 3: HITL triggers per head

The baseline CSV includes a hitl_trigger column, and the reward mixer exports mapSignalToTriggerHead. Concrete, shipped pattern: on top of the score itself, the mixer can emit structured triggers that the rest of the pipeline can act on.

Closing

CLIP-only ranking felt like steering a car by watching a single wheel. The MR‑GRPO reward mixer is what finally made the whole vehicle track straight, because it scores candidates the way production behaves: multi-factor, partially missing, and always relative to the alternatives you actually have in hand.

RESEARCH SOURCES:

  • Radford, A., Kim, J. W., Hallacy, C., et al. "Learning Transferable Visual Models From Natural Language Supervision" (CLIP). https://arxiv.org/abs/2103.00020. Reference for CLIP, and support for the earlier point about why CLIP similarity can be an incomplete single signal.