Diversification After Scoring: The Step That Stops My Scene Compiler From Picking Five Paraphrases

A few weeks into running my scene compiler at scale, I hit a pattern I wasn't proud of. I'd ask for five candidates, score them, and the "top five" would come back as one idea wearing five outfits.

The ideas weren't bad. They often carried the highest composite scores. But they clustered so tightly that the selection step was basically doing beam search without admitting it. The result felt like creative mode collapse: the system picking the best single candidate five times over instead of picking the best set.

What makes this worth writing up is where the failure lived. Not in the model call. It sat in the unglamorous part after scoring, in the filtering and selection logic of the pipeline.

Where the missing step fits

The pipeline already has two strong forces:

• A composite score that ranks candidates.
• An uncertainty (OOD-style) gate that can route "risky/uncertain" cases away from a cheaper path and toward a more expensive, higher-fidelity path.

Neither of them cares about set diversity. Composite scoring is pointwise. Uncertainty gating is about risk and cost routing. So the missing piece is a post-score diversification pass:

1. Start from scored candidates.
2. Compute how redundant candidates are with each other.
3. Turn redundancy into a penalty.
4. Apply that penalty after base scoring, then re-rank.
5. Only then do final selection.

That ordering matters. Diversify too early and you distort the very measurements you're trying to score on. Diversify too late, after selection has already happened, and it does nothing.

The pieces already in place

The system includes:

• A scoring and analytics layer that computes statistics over the individual scoring components, for example per-component contribution variance and dominance ratios.
• A continuity "scorecard" concept that gets attached to a scene update and carries fields like candidatesGenerated, routerConfidence, and retryCount.
• A dedicated API surface with route names that strongly suggest separate endpoints for generating storyboards, generating suggestions, and computing embeddings.

What I'm not detailing here are the internals for:

• The exact similarity metric behind "candidate A is too close to candidate B."
• The exact penalty function and its tuned parameters.
• The exact call site where the pipeline currently does "sort by score, take top N."

The wrong assumption underneath it

My first version assumed, in effect:

If I score candidates well enough, the top-N will naturally be diverse.

That assumption is false in any system where scoring reliably converges on an optimum. Once the generator produces several candidates in the same basin, pointwise scoring will happily rank them 1 through 5, and selection hands back a cluster.

I'm not reproducing the exact "top-N slice" line from the selection code here. But the failure mode is consistent with a selection policy equivalent to "sort descending by composite score and take N."

A minimal diversification pass (implementation skeleton)

The obvious ask at this point is a full implementation: metric, penalty formula, defaults, and the exact integration point. What I'm handing over here is smaller and still useful:

• A drop-in interface that makes the diversification pass explicit.
• A clear integration seam: after scoring (and after any routing or gating decision that feeds logging and baselines), before final selection.

Below is an implementation skeleton. Anything marked FILL IN FOR YOUR SYSTEM is left abstract on purpose.

/** A scored candidate produced by your generator + scorer. */
export type ScoredCandidate = {
  id: string
  text: string
  baseScore: number
  // FILL IN FOR YOUR SYSTEM:
  // If you already compute embeddings elsewhere in the system, attach them here.
  embedding?: number[]
}

/** The diversification pass returns candidates with an adjusted score used for final ranking. */
export type DiversifiedCandidate = ScoredCandidate & {
  diversityPenalty: number
  finalScore: number
}

export type DiversificationConfig = {
  // FILL IN FOR YOUR SYSTEM:
  // Define how you measure redundancy (e.g., embedding similarity, string similarity, etc.)
  redundancy: (a: ScoredCandidate, b: ScoredCandidate) => number

  // FILL IN FOR YOUR SYSTEM:
  // Define how redundancy turns into penalty.
  penalty: (redundancyToSelected: number) => number
}

/**
 * Diversify a ranked list by penalizing candidates that are redundant with already-selected ones.
 * Integration point: call this AFTER base scoring (and AFTER routing/gating), BEFORE final top-N.
 */
export function diversifyAfterScoring(
  scored: ScoredCandidate[],
  k: number,
  cfg: DiversificationConfig,
): DiversifiedCandidate[] {
  // Defensive copy + initial sort by base score.
  const pool = [...scored].sort((a, b) => b.baseScore - a.baseScore)

  const chosen: DiversifiedCandidate[] = []

  while (chosen.length < k && pool.length > 0) {
    let bestIndex = 0
    let best: DiversifiedCandidate | null = null

    for (let i = 0; i < pool.length; i++) {
      const cand = pool[i]

      // Redundancy is measured relative to what we've already chosen.
      const maxRedundancy = chosen.length === 0
        ? 0
        : Math.max(...chosen.map(sel => cfg.redundancy(cand, sel)))

      const diversityPenalty = cfg.penalty(maxRedundancy)
      const finalScore = cand.baseScore - diversityPenalty

      const enriched: DiversifiedCandidate = {
        ...cand,
        diversityPenalty,
        finalScore,
      }

      if (best === null || enriched.finalScore > best.finalScore) {
        best = enriched
        bestIndex = i
      }
    }

    if (!best) break

    chosen.push(best)
    pool.splice(bestIndex, 1)
  }

  return chosen
}

What this gives you, mechanically:

• The first pick is basically "best by score."
• Every pick after that pays a penalty for resembling what's already been selected.

Where to hook it in

The right insertion point depends on your code. Here's the inspection checklist I'd use to find it:

1. Find the function that returns a list of candidates and assigns each a composite score.
2. Find the next step that reduces a list to N (look for sorting followed by slicing or taking).
3. Insert diversifyAfterScoring(scored, N, cfg) right before that reduction.
4. Keep routing and gating evaluation before diversification if that evaluation feeds baselines or telemetry comparisons, so you don't change what gets measured.

That's the part that mattered for me: diversification isn't a new "reward head." It's a selection policy.

Illustrative numeric example (not measured)

These numbers aren't empirical. They're here to show the shape:

• Candidate A: baseScore = 0.92
• Candidate B: baseScore = 0.91
• Candidate C: baseScore = 0.89

If B is essentially a paraphrase of A, while C represents a different direction, then a redundancy-aware penalty should push B below C. Keep the best idea as pick #1, then spend picks #2 through #N buying exploration instead of paraphrases.

Diagnostics: measuring collapse

It would be easy to close this out with a telemetry schema, table definitions, SQL queries, and automatic tuning. Naming internal tables and columns would be a project fingerprint.

So here's the non-identifying version:

• The codebase shows a pattern of structured scorecards being computed and attached to scene updates, including a count of candidates generated.
• The codebase structure also suggests there's already an endpoint for computing embeddings, plus endpoints for generating candidate sets.

A practical diagnostic, consistent with that pattern: for each generation request that produces K candidates, log a small summary artifact alongside whatever per-scene scorecard you already store:

• request identifier
• candidate count (the scorecard concept already tracks this)
• a per-request statistic that captures redundancy, say a histogram or quantiles of pairwise redundancy
• any routing context you already keep in the scorecard (routerConfidence lives there already; if you store other routing metadata elsewhere, match that style)

Your "mode collapse detector" then becomes something small: trend that redundancy statistic over time, sliced by whatever routing categories you already persist.

Closing

Composite scoring answers one question. Which single candidate is best? Diversification answers a different one: which set gives me five meaningfully different options?

Separate those two questions, put diversification after scoring and before selection, and the pipeline stops paying five times over for the same thought.