Defensive Multi‑Agent Scoring: How I Made LLM Reviews Clamp, Stream, and Fail Loudly

A few days ago my review stage did the most dangerous thing a multi-agent system can do. It looked like it worked.

The UI showed progress. The pipeline marched forward. And one of the agents had effectively returned "nothing," which meant my final decision was being computed from a lie: an average that pretended a missing opinion existed.

Somewhere in there, "LLM evals" stops being the useful frame and defensive systems engineering takes over.

This post is about one very specific feature in my codebase, the multi-agent scoring utilities used by my blog review stage. (It runs on a real project site. I'm not naming the domain or the repo layout.) The pattern is simple to say and annoyingly subtle to get right:

parse whatever the model emits (even when it's truncated or malformed),
normalize it into a strict shape,
clamp numeric fields so they can't poison the aggregate,
treat "no response" as a first-class failure state,
stream intermediate output for debugging and replay,
and only then do weighted scoring and gating.

I'll stick to three things: the review shape, a broadcast hook imported for agent streaming, and a centralized quality gate object in the shared utilities. Where the implementation stays out of this post, the design rule is still the point.

Scoring is an input-validation problem

A weighted average is the easy part. The real engineering is deciding what to do when one judge:

emits JSON that doesn't parse,
returns a partial object missing fields,
produces out-of-range numbers,
or fails completely (timeout / refusal / empty stream).

Leave those states implicit and you get what I call polite failure: the system continues, produces a number, and hands you false confidence.

In my pipeline, I standardized on a single sentinel for a completely failed agent:

score = 0 and confidence = 0

That pair means "this agent did not meaningfully participate." Everything else is treated as a real opinion. Possibly low quality, but at least present.

The contract: one strict review shape

At the center is a shared helper module used by the review stage. The codebase defines the review payload shape like this:

export interface AgentReview {
  agent: string;
  score: number;
  confidence: number;
  issues: string[];
  suggestions: string[];
}

That interface looks unremarkable until you treat it as a hard boundary between:

a streaming LLM call,
whatever "JSON-ish" bytes the model dribbles out,
normalization,
scoring,
and the downstream quality gate.

Let any of those boundaries go fuzzy and you get phantom scores.

Streaming output is an observability primitive

The review helper imports a broadcast function for agent streaming: a broadcastAgentStream symbol sits in its shared-utilities import list. The key point I care about is architectural, not proprietary:

a streaming call can fail after producing a partial response,
and those partial tokens are often the only evidence you'll ever get.

Without streaming, all you see is "failed." With streaming plus a broadcast hook, you can capture the partial output as it arrives, which makes the failure debuggable. Its job is making failures legible. Looking alive in the UI is a side effect.

I'm leaving the provider-specific request object out of this post. It carried a response_format: { type: 'json_object' } field, and that parameter doesn't travel between clients. I am not reproducing a complete, runnable streaming call here either.

The failure rule: a broken judge must stay visible in the aggregate

This is where I made a concrete wrong turn.

I had two competing impulses:

"If anything fails, fail the run."
"If something fails transiently, keep going, but don't lie."

My first attempt biased toward (1): treat a sentinel review as catastrophic and collapse the entire run's score. Strict. Also brittle.

The design I actually wanted is (2). The sentinel exists so the pipeline can stay deterministic while still reflecting reality.

Concretely:

A completely failed agent should still appear in the raw list of AgentReviews as the sentinel (score=0, confidence=0, empty arrays), so degradation is explicit.
Aggregation should not treat that sentinel as a real "vote."

I'm not including the full computeWeightedScore(...) implementation in this post. A stub would be worse than publishing nothing: it would look authoritative while being unverifiable.

The intended behavior follows from the design above:

filter out reviews that match the sentinel (score=0 and confidence=0) before computing an aggregate,
compute the aggregate from the remaining reviews,
if no non-sentinel reviews remain, return a clearly failed aggregate (a zero score, say, or a separate "no valid reviews" state, whatever your pipeline expects),
keep the sentinel reviews in the recorded run output so the system is honest about partial failure.

Weights and gating policy: keep it centralized, and don't assume weights sum to 1

Weights and thresholds live in a centralized QUALITY_GATE object with a WEIGHTS field and several threshold values loaded via a helper like envInt(...).

The exact environment variable key names and the exact numeric weights stay out of this post. Those details increase the inference risk of fingerprinting a project, and the numbers had an unresolved correctness problem underneath them too: the weights summed to 1.1, with nothing to say whether the aggregation normalizes them.

The durable lesson doesn't depend on the exact numbers:

Put weights in one shared place.
Treat the weights as policy, not incidental constants.
In aggregation, either (a) ensure weights are defined to sum to 1, or (b) normalize by the sum of weights actually used.

Skip that and "weight tuning" becomes a hidden scale factor that shifts your output without ever announcing itself.

A concrete sequence: one agent fails, the system stays honest

Think of the scoring run like a panel of judges where one judge sometimes doesn't show up and occasionally hands in a napkin with half a sentence. Making sure the scoreboard reflects what really happened is a different job from computing an average.

The flow is the important part:

A failure timeline that used to be messy, now made explicit:

One agent starts streaming, emits partial JSON, then stops.
Parsing fails to produce a trustworthy object.
Normalization yields the sentinel review shape, so downstream code sees a valid AgentReview object carrying an explicit failure marker (score=0, confidence=0).
Because streaming output was broadcast, you have evidence of the partial emission rather than a black-box "no output."
Aggregation excludes sentinel reviews from the weighted calculation.
The run output still contains the sentinel review, so the system is visibly degraded.

That's the whole trick. Continue without lying.

What went wrong first (the real mistake)

My initial approach treated "one agent failed" as equivalent to "the run is invalid," and collapsed the aggregate to a meaningless value.

That conflates two different states:

Partial degradation: one agent flakes out, others produce usable reviews.
Systemic failure: you can't trust any of the reviews.

Streaming systems fail partially in the real world. The point of a sentinel review is to represent that partial failure without smearing it into your math.

The defensive invariants I enforce now

I keep coming back to a small set of invariants:

Treat model output as unknown until proven otherwise.
Clamp numeric fields into a known range (my review schema uses a 0–100 scale).
Normalize "list-like" fields into string[] and default missing values to empty arrays.
Represent total agent failure explicitly (my sentinel is score=0, confidence=0).
Capture intermediate output so partial failures are diagnosable.
Centralize weighting and gate policy, and ensure the aggregation math doesn't depend on accidental weight sums.

Multi-agent scoring isn't a number you compute. It's a contract you enforce. It stays trustworthy only as long as every failure mode becomes visible before it becomes persuasive.