Skip to content
← Back to Blog
typescriptnextjsocrvideo-processingscene-compilerscoring

Text in a Frame Is Contamination, Not Decoration

Daniel Anthony Romitelli Jr. · March 29, 2026

Hook

A clean-looking frame failed the chain, and none of the usual suspects explained it. Not blur. Not framing. Not a weak prompt. The frame carried text, and that text was the thing poisoning the next step. Once I stopped treating it as a visual annoyance and started treating it as contamination, the whole design got easier to reason about.

That's why the text detector module exists. "Is there text here?" is a question it answers, but on its own that answer isn't enough. What I need answered is how dirty the frame is, where the dirt sits, and how much the pipeline should care. A subtitle strip across the bottom, a tiny watermark in the corner, and text baked into the scene itself are three different failure modes. They should not get the same response.

The module sits in front of the handoff into the next scene, an airlock between what the system sees and what it passes on. If a frame carries semantic residue, I want to know before it gets promoted into reference material for the next generation step.

The shape here is the whole point. The detector is isolated from the rest of the visual scoring system. It doesn't try to be composition analysis, and it doesn't pretend OCR is a style model. It takes region boxes, classifies them by where they sit, computes a weighted contamination score, and hands the rest of the system a number that can influence routing and recovery.

Why text earned its own score

The naive version: detect text, return a boolean, move on. I tried that early. Too blunt to be useful. A subtitle bar across the lower third and a watermark in the corner are both text, and they don't deserve the same treatment. The first is usually a hard sign the frame carries unwanted overlay material. The second is smaller, still meaningful, because it can carry branding or UI residue that shouldn't bleed forward into the chain.

So the detector splits into zones:

  • subtitle
  • watermark
  • scene-content

That's the idea the module is built around. Text is a location-dependent failure mode. The same glyphs mean one thing in the bottom 20% of the frame, something else in a corner, and something else again when they show up inside the actual scene.

It's also why the detector returns a structured result rather than a binary alarm. The pipeline needs more than a yes or no. It needs the region count, the breakdown by zone, the classified boxes, and the final normalized score that feeds router feedback.

The actual contract of the detector

The implementation is a small set of types and helpers. I kept the surface area deliberately narrow so the result stays legible: normalize the OCR output, classify each region, score what comes out.

export interface TextRegion {
  x: number;
  y: number;
  w: number;
  h: number;
  label: string;
}

export type TextZone = 'subtitle' | 'watermark' | 'scene-content';

export interface ClassifiedTextRegion extends TextRegion {
  zone: TextZone;
}

export interface TextDetectionResult {
  score: number;
  regionCount: number;
  subtitleCount: number;
  watermarkCount: number;
  sceneContentCount: number;
  regions: ClassifiedTextRegion[];
}

const SUBTITLE_ZONE_TOP = 0.80;
const WATERMARK_MARGIN = 0.15;
const WATERMARK_MAX_AREA_RATIO = 0.02;

const SUBTITLE_WEIGHT = 3.0;
const WATERMARK_WEIGHT = 1.5;
const SCENE_CONTENT_WEIGHT = 1.0;
const SATURATION_COVERAGE = 0.05;

function clamp01(value: number): number {
  return Math.max(0, Math.min(1, value));
}

function boxArea(region: TextRegion): number {
  return Math.max(0, region.w) * Math.max(0, region.h);
}

function zoneWeight(zone: TextZone): number {
  switch (zone) {
    case 'subtitle':
      return SUBTITLE_WEIGHT;
    case 'watermark':
      return WATERMARK_WEIGHT;
    case 'scene-content':
    default:
      return SCENE_CONTENT_WEIGHT;
  }
}

export function normalizeOcrBoxes(boxes: unknown[], labels?: string[]): TextRegion[] {
  if (boxes.length === 0) return [];

  const first = boxes[0];

  if (typeof first === 'object' && first !== null && 'x' in first && 'w' in first) {
    return boxes.map((box) => {
      const b = box as { x: number; y: number; w: number; h: number; label?: string };
      return {
        x: b.x,
        y: b.y,
        w: b.w,
        h: b.h,
        label: b.label ?? '',
      };
    });
  }

  if (Array.isArray(first) && first.length >= 8) {
    return boxes.map((quad, index) => {
      const q = quad as number[];
      const xs = [q[0], q[2], q[4], q[6]];
      const ys = [q[1], q[3], q[5], q[7]];
      const minX = Math.min(...xs);
      const minY = Math.min(...ys);
      const maxX = Math.max(...xs);
      const maxY = Math.max(...ys);

      return {
        x: minX,
        y: minY,
        w: maxX - minX,
        h: maxY - minY,
        label: labels?.[index] ?? '',
      };
    });
  }

  return [];
}

export function isSubtitleZone(region: TextRegion, frameHeight: number): boolean {
  const centerY = (region.y + region.h / 2) / frameHeight;
  return centerY >= SUBTITLE_ZONE_TOP;
}

export function isWatermarkZone(region: TextRegion, frameWidth: number, frameHeight: number): boolean {
  const centerX = (region.x + region.w / 2) / frameWidth;
  const centerY = (region.y + region.h / 2) / frameHeight;
  const areaRatio = boxArea(region) / (frameWidth * frameHeight);

  if (areaRatio > WATERMARK_MAX_AREA_RATIO) {
    return false;
  }

  const inCorner =
    (centerX <= WATERMARK_MARGIN || centerX >= 1 - WATERMARK_MARGIN) &&
    (centerY <= WATERMARK_MARGIN || centerY >= 1 - WATERMARK_MARGIN);

  return inCorner;
}

export function classifyTextRegions(
  regions: TextRegion[],
  frameWidth: number,
  frameHeight: number,
): ClassifiedTextRegion[] {
  return regions.map((region) => {
    let zone: TextZone;

    if (isWatermarkZone(region, frameWidth, frameHeight)) {
      zone = 'watermark';
    } else if (isSubtitleZone(region, frameHeight)) {
      zone = 'subtitle';
    } else {
      zone = 'scene-content';
    }

    return {
      ...region,
      zone,
    };
  });
}

export function computeTextPresenceScore(
  regions: TextRegion[],
  frameWidth: number,
  frameHeight: number,
): TextDetectionResult {
  if (regions.length === 0) {
    return {
      score: 0,
      regionCount: 0,
      subtitleCount: 0,
      watermarkCount: 0,
      sceneContentCount: 0,
      regions: [],
    };
  }

  const classified = classifyTextRegions(regions, frameWidth, frameHeight);
  const frameArea = frameWidth * frameHeight;

  let weightedCoverage = 0;
  let subtitleCount = 0;
  let watermarkCount = 0;
  let sceneContentCount = 0;

  for (const region of classified) {
    const areaRatio = boxArea(region) / frameArea;
    weightedCoverage += areaRatio * zoneWeight(region.zone);

    switch (region.zone) {
      case 'subtitle':
        subtitleCount += 1;
        break;
      case 'watermark':
        watermarkCount += 1;
        break;
      case 'scene-content':
        sceneContentCount += 1;
        break;
    }
  }

  const score = clamp01(weightedCoverage / SATURATION_COVERAGE);

  return {
    score,
    regionCount: classified.length,
    subtitleCount,
    watermarkCount,
    sceneContentCount,
    regions: classified,
  };
}

Zone classification alone wouldn't be enough. The detector is classification plus weighted coverage plus a normalized score, and the score is the part the rest of the pipeline can actually use. Clean frames come out low, subtitle contamination pushes hard, and a corner watermark contributes less than subtitles do while still moving the number. Scene-content text gets tracked too. Even when it's legitimate on-frame text, it shouldn't disappear into a blind spot.

Normalization is where the format differences die

The biggest implementation trap in OCR pipelines is format variance. Different detectors expose region data in different shapes, and if that variation leaks downstream, every consumer of the data turns brittle.

normalizeOcrBoxes is the first guardrail against it. I built it to handle two formats:

  1. already-normalized { x, y, w, h, label } objects
  2. raw Florence-2 quad boxes with eight floats: [x1, y1, x2, y2, x3, y3, x4, y4]

They aren't interchangeable, and pretending otherwise is how the geometry goes wrong. The normalized path is easy, since the region already carries width and height. The Florence-2 path needs a min/max pass across all four corners to build an axis-aligned box.

That conversion is where mistakes hide. Grab a partial coordinate pair, or derive height from the wrong point pair, and you get a box that doesn't actually cover the text. From there, both zone classification and area scoring go noisy.

Done right, normalization hands everything downstream a stable contract. The router shouldn't have to care whether the OCR came from fal.ai's normalized region output or a raw Florence-2 quad. The detector absorbs that difference once. After that, the rest of the system sees one shape.

The zone rules are simple on purpose, but not loose

The zone logic isn't fancy, and I like it that way. Simplicity makes it easier to keep the scoring trustworthy. Simple doesn't mean vague, though. The thresholds are specific:

  • subtitle: text center in the bottom 20% of the frame
  • watermark: text center in a corner with a 15% margin, and the box area must be under 2% of frame area
  • scene-content: everything else

That area cap on watermarks does real work. Without it, a large corner overlay would get the softer watermark treatment purely for sitting near an edge. The corner check and the size guard work together.

Same care in the subtitle rule, which uses the region center rather than the top edge or the bounding box intersection. Text that grazes the lower part of the frame without behaving like a subtitle strip shouldn't set it off.

And the order matters. Watermark first, then subtitle, then scene-content. Otherwise a corner overlay that happens to live low in the frame would get swept into the subtitle bucket, and the zones stop meaning what they say.

The score is where classification becomes policy

Subtitle text carries the heaviest weight, watermark text next, scene-content least. That mirrors the actual failure hierarchy. Subtitle-like text is almost always unwanted overlay. Watermarks are smaller but still harmful, since they usually indicate branding or generated residue. Scene-content text isn't necessarily wrong, and it still counts, because it's part of the image's semantic load.

The weighting runs on area coverage rather than region count. That detail is doing a lot of work. Two tiny boxes shouldn't land the same way as a full-width subtitle strip, even inside the same zone. Multiply area ratio by zone weight, normalize the aggregate into the [0, 1] range, and the router gets back something that behaves like a real measure of contamination instead of a checklist.

The saturation threshold is deliberate too. Once weighted coverage reaches 5% of frame area, the score clamps to 1.0, which tells the rest of the pipeline early and without ambiguity that the frame is dirty enough to treat as a hard call rather than a weak hint.

Here's a concrete example. Three regions in a 1920 by 1080 frame:

  • one subtitle region covering 1% of the frame
  • one watermark region covering 0.5% of the frame
  • one scene-content region covering 0.5% of the frame

The weighted coverage:

  • subtitle: 0.01 × 3.0 = 0.03
  • watermark: 0.005 × 1.5 = 0.0075
  • scene-content: 0.005 × 1.0 = 0.005

Total weighted coverage = 0.0425. Divide by 0.05 and the score lands at 0.85. That's the right shape. The frame is well past slightly noisy, contaminated enough that I want the pipeline to think twice before reusing it as a reference.

Why scene-content still matters

Scene-content is the least suspicious category, which makes it tempting to ignore. That would be a mistake. Sometimes text really does belong in the scene: signage, labels on products, interface elements in a diegetic screen, text that's intentionally visible in the shot. I track it anyway, because its presence tells me something about the structure of the frame, even when it doesn't immediately call for the same response as a subtitle strip.

That distinction helps the router stay nuanced. A frame with only scene-content text can score lower than one with overlay-like text, and the evidence is still there. Useful for gating, useful for analysis.

What I'm avoiding is the collapse into binary failure. The detector holds enough structure for the pipeline to behave differently depending on which kind of text contaminated the frame.

How this feeds the rest of the system

The detector isn't a dead-end report. It feeds back.

The surrounding generation path already works from several kinds of evidence at once. Candidate selection isn't based on a single metric. The reward mixer evaluates multiple measures, the progressive pipeline retries weak generations, and the feedback layer turns outcomes into calibration samples. The text detector plugs into that system as one more source of evidence.

Which matters, because text contamination is one of the easiest ways for a scene chain to lie to itself. Let a contaminated frame pass forward silently and the next step may treat the contamination as visual truth. That's the failure I wanted stopped at the boundary.

The detector hands the router three things at once:

  • a normalized score
  • a breakdown of what kind of text was found
  • the classified regions themselves for inspection and debugging

That combination lets automation make a decision while still leaving me a paper trail when I go back and inspect a bad run.

Debugging the failure modes

The edge cases are what made this worth building carefully.

A box in the bottom part of the frame isn't automatically a subtitle. Its center has to cross the bottom-20% threshold. That prevents accidental overreach.

A tiny logo in the corner isn't a watermark either, not unless it sits genuinely inside the corner window and under the area cap. Large corner overlays would otherwise slip through wearing the wrong label.

And a wide text strip sitting slightly above the subtitle boundary isn't necessarily a subtitle, even if a human might casually call it one. I'd rather the detector stay consistent with the spatial rule than slide into subjective labeling.

Then there's the OCR input shape changing underneath it. Normalization is what keeps the downstream score from collapsing when that happens, the kind of work that tends to be invisible while it holds and catastrophic when it doesn't. Once the detector owns that translation, nothing else has to care where the boxes came from.

Why the module stays pure

The text detector is pure on purpose. No API calls. It doesn't reach into storage, and it doesn't make routing decisions directly. It transforms OCR regions into a structured judgment, and that's the whole job.

Two things come out of that separation.

It's easy to test. I can throw synthetic regions at it and assert exactly how they land in each zone, what the score should be, how many regions belong in each bucket.

And the contamination rule lives in one place. If I change frame extraction later, or swap OCR providers, or adjust the calibration path, the detector still has one job: tell me whether the frame contains text, where it sits, and how badly it should count against the scene.

That's the separation that makes the pipeline maintainable. I can tune policy without rewriting geometry.

Closing

Text in a frame is contamination, not decoration, and treating it that way closed off a class of failures that used to slip through generation unnoticed. The detector doesn't guess. It measures coverage, classifies by zone, weights by severity, and returns a number the router can act on without asking permission. Every frame either earns its way forward or gets cut. The pipeline stopped being fragile the moment I stopped being polite about what contamination looks like.