When the Edge Lives in Colour

A red car can vanish at the exact edge a person sees clearly.

The paint is red. The sky behind it is blue-grey. On screen, the outline looks obvious. Then the cleanup step turns both sides into one brightness number, and the contour that mattered becomes a weak grey ramp.

That is the visible failure this part of CarSegNet v2 fixes. NeuralSegJet (NSJ), the 93,090-parameter alpha refiner I built for vehicle matting, repairs only the uncertain strip around a mask. Its final colour-guided filter decides whether that repair lands on the real silhouette or bleeds across it.

The README files this under bugs. It was design, not a crash: nothing threw, nothing logged. The mattes were soft in the wrong places, and every downstream correction — harmonisation, defocus match, contact shadow — inherited that error and made it look deliberate.

The mask is a first guess

A binary semantic mask answers one rough question: is this pixel vehicle or background? Dealership composites need a fractional answer. A windshield edge, glossy fender, roof antenna, wheel spoke, or contact line rarely lands on whole pixel coordinates.

CarSegNet v2 uses Segment Anything Model 3 (SAM 3) for still-image concept masks, NSJ for alpha refinement, then the compositor places the vehicle onto a generated plate. Video uses Segment Anything Model 3.1 Object Multiplex before the same per-frame refinement path.

There are three useful mental models:

A guided filter is the last step of that refinement. It fits a local linear model — alpha as a function of the image inside a small window — and re-evaluates it per pixel, so the output snaps onto edges present in the guide instead of blurring across them. Everything depends on what the guide is. In PAPER.md the hard case is red bodywork around (200,30,40), Y≈88, against blue-grey sky around (120,140,180), Y≈137 in places and less elsewhere. Collapse the image to that one brightness number and the model has almost no slope left to fit.

How much signal is actually there

I measured it on the synthetic scene the self-test already builds: a rounded body with a thin roof antenna, painted (200,30,40), over a background that ramps from (120,140,180) at one corner toward (40,70,180) at the other.

Take a 7×7 window at every pixel on the true contour and record the range — max minus min — that each representation sees. For luminance that is one scalar. For RGB it is three, and the comparison has to be stated carefully: the Euclidean norm of three channel ranges is up to √3 larger than any one of them by construction, before a single unit of chromatic separation enters. The norm is not the number to quote.

Against the per-axis figure the colour guide carries 4.7× the separation, not the 8.2× the raw norm suggests. Against the single strongest channel it is 5.7×. Either way the grayscale guide works with a fraction of what is present, and the mean hides the worst of it: 29% of contour pixels have a local luma range under 20, first percentile 12.7.

The region means show why, and the arithmetic is the whole argument. Eroded foreground core against far background:

Every channel separates car from sky by 77 to 141 units. Rec. 601 then weights those differences and adds them: 0.299(+119.7) = +35.79, 0.587(−76.9) = −45.17, 0.114(−140.5) = −16.02. The positive red term is cancelled by the negative green and blue terms, and 25.4 units survive out of a per-axis RMS of 115.5.

So luminance does not fail here because it is one number instead of three. It fails because the red difference has the opposite sign to the other two, and the weighted sum destroys them against each other. A guide can lose an edge that every one of its own channels can see.

The background's own luminance sweeps from 73.6 to 138.6 across the frame, and the car sits at 81.45 — inside that range. Whether the vehicle is darker or brighter than what is behind it depends on where you stand along the silhouette. On this scene the weakest contour pixels still hold a local luma range near 13, so the grayscale guide is degraded rather than blind. On a real lot the background is not a smooth ramp, and nothing keeps that number above zero.

Where the refiner is allowed to write

make_trimap in carsegnet/refine.py builds the unknown region from two terms and unions them:

hard = (p > 0.5).astype(np.uint8)
r = max(1, int(band_px))
k = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (2 * r + 1, 2 * r + 1))
band = cv2.dilate(hard, k) - cv2.erode(hard, k)
soft_band = ((p > soft_lo) & (p < soft_hi)).astype(np.uint8)
band = np.clip(band + soft_band, 0, 1).astype(np.uint8)

The first term is a morphological ring of radius band_px around the thresholded mask. The second is the soft iso-band: every pixel where the prior sits between 0.05 and 0.95. The ring covers a boundary that is displaced. The iso-band covers a boundary that is genuinely soft — glass, motion blur, a spinning wheel — which a fixed dilation radius misses when the transition is wider than the ring.

Everything outside that union is frozen at prior > 0.5. The refiner does not see the whole frame either: band_bbox crops to the band's bounding box with 16 px of padding, aligned to 8 so the /4 decoder needs no odd padding.

Both band terms are functions of the prior alone. That has a consequence worth stating on its own:

Any refinement stage whose trust region is derived from the prior it is correcting is structurally blind to that prior's confident errors.

A filled wheel opening is confidently foreground with no nearby iso-contour. A dropped antenna is confidently background, same story. Neither produces a band, so the refiner is never asked about either. Training on corrupted priors does not rescue it, because at inference the band is rebuilt from the prior and excludes exactly the region the corruption was teaching the network to repair. The repo states this inside _disagreement_band, a function that exists only because of it.

That third term consults the photograph, marking pixels where the image has strong gradient and the prior is locally flat. It is off by default (use_disagreement_band: false). Expanding the trust region at inference with a guide training never used is a distribution change, and it produced a real regression before it was disabled.

The one-channel version is too small

The old guided-filter form in carsegnet/nsj.py is compact. It computes local means, variance, covariance, then fits a linear relationship between a one-channel image and the source alpha.

def guided_filter(guide: torch.Tensor, src: torch.Tensor, r: int = 8,
                  eps: float = 1e-4) -> torch.Tensor:
    """He et al. guided filter, grayscale guide. Both tensors are [B,1,H,W]."""
    mean_i = box_filter(guide, r)
    mean_p = box_filter(src, r)
    corr_i = box_filter(guide * guide, r)
    corr_ip = box_filter(guide * src, r)
    var_i = (corr_i - mean_i * mean_i).clamp_min(0.0)
    cov_ip = corr_ip - mean_i * mean_p
    a = cov_ip / (var_i + eps)
    b = mean_p - a * mean_i
    return box_filter(a, r) * guide + box_filter(b, r)

The limitation is in the shape: [B,1,H,W]. Each pixel enters the solve as one scalar. If the contour is carried by hue, the evidence is gone before the math starts.

That function is still in the tree, and it has exactly one caller in the whole repository: the self-test that checks a guided filter snaps a 4 px-displaced edge back onto a step in the guide. No production path calls it. It stays as the reference implementation the colour version has to agree with on a case where luminance is sufficient.

The colour solve keeps the missing axes

guided_filter_color changes the image input to [B,3,H,W] and solves a local 3×3 covariance system per pixel. In the paper notation:

a = (Σ_I + εU)^-1 cov(I,p)
b = p̄ - a^T Ī

Six covariance terms — red-red, red-green, red-blue, green-green, green-blue, blue-blue — epsilon on the diagonal, six cofactors, one determinant, a symmetric cofactor inverse, three linear coefficients, then the averaged coefficients applied back to the RGB image and cast to the source dtype.

The cost is countable. The grayscale form makes six box-filter passes over one channel each. The colour form makes twelve calls covering seventeen channel-passes, plus the cofactor arithmetic. Timed on CPU at 256×384 I measured 11.88 ms against 17.60 ms per call — about 1.5× wall clock, well under the 2.8× ratio of filtering work, because the extra covariance terms are elementwise and cheap next to the box filters.

The precision constraint is the part that bit. The config ships half: true, and _run_nsj acts on it: x = x.half() before the forward pass. Without a guard the covariance solve runs in 16-bit. A flat or near-flat colour patch has covariance eigenvalues close to eps, so its determinant lands near eps**3 — 1e-12 at the training default — and underflows to zero in fp16. The clamp meant to catch that, torch.full_like(det, 1e-12), is also zero in fp16. The reciprocal produces inf, 0 * inf produces NaN, and the NaN propagates through alpha into the loss.

The fix is one context manager. The entire covariance solve runs under torch.autocast(..., enabled=False) on float32 copies, and the result casts back to the source dtype at the end. Tensor.float() and the final cast are both differentiable, so gradients still flow through the guide and the source. A self-test named colour guided filter stays finite and differentiable in fp16/AMP holds it in place, and it builds the exact failure: a constant guide, singular covariance everywhere, checked under AMP autocast and again under an explicitly halved model.

That is the price of the better representation. A scalar pass is simpler and numerically easier. The RGB solve sees the chromatic contour, but it needs more arithmetic and stricter precision handling.

The fallback is two passes, not one

weights: null is the shipped default, and it is not a degraded mode bolted on for demos — it is the NSJ head with the learned part removed. AlphaRefiner._run_fallback runs the colour guided filter twice:

r_wide = max(12, int(self.band_px * 2.5))
r_fine = max(3, int(self.band_px * 0.6))
a = guided_filter_color(g, p, r_wide, 1e-3).clamp(0, 1)
a = guided_filter_color(g, a, r_fine, 1e-4).clamp(0, 1)

At band_px: 12 that is radius 30 with eps 1e-3, then radius 7 with eps 1e-4. The wide pass has enough support to pull alpha onto the true silhouette when the prior is displaced by several pixels. The narrow pass recovers the antenna and the wheel-arch detail the wide pass smears.

Two passes still leave a level problem. A prior dilated by 7 px yields an output whose foreground plateau and background floor are both wrong, and a fixed 0..1 stretch does not correct it. So the fallback measures its own references — the median of the filtered alpha inside an eroded core, and the median far outside a dilated hull, both at radius band_px * 2 — normalises between them, then applies a smoothstep to sharpen the transition. Skip that step and a prior which is 7 px too fat stays 7 px too fat.

Trust is enforced twice

NSJ blends its raw head output with the filtered version through a learned scalar gate:

refined = guided_filter_color(x[:, 0:3], raw, self.guided_radius,
                              max(self.guided_eps, 1e-4))
gate = torch.sigmoid(self.gf_gate)
alpha = gate * refined + (1.0 - gate) * raw

prior, band = x[:, 3:4], x[:, 4:5]
alpha = band * alpha + (1.0 - band) * (prior > 0.5).to(alpha.dtype)

gf_gate initialises at 1.5, so training starts with the guided filter carrying 0.818 of the output and the network free to move it. The epsilon passed in is floored at 1e-4 regardless of config, because a smaller value makes the 3×3 solve less stable for no measurable edge gain.

The band constraint on the last line runs inside the network. AlphaRefiner.refine applies it again on the numpy side after the crop is written back. That second pass is not decoration: max_side: 2048 means a large crop is downscaled before the forward pass and bilinearly upscaled after it, and bilinear interpolation smears predicted values a pixel or two past the band edge. The re-enforcement clips that leak, and it holds if someone swaps in a model implementation that ignores the constraint internally.

Temporal smoothing obeys the same rule. The EMA against the flow-warped previous alpha applies only where band > 0, so confident interior pixels do not drift frame to frame. The default configuration reflects all of it:

refine:
  enabled: true
  weights: null              # null -> colour-guided-filter fallback (works today)
  band_px: 12                # widen if SAM masks are consistently fat/thin
  tile: 1024
  guided_radius: 8
  guided_eps: 0.0001
  use_disagreement_band: false # experimental; not part of NSJ training bands
  temporal: true
  temporal_ema: 0.35         # raise to kill flicker, lower to keep motion crisp
  half: true
  compile: false
  max_side: 2048

Widening the writable strip makes the refiner more expressive. It also lets local texture steer alpha farther from the original mask. I prefer a narrow repair tool with a clear failure boundary to a matte stage that rewrites vehicle structure.

Luma versus colour, measured

PAPER.md lists this ablation as pending, and the full version — trained checkpoint, real vehicles, whole path held steady — still is. I ran the scoped version on the synthetic harness, changing nothing but the guide: same scene, same degraded prior (dilated 7 px, blurred), same band at band_px: 10, same wide-then-fine schedule, same level normalisation and smoothstep.

SAD and gradient error are frame totals divided by 1,000, so they compare within a table and not across frames of different size.

The grayscale guide is not useless here. It takes SAD from 4.82 to 3.51 and recovers a boundary IoU of 0.379 from a prior that scores zero. It leaves most of the available improvement on the table. Colour roughly doubles the boundary IoU and halves the gradient error.

Restricted to the antenna, the thinnest structure in the scene, mean absolute alpha error runs 0.813 for the prior, 0.255 for luma, 0.216 for colour. The gap narrows there, which is what I expect: a 2 px line against sky is a support problem before it is a contrast problem, and the wide pass is what recovers it either way.

This is one synthetic scene with one paint colour, constructed to be the hard case. It confirms the mechanism and the implementation. It is not the ablation the paper owes.

592 of 608 frames go to a human

The colour filter is a boundary tool, and the pipeline's unsolved problem is interior topology. Run the fallback across the 608-frame Carvana checkpoint-selection split and 592 frames fail the QA gate. Not on region overlap — that same fallback scores 0.9815 IoU on that split. They fail on enclosed background components, matched one to one against ground truth.

The fallback finds 28 real openings where raw SAM finds one, and invents 2,492 that are not there. A filter that snaps alpha to image structure snaps to reflections, decals and shadow edges as readily as to a wheel gap. That is the whole argument for the trust band: a strong edge tool with no semantic model of a vehicle needs a small writable region.

On real vehicle photographs the ranking changes again. Measured on 99 held-out frames with the same prior, only the weights differing:

The deterministic filter beat a 40-epoch trained checkpoint on boundary SAD and lost to it on gradient error. That checkpoint scored a 0.034 validation loss on Carvana's own studio cutouts, which is where its apparent quality came from. It has since been retired and replaced by one trained from random initialisation on a CC0 corpus. Why it was retired is a licensing story, and a separate post.

Still open

The synthetic result changes the representation and isolates the mechanism, on a scene I built to make the point. The version that would settle it runs the trained checkpoint on real dealer imagery, swapping only guided_filter_color for guided_filter inside NeuralSegJet.forward, with the band, the gate, the temporal channel and the depth channel unchanged. That is one of five ablations the paper still owes, alongside NSJ versus fallback versus dense CRF, and the temporal and depth channel drops.

Until then the claim is bounded. On a scene where the luminance step is 25 units and the per-axis colour step is 115, colour is worth roughly double the boundary IoU and half the gradient error. Luminance is not useless there, only partial — the table above is the size of the effect, and "required" is not what it says. How much colour contributes on a dealer lot in overcast light is not yet a number I have.

What transfers

Two things here generalise past vehicle matting.

Pick the representation before you tune the filter. The colour form costs 1.5× the wall clock and needs a precision guard the scalar form does not. It is still the right default, because no amount of radius and epsilon tuning recovers a difference the weighted sum already cancelled. Representation failures do not look like failures. They look like a stage that is working and needs more tuning.

The writable region is the safety mechanism, not the filter. A trust region derived from the prior it is correcting cannot see that prior's confident errors, so either supervise a second stage on what the band excludes or route those frames to a person — widening the band trades a known limitation for an unbounded one. A filter with no semantic model then snaps to whatever is sharp: decals, badge edges, reflections and the shadow line under a rocker panel all look like a wheel gap to it. On Carvana the fallback and the trained network write into the same band, and the fallback still invents 2,492 enclosed components against the trained network's 111.

If the signal lives in colour space, collapsing it to luminance turns a visible edge into missing data.