Skip to content
← Back to Blog
computer-visionvideoinferencetrackingdebugging

I got SAM3 video tracking wrong: the session wasn’t the problem, my reprojection was

Daniel Anthony Romitelli Jr. · February 28, 2026

I assumed the flicker was SAM3 being temperamental.

Masks snapped on and off across consecutive frames, and my first instinct was to reach for the usual bag of excuses. "It's a hard scene." "Video is noisy." "The model needs more context."

What actually broke was simpler, and more embarrassing. I was letting the model's outputs live in the wrong coordinate system, then trying to stabilize a tracker on top of that. Once I fixed session reuse and reprojection back into each frame's original_sizes, the churn stopped looking like AI chaos and started looking like normal engineering again.

Streaming masks only look unstable when you keep moving the floor under them

In this pipeline I'm doing per-frame segmentation in a video stream. Every frame carries two sizes that matter:

  • the size I feed into the model (whatever preprocessing produces)
  • the size I need to draw on screen (the frame's original dimensions)

Let those two slip apart, even slightly, and your tracker is trying to associate masks that are effectively being scaled or shifted between frames. Calling that "flicker" lets me off the hook. I was changing the ruler every 33ms.

The fix came out as a set of decisions that all push the same direction:

  • create an inference session once and reuse it across frames (warm path)
  • when I must restart, do it deliberately and recover state (warm→restart)
  • always reproject model outputs into original_sizes before I stream them
  • handle mixed-resolution frames explicitly (or you'll debug ghosts)

Below I walk through the session lifecycle, the streaming output assumptions (including why pixel_values[0] is acceptable in this particular stream), and the diagnostic endpoint I added, /segment/debug-model, so I could root-cause flicker instead of arguing with vibes.

That diagram is the mental model I wish I'd had earlier. A tracker can only be as stable as the coordinate system it's fed.

How I run the inference session across frames (warm, and warm→restart)

On the GPU server side I ended up treating the inference session like a long-lived object. Create it, feed it frames, and only restart when something actually invalidates the state.

I added a dedicated model introspection endpoint: debug: add /segment/debug-model introspection endpoint (commit a1389d6). That exists because I needed visibility into what the server thought it had loaded, and what configuration it was running under.

The pattern I landed on is "configured or no-op". Check prerequisites at module load, and if the session isn't in a known-good state, refuse to proceed rather than produce garbage outputs. Same defensive shape across the SAM3 inference session, the RGB-X pipeline, and every GPU server endpoint in this codebase. Validate state before acting. Never assume continuity.

Applying that mindset to streaming segmentation taught me something I hadn't expected, and it had nothing to do with error handling. My "session is fine" assumption kept turning out to be wrong because the inputs were sliding around on me, resolution and orientation, and not because the model had crashed.

Warm path vs warm→restart

I've hit "state mismatch" style problems elsewhere, over and over, when the shape of the data didn't match what downstream code assumed. One concrete example lives in the RGB-X pipeline: I had to fix output unwrapping because the pipeline returned a nested list.

From the rgbx endpoints module (commit 0ca5907):

# Pipeline returns nested list: result.images[0][0] is the PIL Image
if hasattr(result, "images") and len(result.images) > 0:
    img = result.images[0]
    # Unwrap nested list (RGB-X wraps each channel in an extra list)
    if isinstance(img, list) and len(img) > 0:
        img = img[0]

The non-obvious lesson for me: in streaming inference, "session stability" problems often show up first as the shape or format changing underneath me. I stopped trusting any output container until I'd asserted what it actually was.

Reprojecting outputs into original_sizes for streaming

The clearest example of this same class of bug lives in the RGB-X endpoints. Bounding boxes coming from a different resolution than the input image.

From the rgbx endpoints module (commit 43a8bb0):

# Scale bbox if it's from a different resolution than the input image
bbox_right = bx + bw
bbox_bottom = by + bh
if bbox_right > img_w or bbox_bottom > img_h:
    scale_x = img_w / max(bbox_right, 1)
    scale_y = img_h / max(bbox_bottom, 1)

That snippet is the same species of problem as SAM3 mask reprojection:

  • you get coordinates (or masks) in one space
  • you need them in another
  • if you skip this, everything downstream looks "unstable"

In my SAM3 stream the tracker sat downstream, so the tracker got blamed. The real issue was that I fed it masks that didn't line up frame to frame, because I wasn't consistently mapping back to each frame's original_sizes.

One of the most practical guardrails I adopted: every streaming frame output must carry both the model-space size and the original frame size, and I treat any mismatch as a first-class event. Log it, debug it, maybe restart the session.

How the tracker associates masks across frames (and why confidence thresholds mattered)

I had to tune SAM3 confidence thresholds in response to "0 segments" and label churn, and those changes are sitting there in the commit history:

  • fix: lower SAM3 internal confidence_threshold from 0.5 to 0.25 (commit c8086b8)
  • fix: lower SAM3 default confidence to 0.15, remove debug endpoint (commit f3bd706)
  • debug: add SAM3 segment debug logging to diagnose 0 segments (commit 95cf2fd)
  • debug: add /segment/debug-model introspection endpoint (commit a1389d6)

Those commits tell a story I recognize from operating this kind of stream. If your confidence gating is too aggressive, you don't get "cleaner masks." You get churn. Objects disappear, reappear, and your association logic has nothing stable to latch onto.

What went wrong first (concretely)

I started with SAM3's internal confidence_threshold at 0.5.

That was a mistake, and not a marginal one. It produced the failure mode captured in my own commit message: I was diagnosing 0 segments. Hence the debug logging (95cf2fd).

Lowering that internal threshold from 0.5 to 0.25 (commit c8086b8) was the first time the stream started behaving like a tracker problem instead of a blank-output problem. Then lowering the default confidence to 0.15 (commit f3bd706) reduced churn further.

I didn't enjoy admitting this one. The tracker can't associate what the model refuses to emit.

Diagnostic endpoints: making flicker debuggable

When I'm dealing with flicker, I want to know three things immediately:

  1. what model/config is loaded?
  2. what thresholds are active?
  3. what does the server think the input/output shapes are?

Hence /segment/debug-model (commit a1389d6). A simple GET that returns the loaded model name, active thresholds, and expected input/output shapes. Nothing clever. Just enough state exposure that when something flickers, I can ask the server what it thinks is happening instead of guessing.

A debug endpoint was the obvious thing to build. What I hadn't expected was the effect of having one: I stopped "tuning" blindly. I could correlate flicker with specific conditions: mixed-resolution inputs, orientation mismatches, or thresholds that were too strict.

Mixed-resolution frames: the silent killer

The RGB-X bbox scaling fix (43a8bb0) shows exactly how mixed-resolution data sneaks in. A bbox that made sense in one resolution becomes out-of-bounds in another.

On the iOS capture side I hit a close relative: orientation mismatches. In VideoTrackingManager.swift I explicitly rotate ARKit camera buffers, because they're always landscape-left.

The comment I left there (commit e68236a):

  • "ARKit camera buffers are always landscape-left orientation."
  • "Rotate to portrait when device is upright so masks from the GPU server align with the on-screen display."

That's the kind of bug that masquerades as "model flicker" when the truth is that I rotated one side and not the other.

Here's the one analogy I'll use, because it's how the debugging felt. Tracking across frames with inconsistent reprojection is like trying to draw on tracing paper while someone keeps swapping the paper size when you blink. Your hand is steady. The reference frame isn't.

Why pixel_values[0] can be acceptable in a batch size 1 stream

This is a per-frame stream, not an offline batch job. Every time I accidentally treated a nested structure as flat (RGB-X's result.images[0][0] case), I paid for it in runtime errors or misaligned outputs.

So the rule became: I only index [0] when I've asserted, via logging or debug endpoint output, that the stream is batch size 1 and the container shape is stable. That discipline came straight out of the RGB-X unwrapping bug (0ca5907).

Strategies for graceful recovery when a session restarts

The broader recovery pattern comes from the iOS to web pipeline, where I upload segmentation results "fire-and-forget" from the device using a detached task.

From ios-lidar-app/SidingAILiDAR/ContentView.swift (commit d770ee8):

// Upload segmentations to Supabase (fire-and-forget)
let segmentsToUpload = segments
Task.detached {
    await SupabaseManager.shared.uploadSegmentations(
        projectId: projectId,
        captureId: captureId,
        segments: segmentsToUpload
    )
}

Same philosophy on the GPU side. Don't block the live experience on perfect continuity. When a session restarts I want the stream to keep moving, and I want enough persisted context (segments plus measurements) that the rest of the system can stay coherent.

What bit me early was assuming restarts are rare. They aren't. Deploys, GPU hiccups, inputs whose resolution or orientation shifts under you. Plan for none of that and you end up with a tracker that behaves like it has amnesia.

The part I didn't expect: lowering confidence reduced churn more than any tracker tweak

I went into this thinking the tracker was the hero.

The commit history says otherwise. First I had to make SAM3 emit segments reliably (internal threshold 0.5 → 0.25), then make the defaults more permissive (0.15) to reduce label churn. Only after that did the rest of the pipeline start behaving predictably: reprojection into original_sizes, session reuse, association.

Once I stopped moving the coordinate system under the masks, flicker stopped being a mysterious model trait and went back to being what it always was. A bug I could point to, log, and kill.