A depth pipeline should read a room the way a carpenter reads a level, not the way a photographer admires a picture. It thresholds, groups, checks for discontinuities, and validates whether the surfaces that come out can be trusted. That's the whole point of what I built: a segmentation path that still has something useful to say when the room is nearly dark and the RGB frame is useless.
The failure that started it was a capture where the image looked worthless and the depth map still had structure. Segmentation stopped being a color problem right there. It became a geometry problem, because if the scene is dark enough, texture is the wrong witness, and the depth field is the one telling the truth.
The shape-first path
The route I care about starts in the web app, but the real work happens on the GPU server. The browser sends a depth payload to the API route. That route forwards the request to the depth segmentation service, and from there the server turns a depth map into labeled regions by hunting for geometric structure instead of visual texture.
That's the whole idea in miniature. I'm not asking the model to "understand" a wall the way a vision model reads paint or grain. I'm asking it to find contiguous surfaces, split them where the depth jumps, and stay usable when the RGB frame is nearly empty.
The API route is deliberately thin. Its job is to move the request into the GPU service and hand the result back to the app, without turning the web tier into an image-processing graveyard.
import { NextRequest, NextResponse } from 'next/server';
const RUNPOD_ENDPOINT_URL = process.env.RUNPOD_ENDPOINT_URL;
export async function POST(request: NextRequest) {
try {
const body = await request.json();
const gpuServerUrl = process.env.SAM3_SERVER_URL || 'http://localhost:8000';
const response = await fetch(`${gpuServerUrl}/segment/depth`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(body),
});
if (!response.ok) {
const error = await response.text();
return NextResponse.json(
{ error: `GPU server error: ${error}` },
{ status: response.status }
);
}
const data = await response.json();
return NextResponse.json(data);
} catch (error) {
return NextResponse.json(
{ error: error instanceof Error ? error.message : 'Depth segmentation failed' },
{ status: 500 }
);
}
}
What I like about this route is how little personality it has. It doesn't interpret the scene, and it doesn't pretend to own the algorithm. It forwards the request, preserves the server's response, and keeps the failure mode obvious when the backend complains.
Why this isn't ordinary segmentation
Ordinary image segmentation gets to lean on texture, edges, contrast, and every other visual cue that makes a photo interesting. This path doesn't. It was built for total darkness scenarios, and the docstring says so outright: it uses LiDAR depth maps to detect walls, windows, doors, and trim via RANSAC plane fitting and connected component analysis, with no RGB required.
The naive move would be to treat every boundary in the image as a visual boundary. In a depth map, that instinct is backwards. A glossy surface can look noisy in RGB and still be flat in depth. A dark room can be visually unhelpful and geometrically rich. So I built the pipeline around shape: threshold the depth values, group connected regions, then test whether those regions behave like planes or like fragments of planes.
The geometric tests that do the real work
Here's where the pipeline stops being plumbing and gets interesting. The codebase gives me a vocabulary for geometric tests: depth analysis, geometry detection, multi-reference validation, and auto-scale correction.
The depth analysis interface carries a perpendicularity check, a tilt angle, a perspective correction factor, average depth, depth variance, and a gradient direction. That tells me the pipeline does more than carve masks. It asks whether a surface behaves like a stable reference plane. A flat region with low variance is one thing. A tilted, gradient-heavy region is something else entirely, and treating the two alike is how you end up with a confident wrong answer.
The geometry detection layer goes a step further and classifies surfaces as flat, angled, or multi-plane. It tracks peak counts, detected peaks, and a confidence factor. For adjacent planar regions, that's the right shape of heuristic: if the histogram of depth values suggests multiple peaks, pretending the whole surface is one plane is a lie. Better to split it, warn about it, or lower the confidence.
export interface GeometryAnalysis {
/** Whether surface appears flat (single depth plane) */
isFlatSurface: boolean;
/** Number of detected depth peaks */
peakCount: number;
/** Complexity classification */
complexity: 'flat' | 'angled' | 'multi-plane';
/** Detected peak depths (normalized 0-1) */
peaks: Peak[];
/** Warning message for user */
warning?: string;
/** Confidence factor for calibration (0-1) */
confidenceFactor: number;
}
I like this interface because it refuses to collapse geometry into a yes or no. A surface can be flat, angled, or multi-plane, and the rest of the pipeline needs that nuance if it's going to keep users out of trouble.
The confidence factor is the non-obvious piece. It bridges geometry and behavior. Once a region starts looking like a bay window or a composite surface, I don't label it and move on. I lower trust, surface a warning, and let the downstream calibration logic react.
The multi-reference validator runs a rule that's simpler than it sounds and matters more than it looks. When both a door and a window are detected, it calculates scale from each and compares them. The expected behavior is spelled out: one reference, use it directly; both references, compare them and warn if the ratio falls outside 0.85-1.15; prefer the door as the primary reference, because it's larger and more reliable.
That's the kind of heuristic I trust in production. No magic in it. Just a guardrail around geometry that stops the system from lying confidently when the scene is awkward.
Adjacent planes, where the geometry gets annoying
A wall next to trim. A door next to a window. A bay window with several surfaces folded into it. Any of those can pass for a single shape until depth exposes the seams, which is why the system carries both depth-variance analysis and peak-based geometry detection. A single plane shouldn't produce multiple strong depth peaks. When it does, the surface probably wants to be split or downgraded in confidence.
The multi-reference validator is the practical version of that same idea. It compares scale estimates from different reference objects and checks whether they agree. Agreement earns the measurement more trust. Disagreement gets read as a sign that the scene may contain perspective distortion, lens issues, or a misdetection.
The approach is conservative on purpose. It won't rescue every region with a heroic guess. It asks whether the geometry is consistent enough to deserve confidence, and it only promotes the result when the scene agrees with itself.
How I keep zero-light capture from failing closed
Working in ideal conditions is table stakes. The property I actually care about is that the system still returns something useful when the RGB image is bad. The depth-only segmentation path exists for total darkness scenarios, which is a different failure mode from ordinary photo-based segmentation. If I can still read the LiDAR depth map, I can still find structure.
That matters because out in the field, a hard failure is the wrong answer. In a dark room, giving up because the image is ugly helps nobody. The useful behavior is to extract the geometry that remains, label the regions that survive thresholding and connected-component splitting, and keep the output good enough for calibration or measurement.
Closing
So I ended up with a depth pipeline that reads levels rather than photographs. It thresholds, groups, checks for discontinuities, and validates whether the resulting surfaces can be trusted, which is the reason it still has something to say when the room is nearly dark. The carpenter framing started as a way to explain the thing. It turned into the spec.