**Note** from Bead: Anysurface Simulation · [canonical source](https://redfish.acequia.io/guerin/.agents/11fd8598-5a87-4a96-bd1d-a1d1563c14d7/2026-07-08/notes/02-multiview-geometry.md) · session 2026-07-08 · discussion: Talk: Anysurface Simulation
Directive (Stephen, 2026-07-08): use the pairwise fundamental/essential matrices between the two cameras and the projector to score the uv-uv correspondences, and produce depth maps for both cameras plus projector depth maps related to each camera.
## Conventions (shared with camera-pose.html) Everything mirrors [camera-pose.html](https://gsd.live/people/stephen-guerin/apps/camera-pose.html): each device is an OpenCV-style camera, R rows = [right; down; forward] (proper rotation, y-down folded into the down row), K all-positive, relative pose R_rel = R2·R1ᵀ, t_rel = R2·(C1−C2), E = [t_rel]× R_rel, F = K2⁻ᵀ E K1⁻¹, scored with the Sampson first-order geometric distance in pixels. The projector is simply a third camera: for the 1024×768 raster at fovY 37°, aspect 4:3, fx = fy ≈ 1147.6 px, principal point (512, 384). One subtlety: the taos drape convention (up=+X when pointing straight down, image v flipped) makes the naive [right; up; forward] basis improper (det −1); using the down row restores a proper rotation and the standard K.
## What v0.3.0 computes after each calibration - **Per camera pixel:** triangulate the camera ray against the projector ray through the decoded (u,v) (midpoint of closest approach) → 3D point → camera depth map, plus Sampson error against F(cam↔proj). - **Cam↔cam:** camera pixels that decoded the SAME integer projector pixel are matched (shared-code correspondence) and scored against F(L↔R). - **Projector depth maps:** each triangulated point also has a projector-frame depth; splatted into a ¼-resolution projector raster per source camera (min-depth per bin). The two maps differ where each camera is occluded, which is the "related to each camera" part: their union/difference is the projector's view of what each camera could vouch for.
## Results (defaults: noise σ=1 DN, gray or dual2d code, 8-bit sensors) - Depth rms vs ground truth: **≈1.07 mm** per camera. This is the half-projector-pixel quantization budget at 1.55 m throw (integer (u,v) codes), so the decode is not the bottleneck; finer-than-pixel structured light (phase shift) would push this down. - Sampson rms: L↔proj 0.088 px, R↔proj 0.088 px, L↔R 0.090 px over 4538 exact-code matches. - The L↔R match set is sparse (4538 of ~55k valid pixels) because it requires exact integer-code equality between two differently-sampled cameras. This is the hook for the next phase.
## v0.4.0 additions: epipolar refine option + camera resolution option - **Epipolar refine** (checkbox): snap each decoded (u,v) onto the camera pixel's epipolar line (1-DOF correction) and reject correspondences >1.5 px off the line. Honest finding at default noise: **no depth improvement** (1.09 mm either way, 0 rejected) because the residual error is quantization ALONG the epipolar line, which the constraint cannot see; the perpendicular component was already absorbed by midpoint triangulation. Where it earns its keep is robustness: at noise σ=4 + sensor gamma (bit-flip outliers), depth rms goes **184 mm → ~1–9 mm** and table height rms **41.5 mm → 1.4 mm**, with ~780 outliers gated. - **Camera resolution** (up to 3464² = 12 MP): above the 256² display resolution the decode runs as a streamed fused pass (optics + intensity synthesis + decode + triangulation per pixel, no stored captures, aggregates only). Per-sample depth rms stays ~1.09 mm (integer-code quantization bound), but sample density buys finer, better-averaged height products: table height rms **0.79 mm (256², 256×192 bins) → 0.48 mm (1024², 512×384 bins)**; 1 MP streams in ~7 s, 12 MP ≈ 80 s. Height products (table map, projector depths, drape) scale their bin resolution with camera resolution. - Also in v0.4.0: jet colormap on all depth/height rasters and (same gradient) match-error panes, with color-bar legends; rollover value readouts on every raster and on the 3D table (true vs reconstructed height); the reconstructed height colormap can be **draped back onto the sand** through the projector (AR-sandbox loop, fused projector depths only, no ground truth).
## Next phase (Stephen): minimal patterns for cam↔cam correspondences Also declared (2026-07-08): after a regen of the sand on an already-calibrated rig, explore **minimal projection probes** to recover just the changed height (not a full 22-frame rescan). The epipolar + fused-depth machinery above is the substrate for that: a probe only needs to disambiguate along known epipolar lines, and the prior height map is a strong prior. The epipolar constraint means a candidate match needs only 1 DOF of disambiguation along the epipolar line, not a full 20-bit code. Directions to explore: coarse code + epipolar-guided refinement; per-scanline codes in rectified space; and the dual-photography transport framing where the projector basis is chosen to make T's relevant slice cheap to measure.
## Debbie guardrail Per the standing simplicity offer (tie-points bead `fecb418a`): none of this math appears in the chrome. The UI group is "match error", captions "L↔R", "L↔proj", green-to-red heatmaps only.