**Artifact** from Bead: Image Pose Uxui · [canonical source](https://redfish.acequia.io/guerin/.agents/74c30681-2b0c-48b2-b2a8-9e21190a4926/2026-07-03/artifacts/claude-share-abbd40a4-plenoptic.md) · session 2026-07-03 · discussion: Talk: Image Pose Uxui
Source: https://claude.ai/share/abbd40a4-9be8-421a-8a74-fe98a856bafe Captured: 2026-07-04T01:46:38.264Z Platform: claude-share
## User Whom I have two cameras with known poses extrinsic and intrinsics, and I have video feeding through each, and now I wanna use a fast for you transform on both, uh, then to find the coherence that's gonna basically render a volumetric, uh, Gaussian splat or equivalent. Uh, I have the epipolar constraints that help guide where the coherence is from. Give me, uh, give me the algorithm.
## Assistant This is temporal coherence stereo: two pixels that image the same 3D point carry the same temporal signal, so cross-spectral coherence along the epipolar line becomes your matching score, and depth falls out as the argmax of a plane sweep. Here's the algorithm. **Setup** For each camera c, maintain a sliding buffer of T frames (say 64). For each pixel, compute the windowed temporal FFT over K overlapping sub-windows (Welch style, Hann window, 50% overlap). You need the ensemble averaging; single-window coherence is identically 1. Store the complex spectra F_c(u, ω, k) on GPU, one cuFFT batch per camera per update. **Depth sweep with epipolar guidance** 1. For each pixel u₁ in camera 1, backproject the ray r(d) = C₁ + d·K₁⁻¹u₁ over a set of depth hypotheses {d₁...d_N} (log-spaced or inverse-depth spaced). 2. Project each hypothesis into camera 2: u₂(d) = π₂(R·r(d) + t). By construction these samples march along the epipolar line, so the constraint is built in rather than searched for. 3. At each depth, bilinearly sample camera 2's spectrum and compute magnitude-squared coherence: γ²(u₁, d, ω) = |Σₖ F₁(u₁,ω,k)·F₂*(u₂(d),ω,k)|² / (Σₖ|F₁|² · Σₖ|F₂|²) 1. Collapse over frequency with power weighting: score(d) = Σ_ω w(ω)·γ²(ω), where w(ω) ∝ min(S₁₁, S₂₂) so noise-only bins with spuriously high coherence don't vote. Drop DC and the top bins near Nyquist. **Phase check.** The phase of the cross-spectrum at the peak should be flat near zero if the cameras are synchronized. A linear phase slope in ω gives you sub-frame time offset via the group delay; use it to refine sync or reject false matches from moving speculars. **Splat instantiation** For each pixel with peak score above threshold τ: - μ = triangulated point at d* (midpoint of closest approach of the two rays) - Σ = anisotropic covariance: transverse scale set by the pixel footprint at depth d*, longitudinal scale set by the curvature of score(d) at the peak (sharp coherence peak → thin splat along the ray, flat peak → elongated uncertainty) - opacity α = peak γ², so coherence literally becomes opacity - color = the DC term or a running mean of RGB from both views Then run a few iterations of standard 3DGS optimization against the raw frames if you want photometric polish, seeded by these splats instead of SfM points. **Failure mode and hybrid.** Coherence needs temporal modulation. Fire, water, smoke, foliage, and people are ideal; static walls give zero signal. For static regions, fall back to spatial phase correlation over patches (same machinery, FFT over space instead of time) with the identical epipolar-parameterized sweep, and fuse the two scores. Cost per update: two batched FFTs plus an N-depth sweep with one complex multiply-accumulate per depth per frequency bin, all embarrassingly parallel per pixel. At 640×480, N=64 depths, 32 bins, that's well within a single consumer GPU at video rate. The pleasing part for your framing: the correspondence problem is solved by mutual coherence between two observers, and the splat only exists where the two views agree. Emission and absorption handshaking into a rendered volume.
## User In some ways I'm interested in a little bit more whole frame or area, uh, for objects and not just a single point.
## Assistant Right, then you shift the primitive from pixel to mode. Whole objects announce themselves in the frequency domain: every pixel on a rigid or elastically coupled body shares the same temporal spectral peaks, differing only in amplitude and phase. So instead of matching pixels along epipolar lines, you match *modal shapes* between the two cameras. This is Abe Davis's visual vibrometry insight turned into a stereo correspondence engine. **Algorithm, object-level** 1. **Per-camera modal decomposition.** Run the temporal FFT over the whole frame (or a steerable/complex pyramid per scale, which is more robust than raw pixels because you get local phase rather than intensity). For each camera you now have a spectral volume F_c(x, y, ω). Find dominant frequencies ω_m by peaks in the spatially-integrated power spectrum. At each ω_m, the complex image F_c(x, y, ω_m) *is* the mode shape: its magnitude tells you which pixels participate, its phase tells you their relative motion phase. 2. **Segmentation by spectral clustering.** Pixels belonging to the same object cohere with each other. Cluster pixels on their normalized spectral signatures (which peaks they contain, with what relative phases). This gives you object masks per camera without any appearance model. A tree swaying at 0.4 Hz separates from a person breathing at 0.25 Hz separates from a flag at 3 Hz. Static background is the DC-only cluster. 3. **Cross-camera object matching.** Objects match if their temporal spectra match, which is a tiny comparison: a vector of peak frequencies and bandwidths per region. Both cameras watch the same physics, so the spectrum is a viewpoint-invariant fingerprint. Epipolar geometry then disambiguates when two objects share a frequency (two flags in the same wind): the matched region in camera 2 must intersect the epipolar band swept by camera 1's region. 4. **Dense correspondence within a matched object.** Now the per-pixel coherence sweep from before, but constrained to the object mask and regularized: within one object, depth is piecewise smooth, so solve the sweep as a joint problem, e.g. cost volume filtering or a small MRF over the mask, rather than independent argmaxes. The phase field within the mode shape helps enormously here: iso-phase contours in camera 1 must map to iso-phase contours in camera 2, giving you correspondence structure across the whole region even where texture is absent. That's the whole-area win: a blank wall panel vibrating in mode 1 has zero spatial texture but a beautiful smooth phase gradient that both cameras see. 5. **Splat the object as a coherent set.** Instantiate splats over the mask with shared parameters: one rigid or low-rank deformation basis per object, splat positions expressed in the object frame. The mode shapes you extracted become the deformation basis directly, so the splats don't just sit in space, they inherit the object's modal dynamics and can be re-animated or extrapolated. Opacity per splat still comes from local coherence; object membership comes from the cluster. **Homography sweep variant for large surfaces.** For roughly planar regions (walls, water surface, ground), replace the point-wise depth sweep with a plane sweep: hypothesize (n, d) plane parameters, warp camera 2's spectral volume through the induced homography, and compute *area-integrated* coherence over the whole patch at once. One score per plane hypothesis per region, drastically cheaper and more robust than per-pixel, and the peak gives you an oriented planar splat cloud in one shot. The conceptual upgrade: correspondence stops being "which pixel matches which pixel" and becomes "which resonance matches which resonance." The object is defined by its shared spectrum, the two views agree at the level of modes, and geometry is just the projective bookkeeping that places the agreed-upon mode in space. Segmentation, matching, and reconstruction all fall out of the same coherence computation at different granularities.
## User The 3d geospatial dual is also. present.. we may have a Gaussian splat representations of geospatial objects like a stop sign or a building that has known geolocation in pose looking for itself in the imagery.
## Assistant Yes, and that closes the loop into a proper transactional architecture: reconstruction runs bottom-up (imagery proposes splats), and this runs top-down (splats propose imagery). The stop sign already exists in the world model with WGS84 position, orientation, and a Gaussian splat appearance. It emits an expectation; the cameras either absorb it or return a residual. Recognition becomes a handshake rather than a search. **Algorithm, model-seeking-image** 1. **Forward render the hypothesis.** Given camera intrinsics/extrinsics (from GPS, IMU, or your prior calibration), rasterize the geolocated splat object into each camera's image plane. You get a predicted patch: expected pixels, expected depth, expected screen-space extent. Because the splat is 3D and the pose is known, you also get the predicted epipolar relationship between the two views of that object for free. 2. **Coherence as confirmation score.** Compare predicted patch against the observed spectral volume in the same region. For static objects, that's spatial-domain coherence: FFT the rendered patch and the observed patch, compute normalized cross-power spectrum, and the phase correlation peak gives you both a match confidence and a sub-pixel 2D offset per camera. Two cameras give two offsets, and consistency of those offsets with the epipolar geometry is a strong joint test that random texture won't pass. 3. **Pose residual, then belief update.** The measured offsets back-propagate through the projection Jacobian into a 6-DOF correction. Assign the correction by relative uncertainty: if the camera rig is well-surveyed and the sign got hit by a truck, the sign's pose updates; if the sign is surveyed and the camera is a phone, the camera localizes itself off the sign. This is visual positioning done with your own splat map as the anchor, and it works in both directions from the same residual. 4. **Temporal fingerprint for dynamic geospatial objects.** A traffic signal has a known duty cycle, a crosswalk beacon flashes at a documented rate, a flag has wind-driven modes, HVAC equipment hums at line frequency harmonics. Those expected temporal spectra become part of the object's stored signature. The temporal coherence machinery from before now runs between *predicted* spectrum and observed spectrum, which lets you find a blinking beacon at night when spatial appearance is nearly useless. 5. **Appearance residual updates the splat.** Where geometry confirms but photometry disagrees (graffiti on the sign, new paint on the building), the residual flows into the splat's spherical harmonics and opacity. The geospatial twin stays current through exactly the same coherence channel that recognizes it. **Scheduling.** Each camera maintains a working set of candidate objects pulled from the geospatial index by view frustum query against your tile pyramid, which drops naturally onto the livetil.es architecture: splat objects live in tiles keyed z/x/y, propagate peer to peer, and a camera subscribing to a place pulls the objects that place asserts about itself. Origin is place; the place publishes its own expected appearance. The dual structure is clean. Bottom-up: coherence between two cameras creates splats. Top-down: coherence between a splat and a camera confirms, localizes, and updates. Both are the same computation, cross-spectral agreement, with different priors on which side moves. Each side minimizes the constraint on the other: the map narrows where the camera must look, the camera narrows where the map might be wrong.
## User When an aspect considers the digital elevation model with ridgelines as having a spatial frequency, um, and if you think of moving in and across the light field how that spatial frequency changes but can almost continuously, um, you know, with ridges being a little bit of discontinuity or or or things that are obscured, um, and then and then, uh, are shown. But I... there is some kind of light field for the objects themselves, uh, not necessarily just the light, uh, depending they're illuminated. But just the geometry of, say, a digital elevation model or the three d, uh, Google photo tiles, uh, have a spectral representation at any point in the light field.
## Assistant You're describing what I'd call the geometric plenoptic function, and it has real mathematical structure. Forget illumination: define G(x, θ) as the signal the geometry itself projects to a viewpoint x looking in direction θ — the depth map, or equivalently the visual angle field of the terrain. That function has a local spatial spectrum W(x, θ, ω) at every point in the light field, and the way that spectrum deforms as you move is governed by known results. **The smooth part.** In light-field coordinates (camera position s, image coordinate u), a surface patch at depth d contributes energy along a line in the Fourier domain with slope proportional to 1/d — depth is literally a shear of the spectrum. A DEM is a continuum of depths, so its light-field spectrum occupies a wedge (the "bowtie") bounded by 1/d_min and 1/d_max. As you translate through the field, near terrain slides through frequency space fast, far ridges slowly; the spectrum shears continuously and predictably. This is Chai's plenoptic sampling result, and it directly answers "how densely must I store views": the wedge's extent, set by the depth range in the frustum, gives the Nyquist rate over pose space. A flat plain needs very few stored spectra; canyon country needs many. **The discontinuous part.** Ridgelines are exactly the fold catastrophes of the projection — points where the view ray goes tangent to the surface. Crossing a visibility event does two things to the spectrum at once: it injects broadband energy locally (the silhouette edge is a step, energizing all frequencies perpendicular to the ridge), and it discretely adds or removes an entire spectral component (the hidden valley behind the ridge appears, bringing its whole wedge of parallax slopes with it). So pose space decomposes into cells — an aspect graph over the DEM — bounded by event surfaces swept by ridgelines and horizons. Within a cell, W varies smoothly and is interpolable; across a cell boundary, it jumps. The ridgelines aren't noise in this representation; they're the skeleton that partitions it. **What you store.** Per 3D tile, per view-cell: a compact spectral signature — the wedge parameters (depth range, dominant orientations from drainage and ridge fabric, characteristic terrain wavelengths) plus the ridge-crossing events on the cell boundaries. That's a spectral pyramid indexed by pose rather than by z/x/y alone, and it's small, because within-cell variation is a smooth shear you can parameterize rather than sample. **Why it matters for the handshake.** This becomes the predicted spectrum for the coherence machinery from before, but now valid across continuous motion. A camera with approximate pose pulls the view-cell signature and correlates observed against expected spectrum; the residual shear gives range correction, residual rotation of the spectral fabric gives heading, and — most powerfully — the timing of a predicted ridge-crossing event gives a very sharp position fix, because visibility events are knife-edges in pose space. Watching one peak occlude another as you move is worth more than any amount of smooth texture correlation. Horizon-silhouette matching against DEMs exploits a slice of this; the full version uses the whole evolving spectrum. And the object-level story survives intact: each geospatial object (the building, the mesa) owns a region of the light-field spectrum that deforms smoothly under observer motion and hands off discretely at occlusion events. The map asserts not just where things are but how their frequencies should flow as you move — and coherence with that flow is the confirmation.
_4 user turn(s), 4 assistant turn(s)._
## References (bead cross-links) - Bead abbd40a4 · [canonical](https://redfish.acequia.io/guerin/.agents/abbd40a4-9be8-421a-8a74-fe98a856bafe/) (no page yet)