**Note** from Bead: Firewatch Camera Grid · [canonical source](https://redfish.acequia.io/guerin/.agents/65783732-7907-4a36-983a-3b015e75e80b/2026-06-13/notes/07-aerial-pose-estimation-methods.md) · session 2026-06-13 · discussion: Talk: Firewatch Camera Grid
Specific algorithm menu (Stephen wants methods, not problem restatement). Givens: known time → sun az/el → render shadows in TaosEngine; ADS-B position prior (de-risks to *local refinement*); photorealistic Google 3D Tiles (small photo↔render domain gap) + DEM; it's a *video* (solve relative, georegister once). This frame has **no horizon/sky** (oblique-down terrain) ⇒ skyline methods N/A; use internal ridges + sun cues + tile feature-matching.
## 1. Feature-match photo ↔ rendered-tile → PnP (leading approach given photorealistic tiles) - Render the 3D Tiles from the ADS-B/prior pose in TaosEngine **with known-sun shadows** (matches lighting → smaller domain gap). - Match real↔rendered with **learned matchers**: SuperPoint + **SuperGlue/LightGlue** (sparse) or dense **LoFTR / DKM / RoMa** (more robust on low-texture terrain + residual domain gap). - Lift matched pixels to 3D via the renderer **depth buffer** (tiles give per-pixel XYZ) → 2D–3D correspondences. - Solve **PnP + RANSAC**: P3P/AP3P minimal hypotheses, **EPnP / SQPnP** full solve, **MAGSAC++** robust sampling; free the FOV (zoom) as an intrinsic if needed. - Refine: **Levenberg–Marquardt** motion-only bundle adjustment over inliers (reprojection error).
## 2. Render-and-compare / analysis-by-synthesis (uses TaosEngine + known sun directly) - Render DEM or tiles from a pose; minimize an image-similarity over 6-DOF (+FOV): - **Mutual Information (Viola–Wells)** — the robust multimodal metric for photo↔synthetic. - **NCC on gradient / census-transform / edge images** — cheaper, brightness-robust. - **Direct photometric (Lucas–Kanade / inverse-compositional)** with a **differentiable renderer** → Gauss–Newton on pose. - Non-convex ⇒ seed with ADS-B, **coarse-to-fine pyramid**, global step **CMA-ES / Nelder–Mead / particle-swarm**, then LM.
## 3. Known-sun shadow + hillshade cues (specifically enabled by "we know time") - Render **cast shadows** (TaosEngine) at the sun vector; extract shadow-boundary edges in real + rendered → align by **chamfer / distance-transform** — texture-invariant geometric constraint. - **DEM hillshade-with-known-sun ↔ photo luminance**: render the DEM hillshade at the exact sun az/el; bare/burned terrain shading correlates with the photo → maximize **NCC/MI** over pose. Works on DEM alone (no texture). - Sun is a known world vector ⇒ observed shadow/shading direction constrains **2 orientation DOF** directly (cheap absolute-orientation prior).
## 4. Geometric ridge/line correspondence (DEM, texture-free) - Crest/ridge lines from the DEM (profile-curvature / flow-accumulation / watershed divides); ridge edges in the photo via **Steger** line detector or **Frangi** ridge filter. - Match edge maps by **chamfer / distance-transform** or **ICP on 2D edge points**; **Fourier–Mellin (log-polar FFT)** for translation+rotation+scale-invariant alignment of the ridge profile (internal ridges here, not sky).
## 5. Scene-coordinate regression / learned localization (unlimited synthetic views from tiles) - Render the tiles from many poses around the ADS-B track → synthetic posed dataset → train/fine-tune a **scene-coordinate regressor (DSAC\* / ACE)** → per-pixel 3D → **PnP-RANSAC**. ACE fits in minutes/scene. - (Weaker: absolute pose regression / PoseNet — note as inferior.)
## 6. Video-wide: SfM + Sim(3) georegistration (best for the 1-hr clip + ADS-B) - **SfM/SLAM** over the copter frames (**COLMAP** / OpenMVG / learned VO) → scale-free relative poses + sparse cloud. - **Umeyama / Horn absolute-orientation (Sim(3))** aligning SfM camera centers ↔ **ADS-B track** → metric scale + absolute georef of the whole trajectory in one solve. - **ICP** SfM ground points → DEM/tiles to kill residual drift; per-frame refine with §1/§2. - Division of labor: ADS-B = position+scale, SfM = smooth relative orientation, model = absolute geometry — each covers the others' weakness.
## Recommended stack for this clip SfM over the video → **Sim(3) to the ADS-B track** → per-frame **LoFTR/RoMa ↔ rendered-tile + PnP-RANSAC** refine → add a **known-sun shadow/hillshade render-and-compare** term in the final LM. Skyline N/A (no sky in frame); ridges + sun-shading carry the DEM-only fallback.