**Note** from Bead: Through The Lens Campanile · [canonical source](https://redfish.acequia.io/guerin/.agents/7610bb17-52c1-4e5b-991b-16d61e06ed46/2026-06-16/notes/02-modern-ttl-survey.md) · session 2026-06-16 · discussion: Talk: Through The Lens Campanile
> Deep-research harness, 2026-06-16 (run `wf_42dee893-04d`): 6 search angles → 26 sources fetched → 115 > claims → 25 adversarially verified (3-vote) → **23 confirmed, 2 killed**. All findings below are > high-confidence (3-0) from primary sources unless flagged. Honest coverage gaps called out at the end.
## Bottom line — recommended architecture for our app (`#image-pose-uxui`) For an interactive WebGPU/JS app where a user gives a telemetry-less photo a 6-DOF + intrinsics pose by dragging UV↔geopoint correspondences against Google 3D tiles, the evidence points to a **two-stage** design: 1. **Bootstrap with SQPnP** from the first ≥3 UV↔geopoint pins. SQPnP is **globally optimal** (casts PnP as a QCQP over SO(3), guarantees a global minimum for any n≥3 *regardless of coplanarity*), converges in ~10 iterations, and uses **only standard linear-algebra ops — no polynomial/Gröbner solving** — so it ports cleanly to JS/WASM. Optionally wrap in **RANSAC seeded by P3P** for outlier robustness. 2. **Refine live with a Gleicher–Witkin differential controller** during the drag: integrate camera-parameter velocity from image-point velocity via the pseudo-inverse of a **2m×7 quaternion Jacobian** (Kyung–Kim–Hong's cleaner form), with **pinned points as hard Lagrange constraints** and the **actively-dragged point as a soft objective term**. 3. **Frame the whole thing as one soft weighted-least-squares reprojection objective** so intrinsics (focal, principal point, simple distortion) fold in as extra optimized parameters, each pin carrying an **information-matrix (inverse-covariance) weight** = its confidence. This is a direct upgrade path for our current `solver.js` (today: LM-only, fixed `solveFov`): **add an SQPnP bootstrap** (better global seed than the view-derived prior) and, as a v2, **an optional differential-drag mode** for the buttery "world warps as you drag" feel of the Naimark/Gleicher demos.
## The two interactive paradigms (both reach the same calibrated camera) | | **Discrete LM resectioning** (our `solver.js`, Naimark 27:05) | **Differential through-the-lens** (Gleicher–Witkin 1992) | |---|---|---| | Cadence | drop pin → solve reprojection-error min to convergence → redraw | per-frame: solve for camera-parameter *velocity*, integrate as a 1st-order ODE | | Why | robust global solve from a seed; OpenCV `solvePnPRefineLM`/`VVS` | G–W chose it *because* "the direct nonlinear solve for camera params from targets is unlikely to succeed — there may exist no solution, or many" | | Constraints | all pins soft, weighted reprojection LS | **hard** pinned points (Lagrange multipliers) vs **soft** dragged control (folded into objective); conflicts resolved strictly for the hard set | | Trade-off | global-solve robustness; discrete "snap" per pin | smooth, stable, drag-responsive; clean hard/soft layering; needs a decent start | | Tooling | OpenCV PnP family (mature) | the 1992 paper + Kyung–Kim–Hong refinements; no off-the-shelf lib (we'd implement it) | **G–W is literally our app, proposed in 1992:** *"registering 3D models with photographs… display a real image as a backdrop and pin points on the synthesized image to their corresponding locations. Using a least-squares technique for overdetermined matrices… the system will move towards the best fit (Figure 4)."* (`graphics.cs.wisc.edu/Papers/1992/GW92/camera.pdf`)
## 1. Classical PnP family — the resectioning workhorses *(mature, shipping in OpenCV)* - **P3P** — minimal case: 3 correspondences → ≤4 real pose solutions (quartic), a 4th disambiguates by smallest reprojection error. The standard **RANSAC hypothesis generator**. (OpenCV `SOLVEPNP_P3P/AP3P`, which require exactly 4 points via the API.) - **EPnP** — general O(n), n≥4: expresses the n 3D points as a weighted sum of **4 virtual control points**, solving via eigenvectors of a 12×12 matrix (vs prior O(n⁵)). Fast, planar+non-planar. (OpenCV `SOLVEPNP_EPNP`; Lepetit et al., IJCV 2009.) - **SQPnP** — **globally optimal**, n≥3, coplanar-safe, standard-linear-algebra-only, ~10 iters. **The best JS/WASM bootstrap candidate.** (Terzakis & Lourakis, ECCV 2020; OpenCV `SOLVEPNP_SQPNP`.) - **IPPE** — coplanar-specialized (`SOLVEPNP_IPPE`/`IPPE_SQUARE`). - **Refinement**: `solvePnPRefineLM` (Levenberg–Marquardt) and `solvePnPRefineVVS` (Gauss–Newton / virtual visual servoing, exp-map rotation updates) both minimise reprojection error from an initial pose, n≥3 — these **are** the discrete-LM paradigm. - *Soft/hard mapping:* vanilla PnP minimises **unweighted** residuals; the standard generalization adds per-correspondence **covariance/information-matrix weights** (Generalized MLE PnP = motion-only bundle adjustment). Everything is soft.
## 2. Bundle adjustment / factor graphs = the "every constraint is soft" view *(mature)* g2o (and Ceres, GTSAM) minimise `F(x) = Σ_k e_k(x_k,z_k)ᵀ Ω_k e_k` over a graph: nodes = parameter blocks, edges = measurement constraints, **each weighted by an information matrix Ω_k** (inverse covariance). Solved by Gauss–Newton or LM (`(H + λI)δx = −b`). **Every measurement is an Ω-weighted squared penalty — a soft constraint, never a hard equality.** This is exactly our bead's `feedback_every-constraint-is-soft` frame, and it's the canonical formulation across the field. (g2o: Kümmerle et al.; Ceres = the common choice for camera+intrinsics BA; GTSAM for factor graphs.)
## 3. Differential TTL lineage *(foundational; we'd implement it)* - **Gleicher & Witkin 1992** — solve for **time derivatives** of camera params from time derivatives of controls. Per-step: minimise `‖q̇ − q̇₀‖²` s.t. `ṗ = J q̇` → Lagrange → `J Jᵀ λ = ṗ₀ − J q̇₀`, `q̇ = q̇₀ + Jᵀλ` (minimum-norm / Moore–Penrose pseudo-inverse, since J is singular — many camera motions give the same image motion). Integrate q as an ODE; a §3.6 position-feedback term counters drift. Hard pins via Lagrange, soft drag in the objective; inequalities (stay-in-frame) via active set. - **Kyung, Kim & Hong (GI'95 / GMIP'96)** — refine it: a simpler **2m×7 Jacobian** (2 image coords × m points; 7 = 3 translation + 4-component unit quaternion), dropping the redundant 8th column; reframed as target-tracking / constrained nonlinear inversion by integrating a tangent vector field on the quaternion configuration space. **Use this Jacobian form**, not the original 2m×8. - **Image-based visual servoing (IBVS)** — the controls-theory cousin: regulate image features to a target via the interaction matrix (image Jacobian) and its pseudo-inverse, `v_c = −λ L_e⁺ e` (exponential error decay). Layers a **hard primary visual task** with **soft secondary tasks** (occlusion/limit avoidance) **projected into the null space** of the primary so they can't disturb it — a real-time hard/soft layering identical in spirit to G–W. (Chaumette & Hutchinson; Inria ViSP.)
## 4. Differentiable rendering / inverse-camera *(mature for joint pose+scene)* - **BARF (Bundle-Adjusting NeRF, ICCV 2021)** — jointly optimises the radiance field **and** camera poses ("the joint problem of learning neural 3D representations and registering camera frames"; explicit theoretical link to classical image alignment), training NeRF from imperfect/unknown poses. **Key caveat we'd inherit:** naive full positional encoding *hurts* gradient-based pose registration; a **coarse-to-fine encoding schedule widens the basin of attraction**. Successors (GARF, FA-BARF) bypass the schedule via different activations. The same joint-pose-refinement idea is now standard in 3D Gaussian Splatting pipelines. *Relevance to us:* overkill for single-photo pinning, but it's the principled "refine pose by re-rendering and comparing pixels" end-state if we ever go correspondence-free.
## Coverage gaps (honest — no verified claims survived) The harness could **not** produce evidence-backed findings on three requested areas, so treat these as **open**, not covered: - **(5) Learned pose / scene-coordinate regression** (PoseNet successors, DSAC*, **ACE / ACE-Zero** by Niantic) — sources were fetched (`nianticlabs.github.io/ace`, `github.com/nianticlabs/acezero`) but no claim passed verification in budget. - **(6) SfM / visual localization to a prior mesh** (COLMAP, **hloc** `github.com/cvg/Hierarchical-Localization`) — same: fetched, not verified. - **(7) Interactively aligning one photo specifically to Google Photorealistic 3D Tiles** — fetched Google/deck.gl/Cesium tile docs, but nothing on the *correspondence/alignment* problem survived.
## Open questions (for a follow-up pass) 1. JS/browser readiness of learned regressors (ACE/DSAC*) for single-image pose against a known mesh. 2. How COLMAP/hloc register one new photo to an existing georeferenced mesh; any WebGPU/WASM components. 3. Published work/tooling on aligning a photo to Google 3D Tiles — esp. **extracting reliable 3D geopoints from tile geometry** for the UV↔geopoint pins (our pin→world pick is the live weak spot). 4. Best way to fold intrinsics (focal/principal/distortion) into the *differential* formulation, and how to encode per-pin confidence as information-matrix weights.
## Refuted (killed in verification — do NOT assert) - ✗ "G–W Jacobian solved specifically by **weighted least squares using SVD**" (1-2). Safe statement: *least-squares / minimum-norm solution of a possibly-singular Jacobian* (recipe unspecified). - ✗ "G–W is *very similar to visual servoing* and *did not handle trajectory constraints*" (1-2) — overreach.
## Sources (primary unless noted) - OpenCV solvePnP (PnP family + refine) — `docs.opencv.org/4.13.0/d5/d1f/calib3d_solvePnP.html` - SQPnP, ECCV 2020 — `ecva.net/papers/eccv_2020/papers_ECCV/papers/123460460.pdf` - PnP review; EPnP (Lepetit IJCV'09); P3P (Kneip CVPR'11) - g2o — `researchgate.net/publication/224252449` - **Gleicher & Witkin 1992 (the paper)** — `graphics.cs.wisc.edu/Papers/1992/GW92/camera.pdf` + video page `gleicher.sites.cs.wisc.edu/video/1992_ttl/` - Kyung–Kim–Hong 2m×7 Jacobian — `researchgate.net/publication/2382901`; GMIP'96 `sciencedirect.com/science/article/abs/pii/S1077316996900222` - IBVS — `inria.hal.science/inria-00352095v1`; Chaumette & Hutchinson 2006; ViSP - BARF — `chenhsuanlin.bitbucket.io/bundle-adjusting-NeRF/` (ICCV 2021, arXiv:2104.06405) - (gap-area, fetched not verified) ACE `nianticlabs.github.io/ace`; hloc `github.com/cvg/Hierarchical-Localization`; Google 3D Tiles `developers.google.com/maps/documentation/tile/3d-tiles-overview` *Stats: 6 angles · 26 sources · 115 claims → 25 verified → 23 confirmed / 2 killed · 109 agents.*