**Note** from Bead: Firewatch Camera Grid · [canonical source](https://redfish.acequia.io/guerin/.agents/65783732-7907-4a36-983a-3b015e75e80b/2026-06-13/notes/05-calibration-and-measurement-pipeline.md) · session 2026-06-13 · discussion: Talk: Firewatch Camera Grid
Stephen, 2026-06-13, filling in the "measure" axis (note 04). The point: pose is a *prior* with error; **calibrate it, then geometry does the measurement** (not ML detection).
## What we already have (the prior) - **Pose prior** per camera: position (STAC Point, SoCal), pan/tilt (telemetry `camAzimuth`/`camElevation`). - **FOV from zoom** — a transfer function on `camZoom` (range **0–40**) → **horizontal FOV 62.5° (zoom 0) … 2.5° (zoom 40)**. (Optical zoom ⇒ the curve is the focal-length/tan relation, not linear: `hFOV(z) = 2·atan( tan(62.5°/2) · s(z) )` with `s` chosen so z=40→2.5°; ~28× at the tan level. Reconcile/verify against the reported `camViewWidth` per frame — telemetry z=1 reported 65.33°, so there's a few-% offset the calibration resolves.) `fovy` from sensor aspect.
## Step 1 — Calibration sidecars (Pan/Tilt correction) — THE UNLOCK Solve a per-camera **Δpan/Δtilt** (and possibly Δroll, FOV scale) that maps the noisy prior to truth. Two routes: - **Stars (night frames).** RA/DEC → **alt/az** for the camera's lat/lon + the frame's UTC (deterministic, no deps). Detect bright points in a clear night frame, match the pattern against predicted star alt/az under the prior pose+FOV, least-squares the offset. **We get this nearly free:** the wall *opens at night* (extent starts 07:23Z ≈ midnight PT) and we have exact per-frame UTC + camera lat/lon. Astrometric self-calibration, no human-marked landmarks — like plate-solving with a strong prior. - **3D geopoints (day frames).** Known terrain/landmark points whose bearing/elevation (φ/θ) from the camera is computable (DEM ridgelines, peaks, towers) → match to their pixel → solve the same offset. Backstop when the sky is hazy/smoky. - Output = a **calibration sidecar** per camera (the named deliverable): `{Δpan, Δtilt, Δroll?, fovScale?, method, residual, solvedFromUTC}`. This is also a textbook **active-cataloger** loop (note 03): compose the solver per camera, run it **edge-side**, reify the sidecar. ### Feasibility check (2026-06-13) + stars = citizen/educational, not a plate-solver to build Pulled an actual night frame (15801 South_Mountain_East_1, 2026-05-18 00:36 PT, zoom 1, az 78.51 / el −0.44 burned into the HUD). This particular frame is **gain-blown** — the sky is a featureless grey gradient, no obvious stars (night fire-watch exposure) — but it is rich in two better-for-this-frame features: a **dense field of city/street lights** across the valley + a clear **ridgeline**. **Stars (Stephen): don't build a plate solver — we already have star solvers, and citizens finding the bright ones is fun + educational.** So the star route is reframed: reuse existing solvers + a **crowd-sourced/gamified "find the bright star" affordance** (a citizen clicks a star they recognize → a known alt/az anchor → contributes a calibration constraint; learn-the-sky engagement). Best on clear-dark frames / the brightest objects (Venus, Sirius, Jupiter punch through even some haze). The gain-blown frame above just shows why we ALSO need an all-conditions automated path → city-lights + ridgeline render-and-compare below. The two are complementary: **citizen star-finding (clear nights, educational) + automated render-and-compare (day + hazy)**. ### Calibration via "dual" render-and-compare — objects look for themselves (Stephen, 2026-06-13) > *"dual geospatial object looking for themselves in imagery. eg vector layer of streetlights, stop signs viewpoint-relative ridgetop fft. interesting to think of an fft field of a landscape from a viewpoint. has almost a lightfield sense. eg invariants fft in f(translation, phi/theta, scaling)."* Instead of blind feature detection, take a **known geospatial layer** and **project it into the image** via the pose prior, then optimize Δpan/Δtilt/Δzoom(/roll) until it matches — the objects *look for themselves* (analysis-by-synthesis / render-and-compare). Two complementary registrants, robust day **and** night (which star-cal is not): - **Point layer (great at night):** streetlights / stop signs / town centroids / road-graph nodes as a vector layer → project to expected pixels → match the observed light field by **chamfer / ICP**. The night frame's city-light blobs ARE the streetlight layer. - **Ridgeline / skyline (great by day, lighting-robust):** render the **DEM horizon** from the pose prior; register observed-vs-rendered skyline with **FFT phase-correlation** (translation ⇒ pan/tilt shift) + **log-polar FFT = Fourier–Mellin** (rotation+scale ⇒ roll + zoom). This is the "ridgetop FFT," invariant to lighting/haze because it's edge/frequency-domain. - **The FFT field / lightfield framing:** the set of views over the pose manifold is a (sparse) lightfield; its spectral signature is a field over `f(translation, φ/θ, scaling)`. Precompute DEM-rendered-horizon FFTs over a pose grid; the observed FFT's match-peak gives the correction, and **Fourier–Mellin invariants collapse the search** (no brute-force over the whole manifold). The same invariant search later helps cross-frame/cross-camera correspondence (Step 3). So Step-1 method (revised): **render-and-compare against DEM ridgeline + street-light/road vector layer, FFT/Fourier–Mellin for robustness** — not stars. Pure active-cataloger: compose per camera, run edge-side, reify the calibration sidecar.
## Step 2 — Surface rate of spread (2D, georeferenced) Once pose is calibrated: track the **fire front** across frames, project each pixel ray to the **DEM** (ray-cast against terrain), get the front's ground position over time → **ROS (m/min) + direction**. The spread-out cameras (South_Mountain, Las_Virgenes, Decker, Oat_Mtn) cross-check the same front.
## Step 3 — 3D plume photogrammetry Reconstruct the plume in 3D from the **spread** cameras (the **co-located pairs — Castro 1886/1887, Topanga 2763/2764 — give ZERO baseline**, so they're for coverage/zoom, not depth; the geographically separated cams supply parallax). Correspondence/depth via **optical flow, SIFT, BRIEF, FFT**. Two approaches: - **(a) Fit a fluid-dynamics smoke model — low-hanging fruit.** Constrain TaosEngine's fluid sim (`https://taosengine.com/samples/fluid_test.html`) to the observed plume silhouettes/front from multiple cameras → a plausible animated 3D plume. Pairs with the incident-viewer's smoke-at-leading-edge directive. Cheaper; only needs silhouettes + front + wind prior. - **(b) Sparse photosampling + object inference → 3DGS.** Reconstruct the scene/plume as 3D Gaussian splats from the sparse posed views (`https://taosengine.com/samples/`). More faithful, heavier, demands good pose (⇒ Step 1 first) and enough baseline (⇒ the spread cams).
## Step 2.5 — Annotation → GIS (the detection-free measurement front-end) + COP augmentation Stephen's add: *"plus relevant operating picture letting people scrub time with their data in (augmentation) and generating gis by 'annotation' of an image (point, polyline, polygon, and raster painting)."* **Calibrated pose makes drawing on a posed image = GIS authoring.** Annotate a frame with **point / polyline / polygon / raster-paint** → back-project each mark through the calibrated pose + **DEM ray-cast** → a **georeferenced** feature in world coords: - trace the **fire front** on South_Mountain's image → a perimeter **polyline** on the map/3D (and across frames → **ROS**, feeding Step 2); - drop a **point** on a spot fire → a located point, cross-checked by a second camera (= the triangulation of note 04); - **paint** the burned/smoke area → a georeferenced **raster** draped on terrain. This is the measurement front-end that replaces "detection": a human *or* AI annotates, **geometry does the georeferencing**. The most natural UI (draw on the photo) emits the highest-value product (GIS). Requires Step 1 (calibration) to be accurate. **COP / augmentation.** The picture is time-aware and people **scrub time with their OWN data layered in** — crews, assets, observations, other sources — bring-your-own layers fused on the same timeline. Access governed by the **bind-to-URI authz** (note 02 layer 3): your data, your scope; shared picture, per-parciante augmentation. The manager becomes a shared, time-scrubbable **common operating picture**, not a single-org viewer. ### Step 3 expanded — the plume is a VOLUME, not a surface (commentary 2026-06-13) Key reframe: smoke is **semi-transparent participating media**, not a textured rigid surface — so classic SIFT/BRIEF point-matching is weak *on the plume itself* (no view-consistent features; what you see is a line-integral of density). SIFT/BRIEF/optical-flow are still right for the **ground** (terrain registration, fire-front tracking, advection/wind), just not for plume depth. Treat plume reconstruction as **sparse-view volumetric**, and stage it: 1. **Visual-hull / silhouette space-carving (robust, calibration-only — the real low-hanging fruit).** Segment the plume silhouette in each calibrated camera, back-project as a cone, intersect the cones → the plume's 3D **envelope**: height, **lean/tilt** (→ wind dir+speed), footprint. High-value scalars (plume height → Byram **intensity**; lean → wind) with no sim, no ML — just segmentation + the calibration we're building, anchored at the **triangulated fire base** (note 04). 2. **Fluid-model fit (Stephen's low-hanging — the regularizer).** ~6 unsynchronized, minutes-apart, grazing-angle, long-range cameras can't solve a free volume cold. So don't reconstruct — **constrain a physics sim** (TaosEngine fluid_test): seed the source at the triangulated fire, set wind from the hull lean, then optimize buoyancy/entrainment/wind so the sim's **rendered projections match the observed silhouettes/opacity across cameras** (differentiable render-and-compare). Gives a temporally-coherent, **animatable, predictive** plume (run it forward) — ties to the smoke-at-leading-edge directive + ToA. 3. **Sparse-photosample → 3DGS** — reserve for the **static scene/terrain stage** (good pose + enough baseline); vanilla 3DGS fights smoke (view-dependent + dynamic). Dynamic/4D-GS variants are a research path, not the first plume solver. **Geometry caveats (honest):** co-located pairs (Castro/Topanga) give **zero baseline** → coverage/opacity only, not depth; the spread cams (South_Mountain, Las_Virgenes, Decker, Oat_Mtn, Saddle, Topanga_Canyon) supply parallax. Frames are **minutes apart and unsynchronized** → reconstruction instants are coarse and the plume moves between them; grazing/long-range geometry conditions depth poorly along the line of sight. So the deliverable is a **coarse envelope + a physics-constrained fit → derived scalars** (height, lean→wind, source, growth-rate), not a crisp voxel plume. **Unifying spine:** calibration (ridgeline/citylight render-and-compare), annotation→GIS (back-project), and plume (fluid-fit / 3DGS) are all **analysis-by-synthesis** — forward-render a hypothesis, minimize the image-space residual. One computational backbone, and TaosEngine is the forward-renderer for all of it (terrain, fluid sim, splats) in the same 3D scene as the posed cameras + ToA.
## Dependency order (and why) `pose prior (have) → [1] calibration sidecars → [2] surface ROS → [3] plume (fluid-fit, then 3DGS)`. Calibration gates everything — uncalibrated pose makes ROS and photogrammetry garbage (cf. the FMV attitude-convention bug: a few-degree pose error threw the boresight 50–90°). Calibration is also the most tractable + elegant first build: deterministic astrometry + a strong prior + the night frames we already have.
## Other source: news-copter aerial footage (1–8 h streams) Stephen, 2026-06-13: long news-helicopter streams (e.g. a public FB Live, 1 h) as a catalog source. Significance: - It's **moving aerial FMV without telemetry** — like the MISB FMV but with **no broadcast KLV/pose**. So the challenge is **pose recovery** from the video itself: SfM / landmark-registration / the same render-and-compare calibration loop (note 05 §1) against terrain + the city-light/ridgeline layer, per frame. - **A moving camera is the BEST photogrammetry source we have:** a single aerial platform over time sweeps a **continuous baseline** → strong multi-view parallax (unlike the fixed PTZ cams, and unlike the zero-baseline co-located pairs). Once pose is recovered, copter footage may be the prime feed for the 3D plume/terrain reconstruction + the dual-airbrush carving (note plan-cameras-into-incident-viewer). - **Provenance/rights:** broadcaster-owned public stream → record attribution + source URL in the catalog (advanced-wave / bind-to-URI authz); ToS-aware (downloading for incident analysis; don't redistribute). Large media → externalize (Downloads/object-store, link by URL), never store the video in the bead. - Download: yt-dlp resolves it (public; up to 720p ~549 MB for 1 h). Feasible. **Pose recovery, solved by fusion (Stephen 2026-06-13: "it might be visible in camera and historical adsb").** The copter is its own beacon: - **Historical ADS-B** (ADSBExchange / OpenSky archives) gives the copter's **lat/lon/alt/track per second** → the camera **position** prior *directly* — which is the hard part of telemetry-less pose. SfM/landmark then only has to recover **gimbal orientation** (+ refine), not absolute position/scale. Massively de-risks the news footage. (Anchor the ADS-B query with the video's timestamp — pull it from yt-dlp metadata — + the fire's lat/lon to find the orbiting helicopter; tail/callsign narrows it.) - **Copter visible in a fixed camera** = a **moving, known-position target**: the calibrated fixed cams see the copter at its ADS-B position → (a) cross-validate/refine fixed-cam calibration (pan/tilt/FOV, better than static ridgelines for some params), (b) refine the copter position, and (c) **sync the news-footage clock to the fixed-camera clock** via the sighting. - **Multi-source mutual constraint:** ADS-B ↔ copter footage ↔ fixed cameras ↔ ground landmarks — each source tightens the others. This is the active-cataloger fusing partially-structured sources into one posed, time-aligned picture; every new feed adds constraints rather than just data. ### Acquired instance (2026-06-13) — ABC7 Sandy Fire copter, and it OVERLAPS the wall Downloaded (yt-dlp, public): `C:\Users\steph\Downloads\news-copter-sandy-1627099769423766.mp4` (+ `.info.json`). **NOT in the bead** (578 MB, externalized; bead records provenance only). - **Provenance:** ABC 7 Chicago, "WATCH LIVE — Sandy Fire in Simi Valley, California"; src `https://www.facebook.com/watch/live/?ref=watch_permalink&v=1627099769423766`; article abc7chicago.com/post/sandy-fire-update-…/19125975/. - **Specs:** 1280×720 h264 @ 30 fps, AAC; **59:08** (3548.6 s); **stream start 2026-05-19T13:05:45Z** (timestamp 1779195945) → window **≈ 2026-05-19 13:05:45Z → 14:05Z** (~06:05–07:05 PT, morning May 19). - **Key:** this window is **inside the fixed-camera wall extent** (2026-05-18T07:23Z → 05-19T20:51Z). So at those instants we have the **moving aerial AND all 10 fixed cams on the same fire simultaneously** — a prime multi-view moment for cross-calibration (copter-in-fixed-cam) + the dual-airbrush carving. - **ADS-B query now pinned:** historical ADS-B over Simi Valley (~34.27, −118.78) for **2026-05-19 13:00–14:10Z** → find ABC7's (or the pool) orbiting helicopter → its lat/lon/alt/track = the camera-position prior. ### ADS-B source strategy (2026-06-13) — adsb.lol open archive PRIMARY, FR24 paid fallback The ADSBExchange acquisition (JetNet, 2023) spawned open community forks: **adsb.lol**, **adsb.fi**, **airplanes.live** (adsb.one+adsb.fi merger) — all **unfiltered** (military/fire included), ADSBExchange-API-compatible. - **adsb.lol `globe_history` = the free, open, self-hostable historical source** (matches Stephen's "download + host" preference; the RTE/acequia model). GitHub `adsblol/globe_history_2026` (yearly repos 2023→2026), **one release per day**, **ODbL-licensed**, **unfiltered**. Each day ≈ **1.5–3 GB** split-tar (`vYYYY.MM.DD-planes-readsb-prod-0.tar.aa/.ab/…`) → extract → **one JSON.gz readsb *trace* per aircraft** (timestamped lat/lon/alt/gs/track for the day). **No API key.** Covers 2026 ⇒ **Sandy = `v2026.05.19-planes-readsb-prod-0`**. Caveat: community-feeder coverage of low-altitude fire/heli over the fire may be thinner than FR24's commercial net (but unfiltered + free + hostable → try first). - **FR24 API = paid fallback** for denser coverage (note above: Explorer $9/30-day reaches Sandy if subscribed in ~5 days; Essential $90/2-year). Use if adsb.lol coverage of the Sandy low-altitude traffic is thin. - **ADSBExchange:** $10 RapidAPI = live only; historical = enterprise (10 yr, $$$). Not the bargain for history. Skip unless enterprise archive ever needed. - **Ingest flow (adsb.lol):** download the day's split-tar → reassemble + extract → scan per-aircraft traces for those entering the Sandy bbox during the window → keep the subset (fire 1255 / military / news-heli) → **self-host the filtered subset** + feed the camera-position prior. Identify mil/fire by registration/type/squawk in the trace. ### ADS-B reusable skill (2026-06-13)
- **Squawk for firefighting ops = 1255** (US; 1200 = generic VFR). The pull filters/highlights 1255 + the news-heli + air-tankers/leadplanes. - **Access verdict (probed):** historical is credential-gated — OpenSky anon → **403 "cannot access historical flights"**; ADSBExchange → **402** (paid); live works but is today-only. So 25-day-old data needs an account/key. - **SOURCE = Flightradar24 API** (Stephen has it; spec at `artifacts/Flightradar24-API.yaml`, OpenAPI 3.1, 3547 lines). Meets both requirements — **1255 firefighting AND military** — in one source. (OpenSky=free-but-thin-low-alt, ADSBExchange=paid-unfiltered; FR24 chosen since Stephen already subscribes and its API exposes military.) - **Auth:** `Authorization: Bearer <token>` + `Accept-Version: v1`. Credit-metered → be economical; `/api/usage` to monitor. - **Historic:** `GET /api/historic/flight-positions/{full,light,count}` — back to 2016-05-11; snapshot by `timestamp` + filters; `flight-events` historic too. - **Filters:** `Bounds` (bbox), `Timestamp`, **`Squawks`** (→ `1255`), **`Categories`** (`M`=MILITARY_AND_GOVERNMENT ✓, `H`=HELICOPTERS, P/C/J/T/B), `Callsigns`, `Registrations`, `Flights`. - **Tracks:** `GET /api/flight-tracks` → per-flight lat/lon/alt trajectory = the **camera-position prior**. - Query model: historic is a **snapshot at an instant** (timestamp); to get a track, find the flight id from a snapshot then call `/api/flight-tracks` (or sample timestamps). Economical flow: 1 snapshot in the window (filtered) → flight-tracks for the few of interest. - **Credential storage:** `.credentials/flightradar24-api-token.txt` (+ manifest entry), per credential discipline. - **Reify as a skill** (active-cataloger, note 03): `pull-historical-adsb` (FR24-backed) — params `(bounds, timestamp|window, squawks?, categories?)` → snapshot + per-flight `/api/flight-tracks` → **per-aircraft posed track sidecars** = the **camera-position prior** for telemetry-less aerial footage (note 07 §6 Sim(3) anchor). First run: Sandy window `bounds≈[34.45,-119.0,34.1,-118.5]` (lat-hi,lon-w,lat-lo,lon-e per FR24 order), `2026-05-19 13:00–14:10Z`, `squawks=1255` + `categories=M,H`.
## TaosEngine references (Stephen) - Fluid sim: https://taosengine.com/samples/fluid_test.html - Samples (incl. 3DGS): https://taosengine.com/samples/