**Note** from Bead: Taos Engine · [canonical source](https://redfish.acequia.io/guerin/.agents/c66cbd1d-453c-41f8-8440-179502f25de4/2026-06-09/notes/taos-misb-build-plan.md) · session 2026-06-09 · discussion: Talk: Taos Engine
**Date:** 2026-06-09 · **Bead:** `c66cbd1d-453c-41f8-8440-179502f25de4` · **Status:** workshop / planning (no viewer code written yet) > Goal: reproduce the MISB FMV viewer (`apps/viewer-3d/index-misb.html`, documented in bead [#c38c1239](https://redfish.acequia.io/guerin/.agents/c38c1239-bfd3-44dd-8d97-1a0aa39ac8da/2026-06-08/artifacts/misb-viewer-data-formats.md)) on top of **TaosEngine** (Brendan Duncan's WebGPU geo engine), starting from the captured `geo_osm_projection` prototype ([artifact](https://redfish.acequia.io/guerin/.agents/c66cbd1d-453c-41f8-8440-179502f25de4/2026-06-09/artifacts/geo_osm_projection-prototype.html)).
## 1. What the MISB viewer is (the thing to duplicate) A multi-aircraft Full-Motion-Video viewer (three.js + `3d-tiles-renderer` today): - **Globe/terrain**: Google Photorealistic 3D Tiles / Cesium terrain, positioned so an origin lat/lon maps to scene (0,0,0); local ENU (Y-up) scene frame; geodetic↔local via ECEF. - **Panospheres**: per-aircraft, a gnomonic (rectilinear/pinhole) patch of a sphere, UV-mapped with the aircraft's **MP4 video as a live `THREE.VideoTexture`**, sized by sensor FOV, positioned at the sensor lat/lon/alt, oriented by absolute yaw/pitch/roll. - **Telemetry drive**: MISB ST 0601 samples (two on-disk dialects → one normalized in-memory `sample`), interpolated per video-time; pose = `{lat, lon, altMSL, yaw, pitch, roll, hFOV, vFOV, frameCenter, slantRange}`. Gimbal angles are **body-relative**; absolute = `platform_attitude + sensor_relative`. - **Master UTC timeline**: one clock in UTC ms; each aircraft renders only inside its `[startUTC, endUTC]`; per-aircraft `videoTime = (utc − startUTC)/1000` seeks each `<video>` and queries telemetry. Global range = union of all aircraft. - **Camera modes**: free-fly (WASD/arrows + OrbitControls) and *follow* (camera flown to sensor position, looking along sensor forward, camera vFOV matched to sensor FOV, roll applied so the HUD is level); optional *look-at-frame-center*. Fly-into on H / double-click. - **Catalog**: `incidents.json` — recursive folder tree, aircraft leaves = `{id,name,tailNumber,video,telemetry,startUTC,duration}`; renderer-agnostic by construction.
## 2. What TaosEngine gives us (as discovered) Source of truth: the dist is **minified, hashed bundles** (`geo_osm_projection-B6PjROcy.js` + `geo_scene`, `terrain`, `geo_feature`, `tile_cache`, `engine`, `math`, …). There is **no published README / API doc / unminified source reachable from here** (GitHub repo 404s anonymously; not web-indexed). Everything below is **inferred from minified identifiers and the sample's own usage**, so treat names as approximate until confirmed against real source. **Engine / app shell** - `A.run()` drives the frame loop (engine owns the loop; not a hand-rolled `requestAnimationFrame`). - `A.scene.add(node)` / `scene.remove`, `scene.clear`. - Effects pass config: `re({ sky:{kind:'atmosphere', useMultiscatterLut:true}, ao:'ssao', bloom:true, projectors:true, shadow:{light:()=>sun} })(A)` — **`projectors:true` is the flag that enables projector rendering.** **Geo scene + terrain** - `new GeoScene(device, frame.atLonLat(lon, lat, elev))` — scene anchored at an origin lon/lat/elev (note **lon,lat order**). - `scene.addRasterDemTerrain(demSource, { imagery }).tileset` — AWS Terrarium raster-DEM + an imagery layer (Esri/OSM/CARTO/Stadia URL templates built in). Also `add3DTiles(url,opts)`, `addVectorTiles`, `addStatic(drawable)`. - `scene.heightAt(lon, lat)` — terrain elevation query (analog of the viewer's raycast-against-tiles ground probe). `waterAt(lon,lat)`. - `scene.update({camera/frustum})` returns `{opaque, shadowCasters, water, copyrights, triangles, …}`. `setBudget(bytes, tris)`, `cacheStats()`. - Coordinate helpers on the frame: `atLonLat(lon,lat,elev)`, `worldFromEcefPoint(ecef)`, `worldFromEcefDir(dir)`, `originEcef`. So the **ECEF↔local-world math the viewer hand-rolls in `SceneManagerMISB` is built in.** **Camera + fly controls** - `node.addComponent(createPerspective(fovDeg, near, far, aspect))`. - `FlyControls.create({ yaw, pitch, speed, sensitivity })`; `.attach(canvas)` + a bind helper. (WASD/Space/Shift + drag-look, per the prototype's on-screen text.) **Projector (the panosphere analog — central finding)** - A scene node + a `Projector` component: `let H = new Node({name}); let U = H.addComponent(new Projector());` - **Pose**: `H.setPosition(...ecefOrWorld)`; **orientation** `H.rotation = lookRotation(target.sub(pos))` (quaternion from a look direction — i.e. point the projector at a target). - **Texture**: `U.source = { kind:'texture', texture: textureFromImage(device, bitmap, w, h, label) }`; uploaded via `copyExternalImageToTexture`, `rgba8unorm`. Image obtained with `createImageBitmap(file, {colorSpaceConversion:'none'})`. - **Properties**: `blend` (`alpha`|`add`|`multiply`), `opacity` (0–1), `edgeFalloff` (e.g. .04), `near`, `far`, `fovY`, `aspect`, `shape` (`rect`/ortho **or** `perspective`), `orthoWidth`/`orthoHeight` (ortho footprint), `lit` (bool). - Added to scene via `scene.add(H)`; rendered because `projectors:true`. ### The key conceptual match A TaosEngine **`Projector` with `shape:'perspective'`, `fovY`+`aspect` from sensor FOV, positioned at the sensor and `lookRotation`'d at the frame-center, casting the FMV frame onto the terrain** is a *closer* model of the real sensor than the viewer's panosphere. The viewer paints video on a sphere patch *floating at the aircraft* (a HUD-in-the-sky); TaosEngine's projector paints it **onto the ground** like a real slide projector / shadow-map. Both are valid; the projector is arguably the more correct "where the camera is actually looking" rendering and is what the prototype already demonstrates. We should decide (open question Q5) whether to (a) reproduce the floating panosphere look, (b) adopt the ground-projection look, or (c) offer both.
## 3. Component → TaosEngine mapping | MISB viewer component | TaosEngine concept | Confidence | Notes | |---|---|---|---| | 3D-tiles globe / terrain (`SceneManagerMISB.initializeTerrain`) | `GeoScene` + `addRasterDemTerrain({imagery})` (or `add3DTiles` for photoreal) | High | Prototype proves terrain+imagery. Photoreal 3D-tiles path (`add3DTiles`) exists but unproven by a sample. | | geodetic↔local ECEF math (hand-rolled) | `atLonLat`, `worldFromEcefPoint/Dir`, `originEcef` | High | Engine-native; drop the hand-rolled ECEF code. | | ground-height probe (raycast vs tiles) | `scene.heightAt(lon,lat)` | High | Cleaner than raycasting. | | Panosphere (video-textured FOV patch) | `Projector` (`shape:'perspective'`, `fovY`/`aspect`, `near`/`far`, `blend`, `opacity`, `edgeFalloff`) | **Med** | Geometry/pose path clear; **video texture path is the gap (Q1).** | | panosphere pose from telemetry | `H.setPosition(sensorECEF)` + `H.rotation = lookRotation(frameCenter − sensor)` | High | `lookRotation` at frame-center replaces the manual yaw/pitch/roll matrix in `Panosphere`/`updateCamera`. Frame-center is in the telemetry. | | sensor FOV → patch size | `U.fovY`, `U.aspect = hFOV/vFOV ratio`, `U.far = slantRange*k` | Med | Map MISB hFOV/vFOV onto projector `fovY`+`aspect`. | | live MP4 video as texture (`THREE.VideoTexture`) | **UNKNOWN** — only `textureFromImage` found | **Low** | See Q1. Likely needs per-frame `copyExternalImageToTexture(videoFrame)` or `importExternalTexture`. Biggest risk. | | free-fly camera + OrbitControls | `createPerspective` + `FlyControls.create({yaw,pitch,speed,sensitivity})` | High | Prototype proves fly. Orbit-style controls unconfirmed (Q4). | | follow camera (pose = sensor, vFOV matched, roll) | set camera node `position`+`rotation` from telemetry each frame; set perspective `fovY` from sensor | Med | Same math as `SceneManagerMISB.updateCamera`, expressed on a TaosEngine camera node. Need to confirm we can imperatively drive the camera while FlyControls is attached (detach/re-attach? Q4). | | fly-into / fly-to (`flyToSelectedAircraft`, tween) | manual camera tween, or a TaosEngine camera-goto if one exists | Med | No camera-animation helper seen; likely hand-rolled lerp of position+rotation over ms. | | master UTC timeline + multi-aircraft sync | **app-level, engine-agnostic** — reuse viewer's logic verbatim | High | `TimelineControl`, `AircraftManager`, UTC↔videoTime, union range — all DOM/JS, port as-is. | | `<video>` seek per aircraft per frame | same `<video>` elements; only the *texture upload* differs | Med | Keep the hidden `<video>` machinery; change how its pixels reach the GPU (Q1). | | telemetry parse (ST 0601, 2 dialects → normalized sample) | **reuse `MISBTelemetryParser.js` unchanged** | High | Pure JS, renderer-agnostic. Zero TaosEngine coupling. | | catalog `incidents.json` + `LayersPanel` tree | reuse unchanged | High | Renderer-agnostic by construction. | | telemetry/cursor HUD panels | reuse DOM unchanged | High | | | selection ring, click-to-select raycast | TaosEngine picking / a `addStatic` ring drawable | Low | Picking API not yet discovered. Could keep a DOM overlay or skip for v1. | **Reuse boundary (important):** roughly everything in Layers A–C of the data-formats doc — catalog, telemetry parsing, normalized sample contract, UTC time model, timeline/aircraft orchestration — is **engine-independent and ports verbatim.** The TaosEngine work is confined to the *rendering layer*: `SceneManagerMISB` → a `TaosSceneManager`, and `Panosphere` → a `TaosProjector` (or projector-per-aircraft). That is the whole port surface.
## 4. Smallest first build step (step-0) **Static single projector at one MISB pose, image texture, on terrain — no video, no timeline, no multi-aircraft.** 1. Start from the prototype. Keep its GeoScene + raster-DEM terrain + fly controls + atmosphere/bloom, but **re-anchor the scene origin to a real MISB clip's first sample** (e.g. Palisades `N57B`: lat 34.0558, lon −118.5041) instead of the Grand Canyon. 2. Load that clip's telemetry with the unmodified `MISBTelemetryParser`; take sample[0]. 3. Create one `Projector`: position at the sensor ECEF (`atLonLat(lon,lat,altMSL)`), `rotation = lookRotation(frameCenter − sensor)`, `shape:'perspective'`, `fovY = vFOV`, `aspect = hFOV/vFOV`, `far ≈ slantRange*1.5`, `blend:'alpha'`, `opacity:1`. 4. Texture = **a single still frame of the MP4** (capture one `createImageBitmap` from the `<video>` at currentTime, or a pre-extracted JPEG) — *was* a hedge against the video-texture unknown. **Update 2026-06-10 (§D): Q1 is resolved — you may go straight to live video via `source:{kind:'video', video, backend:'copy'}`. The still-frame remains a fine even-smaller smoke test, but is no longer required to de-risk anything.** 5. Verify the projected frame lands on the terrain at roughly the right ground footprint (frame-center ± target-width). Confirm via CDP, not headless screenshot (WASM/WebGPU + workers — same caveat as the DuckDB skill note). **Acceptance:** one MISB video frame visibly projected onto terrain at the correct geographic spot, viewable by flying the camera. This proves the terrain-anchor, the ECEF pose, and the projector-FOV math before any video/timeline complexity. **Then, in order:** step-1 live video texture (resolve Q1) → step-2 follow camera from telemetry → step-3 single-aircraft timeline scrub → step-4 multi-aircraft + master UTC timeline (port `AircraftManager`/`TimelineControl`) → step-5 selection/HUD/fly-into parity.
## 5. Open questions / unknowns (need resolution before/at build) - **Q1 (~~blocking, highest risk~~ → RESOLVED 2026-06-10, see "API confirmation pass" §D): Live video texture.** Only `textureFromImage`/`createImageBitmap` (still image) was found in the dist *on 2026-06-09*. **Now confirmed:** the `projector_test` sample uses `source:{kind:'video', video:<HTMLVideoElement>, backend:'copy'}` and the core `ProjectorFeature.resolveVideo` does `copyExternalImageToTexture(videoEl)` per frame internally. Native video path exists; no manual pump needed. `backend:'external'` (`importExternalTexture` zero-copy) is declared-but-unimplemented (warns, degrades to copy). No longer the feasibility determinant. - **Q2: Projector vs panosphere look.** Adopt ground-projection (projector-native, prototype-proven, arguably more correct), reproduce the floating panosphere (more faithful to current viewer, but no obvious TaosEngine primitive for it), or both? Affects the whole rendering design. (See §2 "key conceptual match".) - **Q3 (RESOLVED 2026-06-10, see §C): Photoreal basemap.** ~~Does `add3DTiles` consume Google/Cesium-Ion tilesets?~~ **Yes** — `geo_photo` does `scene.add3DTiles(resolveIonAsset(IonAssets.googlePhotorealistic, ionToken))` (faithful Google-3D-Tiles look, needs a Cesium Ion token). Token-free fallback = `addVectorTiles` (OSM 3D buildings) + `addRasterDemTerrain` (proven by `geo_osm_buildings`). Two basemap paths, both sample-confirmed. - **Q4: Camera control under follow.** Can we imperatively set a camera node's position/rotation/fovY every frame while `FlyControls` is attached (detach on follow, re-attach on free?), and is there an orbit-style control or only fly? Needed for follow + look-at-frame-center modes. - **Q5: Multiple simultaneous projectors.** The viewer shows several aircraft at once. Does `projectors:true` support N projectors in one scene, and at what perf cost (each is effectively a shadow-map-like pass)? `setBudget` exists; projector count budget unknown. - **Q6: API stability / real names.** All identifiers here are reverse-engineered from minified code (`u`, `i`, `re`, `f.create`, etc.). We need the real public API — unminified source, a types/`.d.ts`, or a non-minified sample — before writing maintainable code. Without it, the build is brittle to bundle re-hashing. - **Q7: Picking/selection.** No picking API discovered; how to click-select an aircraft and draw a selection ring. May defer to a DOM overlay for v1. - **Q8: WebGPU availability.** TaosEngine is WebGPU-first; confirm the target deployment browsers have WebGPU (fallback to WebGL backend? unknown). The current viewer is WebGL/three and broadly compatible.
## 6. What ports verbatim (no TaosEngine work) `MISBTelemetryParser.js`, `incidents.json` + `LayersPanel` tree walk, the normalized-sample contract, the UTC/videoTime time model, `TimelineControl`, `AircraftManager` orchestration, telemetry/cursor HUD DOM, CORS-`anonymous` `<video>` setup. The port is **only** the rendering layer (`SceneManagerMISB` → Taos scene/camera; `Panosphere` → Taos `Projector`).
*All identifiers attributed to TaosEngine are inferred from minified `dist/assets/*.js` bundles and the `geo_osm_projection` sample's usage; not confirmed against published source (none reachable as of 2026-06-09). Re-verify against real source before relying on exact names — see Q6.*
# API confirmation pass (2026-06-10) **What changed since the 2026-06-09 reverse-engineering:** the upstream repo (`github.com/brendan-duncan/TaosEngine`) is private (404 via web, git, and API) and ships **no `.js.map` sourcemaps** — so true source is still unreachable. BUT the public dist ships a **full samples catalog** (`https://brendan-duncan.github.io/TaosEngine/dist/samples/`, ~50 samples), and crucially the **per-sample app bundles are shipped UN-minified** (real identifiers: `lookRotation`, `textureFromCanvas`, `PhotoOverlay`, `AnchorManipulator`, `resolveVideo`, etc.). The core engine bundle (`audio_listener-*.js`, 1.4 MB) is minified but greppable. This pass reads three samples as primary sources and greps the core for the Projector internals. **Sources read this pass (all `…github.io/TaosEngine/dist/`):** - `samples/projector_test.html` → `assets/projector_test-CA6LR1Sz.js` (7.7 KB, **un-minified**) — *the* Projector reference, in isolation, with a **live video projector**. - `samples/geo_photo.html` → `assets/geo_photo-DIjyNd41.js` (32 KB, **un-minified**) — posed-photo placement on **Google Photorealistic 3D Tiles** (Cesium Ion). - `samples/geo_osm_buildings.html` → `assets/geo_osm_buildings-DL2unv6m.js` (16 KB, **un-minified**) — **OSM vector 3D buildings** + raster-DEM terrain (no Ion token). - `assets/audio_listener-DDBuzL6Q…` core bundle (minified) — grepped for `ProjectorFeature.resolveVideo`, `copyExternalImageToTexture`, `importExternalTexture`. - `assets/geo_scene-syzd8aWA.js`, `assets/terrain-D-THt1nw.js` — grepped for scene method names. > The exact symbols below are **confirmed-from-sample** (copied from un-minified sample source) unless marked *still-unknown* or *core-grep*.
## A. Projector API — CONFIRMED (from `projector_test`) The Projector is a **component** added to a scene `Node`. Pattern (real names from the sample): ```js const proj = new Node({ name:'SlideProjector' }); // Node, not the guessed `s`/`Entity` proj.setPosition(0, 14, 14); proj.rotation = lookRotation(new Vec3(0,-14,-14)); // quaternion from a look-dir const p = proj.addComponent(new Projector()); // Projector ctor takes NO args p.source = { kind:'texture', texture: someGpuTexture }; // OR a video / atlas source — see below p.shape = 'perspective'; // 'perspective' | 'rect' p.focalLength = 35; // mm ← perspective via PHOTOGRAPHIC focal length (NEW — see delta) // p.fovY = 60; // degrees ← ALT perspective control (also valid; pick one) p.aspect = 1; p.far = 60; p.opacity = 1; // can exceed 1 for 'add' blend (sample uses 1.2) p.blend = 'alpha'; // 'alpha' | 'add' | 'multiply' p.edgeFalloff = 0.2; // 0..0.5 p.crop = [x, y, w, h]; // normalized inset crop/zoom (NEW) p.lit = false; // paint as albedo vs emissive overlay scene.add(proj); // rect/ortho variant: p.shape = 'rect'; p.orthoWidth = 7; p.orthoHeight = 7; // decal/gobo footprint ``` **Deltas vs the 2026-06-09 guess:** | Plan guess (§2) | Confirmed | Status | |---|---|---| | `new Projector()` component, `addComponent` | ✅ exactly — ctor takes no args; all config via property assignment after | **confirmed** | | `H.setPosition(...)`, `H.rotation = lookRotation(dir)` | ✅ exactly | **confirmed** | | `shape:'perspective'`/`'rect'` | ✅ | **confirmed** | | `fovY` for perspective size | ✅ valid, BUT primary control is **`focalLength` (mm)** — photographic. `fovY` also works (gobo projector uses it). | **confirmed + NEW** | | `aspect`, `near`, `far`, `opacity`, `blend`, `edgeFalloff` | ✅ all real | **confirmed** | | `orthoWidth`/`orthoHeight` (rect) | ✅ | **confirmed** | | `lit` (bool) | ✅ | **confirmed** | | `source = {kind:'texture', texture}` | ✅ | **confirmed** | | — (not guessed) | **`crop:[x,y,w,h]`** normalized crop/zoom | **NEW** | | — (not guessed) | **`source` can be `{kind:'video', video, backend}`** or an atlas handle (`atlas.add(...)`) | **NEW — resolves Q1** | | `lookRotation` is engine-native | ✗ — it is **defined locally in the sample**, NOT exported by core. We must **port the ~6-line helper** (it builds a quat from `Vec3.FORWARD` → dir). | **corrected** | | texture upload via `copyExternalImageToTexture`, `rgba8unorm` | ✅ exactly (`textureFromCanvas` helper in the sample does this) | **confirmed** | `projectors:true` is confirmed as the render-preset flag (`renderPreset(u({ …, projectors:true }))`), and the sample runs **3 projectors + a video projector + up to 48 atlas projectors simultaneously** → partially answers Q5 (N projectors are clearly supported; an **atlas** path exists for many-small-projectors).
## B. Scene / camera / controls — CONFIRMED (from `geo_photo` + `geo_osm_buildings`) - **App shell:** `const app = await Engine.create({ canvas, contextOptions:{ enableErrorHandling:true, reversedZ:true } })`; `app.ctx` (→ `.device`, `.width`, `.height`, `.fps`, `.elapsedTime`); `app.scene.add/remove`; `app.beforeFrame(cb)` / `app.afterFrame(cb)` / `app.run()`; `app.addFeatureBefore(feature, name)` / `app.getFeature(name)`. **Render preset is a function applied to the app:** `renderPreset({ sky, ao, lighting, projectors, … })(app)` — e.g. `({sky:{kind:'atmosphere',useMultiscatterLut:true}, ao:'gtao'})(app)` or `({sky:{kind:'color',color:[…]}, ao:'ssao', lighting:{disableAerial:true}})`. (The plan's `re({…})(A)` shape was right; real ao values are `'ssao'`/`'gtao'`.) - **GeoScene:** `new GeoScene(device, frame.atLonLat(lon, lat, elev), { budgetBytes })` — **confirms lon,lat,elev order** and a `{budgetBytes}` option. `frame.rebase(lon, lat, elev)` re-anchors an existing scene (confirmed in `geo_osm_buildings` location picker). - **Frame (ECEF) helpers — confirmed:** `frame.atLonLat(lon,lat,elev)`, `frame.worldFromEcefPoint(ecef)`, `frame.worldFromEcefDir(dir)`, `frame.ecefFromWorldPoint({x,y,z})`, and `frame.east/up/north` basis vectors. A free function `ecefToGeodetic(ecef) → {lonRad, latRad, height}` and `geodeticToEcef(lonDeg,latDeg,h)` exist (used as `v(...)` / `C(...)`). **The hand-rolled ECEF math in `SceneManagerMISB` is fully replaceable.** - **Terrain + basemaps (resolves Q3 — see §C):** `scene.addRasterDemTerrain(demSource, { imagery }).tileset`; `scene.add3DTiles(tilesetUrl, opts) → { tileset, ready }` (`tileset.maxSSE`, `tileset.maxScreenSpaceError`); `scene.addVectorTiles(source, { style, debug, wallInsetMeters }).tileset`; `scene.remove(tileset)`. `scene.heightAt(lonRad, latRad)` **(note: RADIANS, not degrees)** → confirmed in `geo_osm_buildings` `pickGround`. `scene.update({ cameraEcef, frustum, screenHeight, fovY, cull, dt, draw, shadows }) → { opaque[], shadowCasters[], … }` (richer than the §2 guess). - **Camera:** `node.addComponent(PerspectiveCamera.createPerspective(fovDeg, near, far, aspect))`; the returned component has **mutable `.fov`, `.near`, `.far`** (set every frame in geo_photo) and `.localToWorld()`, `.projectionMatrix()`, `.viewProjectionMatrix()`, `.inverseViewProjectionMatrix()`. So **driving the camera imperatively per-frame is normal** (geo_photo sets `cam.fov` each frame). - **FlyControls:** `const fly = FlyControls.create({ yaw, pitch, speed, sensitivity, pointerLock? })`; `fly.attach(canvas)`; a **bind helper** `bindControls(canvas, fly)` (the `u(canvas, fly)` / `l(u, j)` call); `fly.update(cameraNode, dt)` each frame; **mutable `fly.yaw` / `fly.pitch`**; gamepad hooks (`fly.inputForward/inputStrafe/inputUp/inputDown`, `fly.applyLookDelta(dx,dy)`). → **partially resolves Q4:** you drive the camera node's `setPosition` + the camera component's `.fov` directly; FlyControls reads/writes the same node, so *follow* = stop calling `fly.update` (or overwrite node pose after it) and set pose from telemetry. No separate orbit control seen (still only fly).
## C. Q3 — RESOLVED: TWO basemap paths, both confirmed by a sample 1. **Photoreal (Google Photorealistic 3D Tiles via Cesium Ion)** — confirmed in `geo_photo`: ```js const url = await resolveIonAsset(IonAssets.googlePhotorealistic, ionToken); const { tileset, ready } = scene.add3DTiles(url); tileset.maxSSE = 16; ``` So **`add3DTiles` DOES consume Google/Cesium-Ion tilesets**, exactly like `3d-tiles-renderer`. It needs a **Cesium Ion token** (geo_photo ships a default Ion JWT in source and reads a localStorage/`?token=` override). **This is the closest match to the current MISB viewer's look.** 2. **OSM vector 3D buildings + raster-DEM terrain (no token)** — confirmed in `geo_osm_buildings`: ```js const terrain = scene.addRasterDemTerrain(awsTerrarium, { imagery: esriImagery }).tileset; const buildings = scene.addVectorTiles(osmSource, { style, wallInsetMeters }).tileset; // extruded footprints ``` Built-in/custom MVT sources + OpenFreeMap styles (Liberty/Positron/Bright/Dark); per-building footprint picking by Alt-click. No Ion token. Lighter, fully open, but stylized (not photoreal). **Verdict:** the photoreal path is available and is the faithful reproduction of the MISB viewer's Google-3D-Tiles globe; the OSM path is the token-free fallback. **Q3 closed.**
## D. Q1 — RESOLVED: native video projector exists (engine-internal copy-per-frame) **Verdict: TaosEngine HAS a real video-texture path. No manual canvas-reupload needed — the engine does the per-frame upload for you.** Evidence (verbatim tokens): - **`projector_test` (un-minified) uses it directly:** ```js const v = canvas.captureStream(30); // or any MediaStream const videoEl = document.createElement('video'); videoEl.muted = true; videoEl.autoplay = true; videoEl.loop = true; videoEl.playsInline = true; videoEl.srcObject = v; videoEl.play(); const i = projNode.addComponent(new Projector()); i.source = { kind:'video', video: videoEl, backend:'copy' }; // ← the video source shape i.shape='perspective'; i.fovY=50; i.far=40; i.blend='add'; i.opacity=1.2; ``` - **Core bundle `ProjectorFeature.resolveVideo` (minified, grepped) — the implementation:** ```js resolveVideo(e,t){ e.backend==='external' && !this._warnedExternal && (console.warn('[ProjectorFeature] video backend "external" is not yet implemented; using "copy".'), this._warnedExternal=true); let n=e.video, r=n.videoWidth, i=n.videoHeight; return n.readyState<2 || r===0 || i===0 ? null : ( (!e.texture || e.texture.width!==r || e.texture.height!==i) && ( e.texture?.destroy(), e.texture=this._device.createTexture({label:'ProjectorVideo',format:'rgba8unorm', size:{width:r,height:i}, usage:TEXTURE_BINDING|COPY_DST|RENDER_ATTACHMENT})), this._device.queue.copyExternalImageToTexture({source:n},{texture:e.texture},{width:r,height:i}), {view:this.textureView(e.texture), crop:t}) } ``` **Reading of the evidence:** - `source.kind === 'video'` with `source.video = <HTMLVideoElement>` is a **first-class projector source**. - **`backend:'copy'`** (and the default) → engine **`copyExternalImageToTexture(videoEl)` into an `rgba8unorm` texture on every projector update** — i.e. the canvas/video-reupload fallback the plan feared is **already implemented internally, automatically, per frame.** It guards on `video.readyState>=2` and lazily resizes the texture to `videoWidth×videoHeight`. - **`backend:'external'`** (the WebGPU `importExternalTexture` zero-copy path) is **declared but NOT yet implemented** — it warns and silently degrades to `'copy'`. (`importExternalTexture` IS present in the core, but used for IBL cubemaps, not yet the projector.) **Consequence for the build:** Q1 is no longer a risk. Feed each aircraft's `<video>` straight into a Projector `source:{kind:'video', video, backend:'copy'}`; the engine re-uploads each frame. We do **not** write our own `requestVideoFrameCallback`/canvas pump. (If perf ever demands zero-copy, `backend:'external'` is the upstream's intended path — currently a no-op upgrade.) **This also obviates the §4 step-0 "use a still frame" hedge — we can go straight to live video** (still-frame remains a fine even-smaller smoke test, but is no longer necessary to de-risk Q1).
## E. Updated status of the open questions - **Q1 (was blocking) → RESOLVED.** Native `source:{kind:'video', video, backend:'copy'}`; engine does per-frame `copyExternalImageToTexture`. Evidence: `projector_test` usage + core `resolveVideo`. No fallback engineering needed. - **Q2 (projector vs panosphere look) → still a design choice**, but now better informed: `geo_photo` shows a **third option** — a world-anchored **textured quad** (`PhotoGeoFeature` builds a model matrix from ECEF center/right/up/normal and draws a quad). So we have (a) ground-projection Projector, (b) floating panosphere, (c) anchored billboard-quad. Projector remains the most sensor-correct. Decision still open (Stephen). - **Q3 (basemap) → RESOLVED.** `add3DTiles` consumes Google Photorealistic 3D Tiles via Cesium Ion (faithful look, needs Ion token) **or** `addVectorTiles` OSM 3D buildings + `addRasterDemTerrain` (token-free fallback). Both sample-proven. - **Q4 (camera under follow) → MOSTLY RESOLVED.** Camera node pose + camera-component `.fov/.near/.far` are imperatively mutable every frame (geo_photo does exactly this); FlyControls shares the same node, so *follow* = drive pose from telemetry instead of (or after) `fly.update`. No orbit control exists — only fly; orbit would be hand-rolled. *Still-unknown:* clean detach/re-attach ergonomics (likely just gate the `fly.update` call). - **Q5 (N projectors) → MOSTLY RESOLVED.** `projector_test` runs 3 + video + up to 48 atlas projectors at once; `projectors:true` supports many. An **atlas** projector path exists for many-small. *Still-unknown:* exact perf budget for ~N full-FOV video projectors (one per aircraft) — measure during build. - **Q6 (real API names) → IMPROVED, not fully closed.** The **app-level** API is now confirmed from un-minified sample source (high confidence: `Engine.create`, `GeoScene`, `Projector`, `FlyControls.create`, `createPerspective`, `add3DTiles`/`addVectorTiles`/`addRasterDemTerrain`, `frame.*`). The **core/engine** symbols are still minified (no `.d.ts`, repo private, no sourcemaps). For a maintainable build, pin to a **specific dist hash** and re-grep on upgrade; brittle to re-hashing remains the risk. Asking Brendan for source/types is still the clean fix. - **Q7 (picking/selection) → PARTIALLY ANSWERED.** No general engine picker, but `geo_osm_buildings` shows a **ground-pick pattern**: unproject screen ray via `camera.inverseViewProjectionMatrix()`, march/bisect against `scene.heightAt` to find the ground hit (`pickGround`). We can reuse that to click-select an aircraft's ground point; selection ring can be an `addStatic` drawable or a DOM overlay. *Still-unknown:* picking a floating projector/aircraft node directly. - **Q8 (WebGPU availability) → unchanged.** Still WebGPU-first; `Engine.create({contextOptions:{enableErrorHandling, reversedZ}})` is WebGPU. No WebGL fallback observed in samples. Confirm target browsers, or keep the existing three.js viewer for broad compat.
## F. Step-0 — now fully specifiable from public material The §4 step-0 (one projector at a MISB pose onto terrain) is **fully specified from public samples** — every primitive it needs is now confirmed-from-sample: - scene anchor: `new GeoScene(device, frame.atLonLat(lon,lat,elev), {budgetBytes})` ✅ - terrain: `addRasterDemTerrain(dem,{imagery})` ✅ (or `add3DTiles` photoreal ✅) - projector: `new Projector()` + `setPosition` + `rotation=lookRotation(frameCenter−sensor)` + `shape:'perspective'` + (`focalLength` mm **or** `fovY`) + `aspect` + `far` + `blend` + `opacity` ✅ - texture: still frame via `copyExternalImageToTexture` **or** go straight to live video via `source:{kind:'video',video,backend:'copy'}` ✅ - camera + fly: `createPerspective` + `FlyControls.create` ✅ - **must port locally:** the ~6-line `lookRotation(dir)` helper (not exported by core). **Only remaining true unknowns** (none block step-0): exact perf at N video projectors (Q5 tail), clean FlyControls detach for follow (Q4 tail), direct node picking (Q7 tail), WebGPU deployment target (Q8), and the Q2 look decision (Stephen's call). Step-0 can be built today from public material alone.
## References (bead cross-links) - Bead: Incident Cataloging · [canonical](https://redfish.acequia.io/guerin/.agents/c38c1239-bfd3-44dd-8d97-1a0aa39ac8da/)