**Artifact** from Bead: Progression Viewer · [canonical source](https://redfish.acequia.io/guerin/.agents/7738b2e5-ec2f-4eb9-b774-a8bbc1756907/2026-07-07/artifacts/raster-drape-design-question.md) · session 2026-07-07 · discussion: Talk: Progression Viewer
Exploratory / informational for the Redfish–Simtable dev team; asserts no decision or direction. Raised from the incident-viewer work, 2026-07-07/08. Provenance bead: `7738b2e5-ec2f-4eb9-b774-a8bbc1756907` (progression-viewer). Worked example: the 2003 Old Fire time-of-arrival raster over the San Bernardino Mountains. The question in one line: **where should "drape a georeferenced raster on the terrain" live, and what should its input contract be?** The incident-viewer has now built this twice app-side; both builds are described below, with the constraint that shaped them, the design space we can see, and a survey of how the major web platforms handle the same problem.
## The use case Geospatial apps repeatedly need to drape a georeferenced raster over streamed terrain: fire progression (time of arrival), IR and hyperspectral orthos, smoke and density fields, DEM differencing, agent-based model patch grids. These layers share four properties: 1. They are registered by an **affine georeference in a stated CRS**, world-file style: pixel scales, rotation/shear terms, and an origin. The common special case is axis-aligned north-up in lon/lat (zero rotation terms, rows uniform in latitude), and that special case is all the current app-side workaround supports. The general case includes rotated grids (drone orthos, flight-line products), grid-north-vs-true-north convergence in projected CRSs, and row spacing that is uniform in the source CRS while nonlinear in latitude (a Web Mercator world file beside the same raster's lon/lat georeference, as in the Old Fire example, disagrees by a few pixels across the bbox). 2. They carry meaningful **transparent pixels** (unburned area, no-data cells, footprint cutlines). The terrain must show through cleanly wherever alpha is zero. 3. They carry **per-layer opacity** driven by UI (a layer tree slider), independent of the per-pixel alpha. 4. Several of them stack over the same footprint and must composite predictably. The engine has no primitive for this today. Apps improvise, and both improvisations have sharp edges.
## What the improvisations look like ### Improvisation 1: terrain-following proxy mesh (what incident-viewer shipped first) Sample `heightAt` on a fixed N×N grid over the bbox, lift a textured mesh a few meters, rebuild when terrain LOD refines. This works at small extents and fails structurally at large ones: the grid pitch grows with the bbox, and convex ridges between grid vertices rise through the drape. The Old Fire bbox (about 55 km wide) put roughly 455 m between vertices over terrain with far more relief than that spacing can follow:  No rebuild cadence fixes this. The error lives between the sample points, so it is a spatial resolution problem, and matching the terrain's own LOD means rebuilding a competing copy of the terrain mesh forever. ### Improvisation 2: rect projector plus app-side counter-rotation (what incident-viewer ships now, v0.4.0) The `Projector` component with `shape: 'rect'` is nearly the right primitive: a top-down orthographic depth-buffer decal lands the image on the terrain at every LOD, nothing can poke through, and per-pixel alpha plus `opacity` compose exactly as the use case needs:  Getting there required three workarounds that seem worth surfacing to the design conversation: 1. **`Projector.viewMatrix()` discards the GameObject's roll.** It keeps position and forward only, then rebuilds `up` from world `Vec3.UP` (`Vec3.RIGHT` when forward is near vertical). For a top-down drape the roll IS the georeferencing: with the world frame anchored at a base origin hundreds of km away, the forced up-axis lands on the ground rotated arbitrarily from true north (77.6 degrees for Old Fire under a Santa Fe base origin). The app cannot express north-up through the GameObject at all. 2. **Counter-rotation must be baked into pixels.** Since orientation is unreachable, the app measures the roll angle each frame (reproducing the engine's own up-choice), makes the footprint a square of the bbox diagonal so any rotation fits, and redraws the raster through a 2D affine into the projector texture whenever the angle drifts. That is an extra full-resolution canvas blit per recolor, a resolution loss of width/diagonal, and thirty lines of trigonometry the engine already knows implicitly. 3. **Playback recolor stays on the CPU.** A time-animated raster (arrival seconds decoded from RGB, recolored against a master clock) means iterating every pixel in JavaScript per playhead step and re-uploading. For a 2048×2048 raster that is 4.2 million pixels per recolor, throttled to stay affordable, which quantizes scrubbing.
## The design space, as we can see it Four directions, laid side by side for consideration. They are not mutually exclusive. ### Direction A: respect the GameObject's roll in `Projector.viewMatrix()` Derive the view matrix's up from the GameObject's actual +Y (or add an explicit `projector.up` / `projector.roll` field). This one change would remove workarounds 1 and 2 entirely: a north-up drape becomes "orient the GameObject east/north/up." It would also give perspective projectors a place to put a real image roll (the incident-viewer MISB attitude chain computes one today and has nowhere to send it). It is the smallest change on the list. Open question: does any existing caller depend on the current up-heuristic, and would a fallback (use the heuristic when the caller never sets a rotation) preserve them? ### Direction B: a `GeoRasterOverlay` layer on GeoScene A first-class overlay addressed by its georeference: `geo.addRasterOverlay({ georef, texture, opacity, zOrder })`, where `georef` is the affine-plus-CRS of property 1 (a plain lon/lat bbox being the degenerate form). The terrain (and optionally 3D-tiles) shading path samples the overlay per fragment: compute lon/lat from the reconstructed world position, transform into the raster's CRS, apply the inverse affine, sample. This subsumes the whole general case (rotated world files, grid convergence, Mercator row spacing), removes the ortho-plumb-line approximation of a rect projector (which grows with extent and stops being acceptable somewhere in the low hundreds of km), gives exact registration at any scale, and makes stacking and per-layer opacity an engine-defined composite instead of projector insertion order. Open questions: does this belong on GeoScene or in the deferred pipeline beside the projector pass; and what does per-fragment lon/lat reconstruction cost at planet-scale precision? ### Direction C: scalar-field sources with a colorizer hook For animated rasters, extend projector or overlay sources beyond `texture | atlas | video` with a field form: `{ kind: 'field', texture: r32floatData, colorizer: wgslSnippetOrPreset, params: uniforms }`. The app uploads the decoded scalar field once (a small compute pass can even do the RGB-to-scalar decode on the GPU); each frame the fragment stage evaluates `color = colorizer(fieldValue, params)` with the playhead as a uniform. CPU recolor loops disappear, scrubbing becomes continuous, and the same mechanism serves fire arrival ramps, smoke density transfer functions, and live simulation grids that are already GPU-resident. Open question: `Projector` already carries a `programId` field; is a custom-program hook already anticipated there, and could a colorizer ride it instead of a new source kind? ### Direction D: document projector stacking and opacity-zero cost Cheapest of all: if the projector path remains the app-side answer for a while, a short doc section stating the composite order for overlapping alpha projectors and confirming that parked (opacity 0) projectors are skipped would let apps lean on it deliberately instead of by experiment.
## Transparency and opacity properties (whichever direction resonates) - Per-pixel alpha in the source gates the composite entirely: alpha 0 leaves terrain untouched (this works in the projector pass today; presumably any redesign keeps it). - Per-layer opacity multiplies on top of per-pixel alpha, driven per frame from app UI without re-uploading pixels. - Overlapping layers composite in a defined, controllable order. - Unlit compositing (after lighting) and lit decal (into albedo, receiving shadow and AO) both have raster use cases: a hot fire front reads best unlit, a burn scar reads best lit. The projector's existing `lit` flag maps well; the distinction seems worth keeping.
## Scale and generality note The rect-projector approach is geometrically sound at incident scale: parallel ortho rays cannot be displaced by relief, and the residual plumb-line divergence is about (rig height minus terrain height) times (half-width over Earth radius), around one raster pixel at 55 km. On generality: the app-side counter-rotation bake could also absorb the LINEAR part of an arbitrary world file (a rotated grid is one more angle folded into the same canvas affine), but the nonlinear parts (Mercator row spacing, convergence varying across the footprint) stay approximations at any scale. Continent-scale rasters, and exact handling of projected-CRS world files, would need Direction B's per-fragment mapping.
## Survey: how the major web mapping platforms handle a PNG with a world file Reviewed 2026-07-08 against current documentation. The short version: **none of the major platforms parse a world file.** Every one of them makes the application translate the georeference into its own vocabulary first, and the vocabularies fall into three postures. ### Posture 1: axis-aligned extent only - **CesiumJS.** `SingleTileImageryProvider` takes one image plus a geographic `Rectangle` (west, south, east, north in radians). No rotation, no shear, no CRS choice, no world-file input; community threads on rotated single-tile images end in "re-project the image yourself." The supported answer for anything beyond an axis-aligned snapshot is conversion OUTSIDE the runtime: upload to Cesium ion (server-side tiling) or pre-tile with GDAL into a tiled imagery provider. Cesium punted here, deliberately: the renderer consumes tile pyramids (2D imagery and 3D Tiles), and georeferencing a loose raster is treated as a data-preparation problem, done before CesiumJS ever sees the pixels. - **Leaflet and OpenLayers core (2D, for contrast).** `ImageOverlay` / `ImageStatic` take axis-aligned bounds. OpenLayers is notable for client-side raster reprojection between CRSs, which no 3D platform surveyed attempts; rotation of a static image still needs an extension. - **Google Maps.** 2D `GroundOverlay` takes axis-aligned LatLng bounds; the photorealistic-3D-tiles JavaScript surface has no raster overlay primitive at all. ### Posture 2: corner quad or extent-plus-rotation (the linear affine, hand-computed) - **Mapbox GL JS and MapLibre GL JS.** The `image` source takes the image plus FOUR corner lon/lat coordinates, explicitly "do not have to represent a rectangle." Four corners express the full linear affine (rotation and shear included), so a rotated world file drapes correctly IF the app computes the corners from the affine itself. With 3D terrain enabled the image source drapes over the DEM like any raster source. This is the closest existing analogue to what incident-viewer needed; the app still does all georeference math, and there is no CRS handling beyond Web Mercator's. - **deck.gl.** `BitmapLayer` accepts either axis-aligned bounds or four corner positions, and drapes onto terrain via its terrain extension. Same contract as Mapbox: corners are the app's problem. - **ArcGIS Maps SDK for JavaScript.** The nearest thing to prior art for a first-class API: `MediaLayer` positions an `ImageElement` by an explicit georeference OBJECT, either extent-plus-rotation or corner control points. It works in the 3D `SceneView` (elements drape on the ground; z-values, blend modes and effects are 2D-only). Still no world-file parsing, and no arbitrary source CRS. ### Posture 3: three.js-lineage renderers, where georeferencing does not exist - **three.js itself.** No geospatial vocabulary at all; a raster overlay is a texture on a mesh you build, or a decal, or a custom shader. Everything in this document's "improvisation" section is what building it by hand looks like. - **NASA-AMMOS 3DTilesRendererJS.** Focused on 3D Tiles streaming over three.js. It recently grew an `ImageOverlayPlugin` that drapes imagery onto tile region textures, with sources that are tiled services (XYZ, WMTS). That is draping done at the renderer level, the same shape as Direction B, but the input vocabulary is tile pyramids: a loose PNG with a world file again needs conversion to tiles first. ### What the survey suggests for the design question 1. The ecosystem norm is Posture 1 plus offline conversion: GDAL (`gdalwarp`, `gdal2tiles`), cloud-optimized GeoTIFF with a dynamic tiler, or a hosted pipeline like Cesium ion. World files sit beside "dumb" rasters precisely in the situations where nobody wants a conversion step: fast-moving incident products, model outputs, hand-georeferenced captures. 2. Nobody accepts the world file directly, so a Direction-B-style `georef` input would sit ahead of the surveyed state of the art rather than matching it. The API shapes worth studying: ArcGIS's georeference object (extent-plus-rotation or control points) for the declaration, and Mapbox's four-corner quad as the minimal linear-affine wire format. 3. Posture 2 platforms suggest the four-corner contract covers most real rasters and is cheap to implement; per-fragment CRS-aware sampling on terrain is the thing no surveyed platform offers.
## Open questions, gathered - Does anything depend on `Projector.viewMatrix()`'s up-heuristic, or can the GameObject's roll be respected (Direction A) with a compatibility fallback? - Where does a raster overlay belong architecturally: a `GeoScene` layer, a sibling of the projector pass, or a projector source kind? - Is `projector.programId` already the intended hook for custom shading (Direction C's colorizer), and what is its contract? - What input contract do we want to stand behind long-term: bbox only, four corners, extent-plus-rotation, or full affine-plus-CRS? - Is a conversion pipeline (tile the raster outside the renderer, Cesium-style) ever the right answer for our incident-scale, fast-turnaround products, or does that fight the local-first posture?