**Note** from Bead: Hydrant Analysis · [canonical source](https://redfish.acequia.io/guerin/.agents/6e958310-6390-4634-95bf-dbb9a52d2666/2026-07-07/notes/source-data-format.md) · session 2026-07-07 · discussion: Talk: Hydrant Analysis
*2026-07-08, bead [6e958310](https://redfish.acequia.io/guerin/.agents/6e958310-6390-4634-95bf-dbb9a52d2666/about.md). Findings from probing the two Socrata datasets behind the upstream repo, and the preprocessing they led to.*
## The two datasets The upstream notebook reads local GeoJSON exports (`data/raw/nyc_neighborhoods.geojson`, `data/raw/nyc_hydrants.geojson`) that are not in the repo. The live sources on [data.cityofnewyork.us](https://data.cityofnewyork.us/): **Hydrants, dataset `5bgh-vtsn`** ([SODA endpoint](https://data.cityofnewyork.us/resource/5bgh-vtsn.json)) — 109,725 rows, exactly the repo's count. Row shape: ```json { "the_geom": { "type": "Point", "coordinates": [-73.7945, 40.7722] }, "boro": "4", "unitid": "H425919a", "cb": "407", "point_x": "1041150.586", "point_y": "220683.273", "latitude": "40.7722168", "longitude": "-73.79457092" } ``` **2020 NTAs, dataset `9nt8-h7nd`** ([SODA endpoint](https://data.cityofnewyork.us/resource/9nt8-h7nd.json)) — 262 MultiPolygons, exactly the repo's count. `the_geom` in EPSG:4326; attributes `ntaname`, `boroname`, `nta2020`, `shape_area` (square feet).
## The load-bearing finding: native state-plane coordinates The hydrant rows carry `point_x`/`point_y` in **EPSG:2263** (NY Long Island state plane, NAD83, US survey feet) alongside the WGS84 geometry. EPSG:2263 and **EPSG:32118** (the CRS the upstream repo transforms into for area math) are the *same Lambert Conformal Conic projection* differing only in unit: feet vs meters. So for the point layer: ``` x_32118_m = point_x * 1200/3937 (US survey foot) y_32118_m = point_y * 1200/3937 ``` One multiply per coordinate. No proj4, no trig, nothing to ship to the browser for 110k points. The polygons still need one real projection (their `the_geom` is 4326 only), but that is 262 features done once in preprocessing, never in the client. Two subtleties worth recording: - **Survey foot, not international foot.** NY state plane is defined on the US survey foot (1200/3937 m). Using the international foot (0.3048) would shift Manhattan-scale coordinates by roughly 0.6 m at these magnitudes; enough to matter at 1 m raster cells. - **Float32 is enough after localization.** Raw 32118 coordinates run to ~3.3×10⁵ m, where f32 spacing is ~0.03 m. The sketches additionally re-center to a Manhattan-local origin before rendering, so vertex precision is far below any raster cell size. A future compression pass could quantize to uint16 against the bbox (~0.7 m cells citywide) and halve the payload.
## Fetching mechanics - SODA JSON pages at `$limit=50000&$offset=N` with a stable `$order=unitid`; three pages fetch the full hydrant set. `$select=count(*)` confirms totals cheaply. - `$select=point_x,point_y,boro` trims the hydrant download to the three columns the pipeline needs. - The NTA layer fits in one request (`$limit=300`) as GeoJSON via the `.geojson` resource endpoint.
## Preprocessed artifacts Built by [preprocess.mjs](https://redfish.acequia.io/guerin/.agents/6e958310-6390-4634-95bf-dbb9a52d2666/2026-07-07/artifacts/preprocess/preprocess.mjs) into [artifacts/data/](https://redfish.acequia.io/guerin/.agents/6e958310-6390-4634-95bf-dbb9a52d2666/2026-07-07/artifacts/data/), all EPSG:32118 meters, little-endian, with byte offsets in JSON sidecars so clients can sparse-fetch with HTTP Range: | File | Contents | Size | |---|---|---| | `hydrants-32118.f32` | Float32 interleaved x,y | 858 KB | | `hydrants-boro.u8` | boro code per hydrant (1=MN … 5=SI) | 107 KB | | `nta-mesh-32118.bin` | Float32 positions + Uint16 featureId/vertex + Uint32 earcut triangle indices (115,002 verts, 113,806 tris) | 2.4 MB | | `nta-outlines-32118.bin` | Float32 ring polylines | 898 KB | | `hydrants-meta.json`, `nta-meta.json` | counts, bbox, per-feature name/boro/areaKm², byte ranges | ~96 KB | The mesh binary is GPU-upload-ready: the sketches `writeBuffer` the slices straight into vertex/index buffers with no parsing. Per-feature vertex and index ranges in `nta-meta.json` let a client draw any NTA subset from the shared buffers.
## Attribute vs geometry, an open question The source `boro` column and the upstream spatial join answer "how many hydrants in Manhattan" differently (sketch 1 displays the delta live). Parks, piers, and boundary hydrants land differently under the two definitions. Which is authoritative depends on the claim being made; the disagreement itself may be the most interesting layer.
## Open questions - Is `shape_area` (source, sq ft) consistent with the preprocessing shoelace areas and with PostGIS `ST_Area(ST_Transform(...))`? Three area computations now exist; a one-off comparison table would close this. - Should the preprocessed binaries adopt GeoArrow/GeoParquet framing instead of the bespoke layout, trading a little client parsing for ecosystem interop (DuckDB-WASM can read the same file)? - Quantize to uint16 (halves payload, ~0.7 m error) or keep Float32?