**Note** from Bead: Wildfire Imagery Acquisition & Catalog Ops · [canonical source](https://redfish.acequia.io/guerin/.agents/b3461c66-1a49-4f62-8920-d6dcc9838942/2026-06-08/notes/watchduty-scrape-approach.md) · session 2026-06-08 · discussion: Talk: Wildfire Imagery Acquisition & Catalog Ops
> Sorrento fire incident: https://app.watchduty.org/i/101783 (WatchDuty `geo_event` id **101783**). > Goal: capture WatchDuty's incident state (reports timeline, evac zones, cameras, perimeter) into > `https://simtable.acequia.io/incidents/2026-CA-Sorrento/watch-duty/` alongside the AlertWest camera frames.
## Bottom line: I (Claude Code) could not crawl WatchDuty directly → CE workflow I tried to pull WatchDuty's API from this session (Claude Code, plain `curl`) and **could not** — every endpoint returns **406** (see below). I have no way to drive the authenticated SPA or replicate its headers from here. So the approach is **not** a direct crawl by me; it is an **in-browser workflow handed to CE (Claude Chrome Extension)**, which runs inside Stephen's real, authenticated WatchDuty session under ambient authority and can monitor the network + WebDAV PUT. The deliverable from this session is therefore the *workflow + prompt* ([ce-watchduty-sorrento-prompt.md](../artifacts/ce-watchduty-sorrento-prompt.md)), not the data — CE produces the data.
## Why not a direct scrape (tested 2026-06-08) WatchDuty's public API base is `https://api.watchduty.org/api/v1/` (confirmed by an older cached file `cache-watchduty-org--api--v1--cameras--realtime.json` in the 2025 Palisades folder). But every direct request now returns **406 Not Acceptable** — including the formerly-cached `cameras/realtime` — with and without `Accept: application/json` / browser UA: ``` [406] api.watchduty.org/api/v1/geo_events/101783/ [406] api.watchduty.org/api/v1/cameras/realtime ``` 406 is content-negotiation/WAF rejection of requests missing the app's own headers (likely a custom `platform`/version header or signed session), **not** an auth 401/403. Hammering it risks a block, so I stopped after two attempts. A headless scrape would have to reverse-engineer those headers; brittle.
## Decision: hand it to CE (Claude Chrome Extension) under ambient authority The right agent is **CE — the Claude Chrome Extension** (corrected 2026-06-08; I had wrongly said "CB"). CE runs in Stephen's actual Chrome, so it has **ambient authority**: it drives the authenticated `app.watchduty.org` session (the SPA's own fetches carry the required headers/cookies), **monitors the network**, and can **WebDAV PUT** the capture under that same ambient auth — no header reverse-engineering, no credentials handled by us. **Not CB.** CB (Claude Browser) runs from Claude Cloud with **no ambient session auth** — per the `9f65eca3` capability check it "cannot access auth-gated redfish.acequia.io paths… cannot PUT" and is handicapped fetching auth-gated URLs (fetch originates from Claude Cloud servers). CB can only explore the open web and write to `/mnt/user-data/outputs/` for CE to PUT. So WatchDuty (authenticated session + network monitor + WebDAV) is **CE-only**. Role set: **CE = Chrome Extension, scribe/PUT/ambient; CB = Browser, explorer/reader, cloud-sandboxed; CV = Claude Code/VSCode.** Precedent for the SandyFire capture: `watch-duty/Report.html` + `Report_files/` was a browser "Save Page As (complete)". We can do better than a flat page-save — capture the structured JSON the SPA loads (geo_event, reports, cameras) plus a readable report.
## Recon first — we don't actually know WatchDuty's surface Important correction (Stephen, 2026-06-08): **don't prescribe the endpoints — we don't know what WatchDuty exposes.** It has more than reports: **firefighting aircraft tracks (GeoJSON/KML)**, **user-submitted imagery**, evac zones, perimeter, and possibly **the AlertWildfire/AlertWest camera imagery itself — but maybe at a different sampling interval / historical depth than our direct grab.** Since the API is 406-gated to outside callers, the only way to learn the surface is to **watch the network from inside the app's session** (which CE, in-browser, can do). So round one is a **reconnaissance report**, not a capture: CE monitors all network requests, inventories every endpoint + content type + layer, PUTs the inventory to this bead's `uploads/`, and **pauses for Stephen to confirm scope** before bulk-downloading. Two high-value joins to surface in recon: 1. **cam-13500 overlap** — is our cam (UCSD_TDLLN_1) in WatchDuty's Sorrento camera set? 2. **camera-cadence diff** — at what interval/window does WatchDuty serve camera frames vs our direct AlertWest pull? If it holds frames we miss, it's a gap-filling second source. Capture destination (round two, after scope confirmed): `simtable.acequia.io/incidents/2026-CA-Sorrento/watch-duty/` — raw JSON per endpoint, `aircraft/` (GeoJSON/ KML), `user-media/` + index, `cameras/<camId>/` (with timestamps, to diff against `ops-alertcalifornia/`), evac/perimeter GeoJSON, optional `report.html`. The full recon→capture→self-document prompt is the artifact [ce-watchduty-sorrento-prompt.md](../artifacts/ce-watchduty-sorrento-prompt.md).
## Phase-0 RESULTS (CE ran it, 2026-06-08) — what we actually learned CE (bead [`e37a8d6e`](https://redfish.acequia.io/guerin/.agents/e37a8d6e-0637-407d-b629-5f77533a4178/about.md)) ran the recon and docked a full [network inventory](https://redfish.acequia.io/guerin/.agents/b3461c66-1a49-4f62-8920-d6dcc9838942/uploads/watchduty-network-inventory.md) into this bead's `uploads/`. It also captured `geo-event-101783.json`, `cameras-realtime.json`, `capture-summary.json` into `…/2026-CA-Sorrento/watch-duty/`. **Sorrento Fire 101783: 90 ac, 40% contained, forward progress stopped, all evacs lifted; lat 32.9101, lng −117.2136 (Sorrento Valley, San Diego).** **WatchDuty's surface (observed):** | Endpoint | Gives | Notes | |---|---|---| | `GET api.watchduty.org/api/v1/geo_events/101783/` | incident JSON | name, lat/lon, acreage, containment, evac status | | `GET …/cameras_gis/realtime` | camera **metadata** (az/fov/status) | **global**; ids = `base64("alertwest:{n}")`; **no image URLs** | | `maptiles.watchduty.org/fireperimeters_active` | perimeter | Mapbox vector tiles (MVT), not GeoJSON | | `maptiles.watchduty.org/evaczones…` | evac zones | MVT; Sorrento zones SDC-1444-B/C, SDC-1492-C (lifted) | | CloudFront `…/re-api/?hexList=` | H3 hex cells | polled ~5 s; meaning unknown | | weatherlayers.com | GFS/HRRR wind | external, global | | aircraft tracks | — | **MEMBERS-tier**, not triggered | | reports/updates | — | `/reports/` 404; API path unknown (UI shows them) | | user imagery | — | no media endpoint seen this session | **Two of my hypotheses — disconfirmed:** 1. **cam-13500 overlap: NO.** WatchDuty's camera list uses **AlertWest** ids (fire-area set CE saw: 1006, 1007, 11018–11021); our cam **13500 is AlertCalifornia** numbering and is not in it. (13500 may still physically see the fire — UCSD/Sorrento Valley are adjacent — WatchDuty just doesn't track it.) 2. **camera-cadence diff: N/A.** WatchDuty exposes **no camera frames or frame URLs** — only az/fov/status. There is nothing to diff; WatchDuty is not a frame source. **But a better integration surfaced — verified:** the AlertWest ids WatchDuty lists **are grabbable via OUR CDN.** `getCameraTimelapse cameraIds=1006` → 44 frames (`camName "Flea_Mtn_1"`). So WatchDuty's `cameras_gis/realtime` is effectively a **directory into the same `api.cdn.prod.alertwest.com` CDN** that `grab-alertwest.py` already uses. WatchDuty's value here is *which cameras*, not *the imagery*. **Caveat on CE's "fire-area" camera list:** `cameras_gis/realtime` is **global**, and 1006 = "Flea_Mtn_1" is almost certainly not near Sorrento Valley — CE likely listed the first entries, not a spatial filter. The correct way to pick fire-facing cameras is **roster proximity to the fire lat/lon** (32.9101, −117.2136) per `01-source-api.md` (filter `getCameraDataByLoc` by bounding box), then grab those with `grab-alertwest.py`. Note: ids 1006/1007/11018–21 **and** 13500 are all absent from our local SandyFire roster snapshot (it's region-filtered) — a fresh `getCameraDataByLoc` pull is needed to resolve geography. **Phase-1 candidates (for Stephen to scope):** 1. Pull a fresh AlertWest roster, find cameras within ~10 km of (32.9101, −117.2136), grab them for the incident window with `grab-alertwest.py` → `…/2026-CA-Sorrento/ops-alertcalifornia/<camId>/`. 2. Decide whether to fetch WatchDuty's MVT perimeter/evac tiles (need z/x/y GET, not just HEAD) for the authoritative fire outline + evac history. 3. Aircraft tracks need a MEMBERS session (CE couldn't trigger the layer) — defer unless Stephen has it. 4. Reports/updates: the API path is unknown (UI-only); would need CE to watch the network while opening the updates panel specifically.
## De-novo CE self-documentation (the part Stephen asked me to look up) Pattern from yesterday's CE/CB session (bead [`9f65eca3`](https://redfish.acequia.io/guerin/.agents/9f65eca3-86c7-4d2b-8980-197a510101c8/about.md), captured under bead `1057235a`'s `ce-cb-watch/`): **CE** — de novo, no memory — documents itself by **following the bead protocol manually** under ambient auth. It GETs `https://redfish.acequia.io/guerin/.agents/beads.md` (web-readable **with** session auth: 302→200), generates a GUID, and PUTs a full `about.md` (standard template) + `<date>/{notes,skills,chats,artifacts}` + a verbatim `chats/` log, cross-linking with absolute `https://` URIs. In that session CE was the scribe (PUT) and CB the explorer (web research, no PUT); CE relayed CB's findings into the bead. The ready-to-paste prompt that encodes the recon + capture + self-documentation is the artifact [ce-watchduty-sorrento-prompt.md](../artifacts/ce-watchduty-sorrento-prompt.md).
## References (bead cross-links) - Bead e37a8d6e · [canonical](https://redfish.acequia.io/guerin/.agents/e37a8d6e-0637-407d-b629-5f77533a4178/) (no page yet) - Bead 9f65eca3 · [canonical](https://redfish.acequia.io/guerin/.agents/9f65eca3-86c7-4d2b-8980-197a510101c8/) (no page yet)