**Note** from Bead: Computer Vision Harness · [canonical source](https://redfish.acequia.io/guerin/.agents/b5221927-24ee-40f6-bb8b-94b3b9a88783/2026-06-29/notes/00-cv-harness-orientation.md) · session 2026-06-29 · discussion: Talk: Computer Vision Harness
> Read-first note for this bead. The app is `repo/index.html`; this note is the *why* and *how it hangs > together* so a denovo agent can extend it without re-deriving the design.
## The ask (verbatim intent) > "write an html harness for selecting different computer vision models using transformers.js. start with > segment anything, depth and scene description, and yolo. also optical flow and brief feature tracking. > allow selecting front and backfacing cameras and be responsive for laptop and mobile."
## The two families (the load-bearing design split) The six pills are **not** all transformers. They split into two families with different runtimes: 1. **Transformer models (transformers.js, run on a captured still).** Slow (seconds), download weights on first use, run once per **run** press (or per tap for Segment): - **Segment** → Segment Anything, point-prompted. transformers.js `SamModel` + `AutoProcessor` + `RawImage`, repo `Xenova/slimsam-77-uniform` (SlimSAM, small enough for the browser). Tap → one `input_points` prompt → `post_process_masks` → draw the best mask by IoU. - **Depth** → `pipeline('depth-estimation', 'Xenova/depth-anything-small-hf')`; output `.depth` is a `RawImage` drawn as a grayscale map (near = bright). - **Describe** → `pipeline('image-to-text', 'Xenova/vit-gpt2-image-captioning')`; one caption string. - **YOLO** → `pipeline('object-detection', 'Xenova/yolos-tiny')`; boxes + labels. Note: "YOLO" here is **YOLOS-tiny**, the canonical transformers.js detector (DETR-style), not Ultralytics YOLO. Flagged in the UI sub-label and the README so nobody is misled. 2. **Classical CV (pure JS, run live per animation frame on a downscaled copy).** No weights, no network: - **Optical flow** → sparse **Lucas-Kanade** on a 16px grid: per point solve the 2×2 structure-tensor system `[Σgx² Σgxgy; Σgxgy Σgy²]·v = -[Σgx·It; Σgy·It]` over a window, draw the motion vector. - **BRIEF tracks** → **Shi-Tomasi** min-eigenvalue corners + NMS → 256-bit **BRIEF** binary descriptors (fixed deterministic sampling pattern) → frame-to-frame **Hamming** match within a local search radius → draw the track segments. The split matters because the UI adapts to it: classical modes show a **LIVE** badge and hide the run button (they run continuously in the rAF loop); transformer modes show a **STILL** badge and a **run** button (Segment instead listens for a tap).
## Architecture map (where to look in `index.html`) - **Model registry** — the `MODELS` array. Add a pill by adding an entry: `kind` is `sam | pipe | classical`; `live` drives the badge/run-button; `task` + `repo` drive the pipeline. This is the one place to extend. - **Camera management** — `startCamera()` (facingMode + deviceId), `populateCameras()` (enumerateDevices), `flip` button toggles `environment`/`user`, front preview + overlay are mirrored (`.mirror`) and the tap coordinate is un-mirrored before SAM. - **Canvas discipline** — three canvases: the visible `#overlay` (display-sized, draws results); the hidden `#capture` (native-resolution still for transformer inference); the hidden `#compute` (240px-wide downscale for classical CV). Always map model-space → overlay-space via `fitScale` / explicit factors. - **transformers.js loading** — `env.allowLocalModels = false`; pipelines are lazy + cached by `task|repo|device` in `pipeCache`; a `progress_callback` streams the download % into the status line. - **Backend toggle** — `device` is `webgpu` when `navigator.gpu` exists else `wasm`; the backend button flips it and clears caches so models reload on the chosen backend. - **Main loop** — one `requestAnimationFrame` loop; only ticks the classical detectors when a classical mode is selected and the video has frames.
## Decisions / trade-offs - **Single file, no build.** Matches the cheap-end-of-the-cognition-ladder posture: a parciante opens one URL. transformers.js is pinned to `@3.8.1` from jsdelivr (latest stable 3.x; has `SamModel`, `AutoProcessor`, `RawImage.fromCanvas`, and all three pipelines). v4.x exists but the major was avoided for stability; bumping is a one-line change if wanted. - **Default mode = optical flow.** A live classical mode means something happens the instant the camera is granted, with zero download — immediate feedback before any model fetch. - **Tiny/small checkpoints only.** Browser latency forces SlimSAM / depth-anything-small / vit-gpt2 / yolos-tiny. Larger checkpoints are a knob, not a default. - **Classical CV is illustrative.** LK and BRIEF here are honest, compact implementations, not tuned trackers (no pyramids, no sub-pixel refinement, no RANSAC). Good enough to *see* flow and tracks; a starting point if a real tracker is ever needed.
## Possible next moves (open, not decided) - [ ] Field-test on a phone (front/back flip, WebGPU on mobile Chrome, model-download UX on cellular). - [ ] Add a **pose/landmark** model (MediaPipe-style) and/or **zero-shot detection** (OWL-ViT) as more pills. - [ ] SAM: cache image embeddings so repeated taps on the same frame don't re-encode. - [ ] Optical flow: pyramidal LK for large motion; BRIEF: add RANSAC homography to reject outlier tracks. - [ ] A "freeze frame" so transformer models run on a held still while you inspect (currently they grab the live frame at run time). - [ ] Fold the best pill(s) into geo.camera as a model-trial side panel.
## Pointers - App source: [`repo/index.html`](../../repo/index.html); deploy: [https://redfish.acequia.io/guerin/apps/cv-harness/](https://redfish.acequia.io/guerin/apps/cv-harness/). - transformers.js: [github.com/huggingface/transformers.js](https://github.com/huggingface/transformers.js). - Deploy precedent: agentscript apps bead [`b6fcda63`](https://redfish.acequia.io/guerin/.agents/b6fcda63-c171-4293-9d99-75297f4205bb/about.md) (git in the deploy dir, anonymous read).
## References (bead cross-links) - Bead: Ants In Taos · [canonical](https://redfish.acequia.io/guerin/.agents/b6fcda63-c171-4293-9d99-75297f4205bb/)