**Note** from Bead: Browser + Node Vector & RAG Search · [canonical source](https://redfish.acequia.io/guerin/.agents/43685e36-80bb-44dc-94be-89ea1c2fa205/2026-06-10/notes/00-orientation.md) · session 2026-06-10 · discussion: Talk: Browser + Node Vector & RAG Search
**Read this first.** This note encodes the frame, the candidates, and the selection criteria for the thread. The raw source is the captured Gemini conversation at [`../artifacts/gemini-share-ed5cf391cad7.md`](https://redfish.acequia.io/guerin/.agents/43685e36-80bb-44dc-94be-89ea1c2fa205/2026-06-10/artifacts/gemini-share-ed5cf391cad7.md). The structured comparison is in [`engine-comparison.md`](https://redfish.acequia.io/guerin/.agents/43685e36-80bb-44dc-94be-89ea1c2fa205/2026-06-10/notes/engine-comparison.md).
## The problem Stephen has a directory of all his websites, each mirrored online. Many sites carry `.md` files for agent-to-agent docs, agent **skills**, and **MCP** configs. He wants AI agents to **semantically search that Markdown commons** — "find my agent skills related to watershed modeling" — and pull the relevant chunks into context. Two runtimes must run the **same search**: 1. **Node.js** — local hard drive, e.g. a VS Code MCP server scanning the site directories via `fs`. Serves Claude/Gemini in the IDE. 2. **Browser (Browser DAV surface)** — the same files mirrored into **OPFS**, **IndexedDB**, and the **Service Worker Cache**. Agents inside the browser dev surface query in-tab. So the engine must be **isomorphic**: one codebase that compiles to Node *and* to WASM/browser, with the storage layer (`fs` vs OPFS/IndexedDB) behind an abstract adapter interface.
## The hard constraint that drives the whole decision The engine sits **behind a web server with many concurrent agents** querying and occasionally writing, while the browser surface simultaneously syncs the same Markdown space. The requirement, in Stephen's words, is "something that kind of acts like node that is able to handle multiple concurrent requests" — **asynchronous, non-blocking, multi-tenant** reads/writes that behave identically in a Node server and a Web/Service Worker. This is the criterion that **rules DuckDB-WASM out as the primary engine**: DuckDB is a single-process-writer OLAP engine. Under many concurrent agents it queues synchronously or throws file-lock errors. (Stephen is *already* on DuckDB-WASM — "duckwasm" — for other work, which is why the comparison exists: should he continue, or pivot?) > Note the contrast with bead [`1c0f5851-...`](https://redfish.acequia.io/guerin/.agents/1c0f5851-d7f5-4ac4-846a-09b71feb82dc/), where DuckDB-WASM is the *right* tool — analytical Parquet querying, single-reader, in-browser. Same engine, opposite verdict, because the workload (real-time high-concurrency document search) is different. Don't let that bead's success bias this decision.
## The leading contender: Orama **Orama** (`@orama/orama`) — 100% TypeScript, zero native C++ deps, compiles to a standalone JS bundle that runs in Node *and* in a browser Web/Service Worker. Built-in **hybrid search** (BM25 full-text + dense vector k-NN, cosine/euclidean). Document-native (stores Markdown as JSON objects, no columnar flattening). Memory-first with non-blocking writes flushed to disk/OPFS via snapshot or append-only log — no multi-process file locks. Vector-search docs Stephen flagged: https://docs.orama.com/docs/orama-js/search/vector-search The candidate **stack** the source converges on: 1. **`@orama/orama`** — the vector + full-text DB engine (k-NN + BM25, hybrid). 2. **`@xenova/transformers`** (Transformers.js; or `@huggingface/transformers`) — local embedding generation. ONNX Runtime wrapper; multi-threaded in Node, WASM in browser, model cached in the browser Cache API. Models named: `Xenova/all-MiniLM-L6-v2` (384-dim), `Xenova/bge-small-en-v1.5`. 3. **`gray-matter`** — YAML frontmatter parser, splits `type: agent-skill` / `site:` / `mcp_protocol: true` metadata from body for **metadata-filtered hybrid search** (filter before you match).
## The architecture (isomorphic "search daemon") A lightweight **Core Indexer** + two environment **Adapters**: - **Node side:** main thread accepts HTTP/WebSocket/MCP requests; hands queries to a **Worker Thread pool** (or `cluster`) that does the vector math, keeping the server thread free. Storage via `fs`. - **Browser side:** the same search logic runs in a **Service Worker / Web Worker**; reads OPFS via async `FileSystemSyncAccessHandle` inside a dedicated worker; the SW intercepts Browser DAV file requests. Storage via OPFS/IndexedDB. - **Sync layer:** source of truth is the **`.md` files themselves**, not the DB. The index is a derived, disposable cache kept alongside the files (snapshot in OPFS / on disk). Source suggests **LSM (log-structured merge) over OPFS** for the write path. Re-derivable from the Markdown at any time.
## Selection criteria (the axes of the comparison) 1. **Concurrency model** — non-blocking multi-tenant (✅ JS-native memory-first) vs single-writer locks (✗ DuckDB). 2. **Isomorphism** — same code Node + browser WASM, small footprint in the SW lifecycle. 3. **Hybrid search** — vector + keyword + **metadata filter** out of the box. 4. **Data alignment** — document-native vs columnar (Markdown is semi-structured/organic). 5. **WASM footprint** — KB-scale JS (Orama) vs multi-MB binary (DuckDB-WASM) in a browser tab/SW. 6. **Source-of-truth discipline** — index is a derived cache over the `.md` files; cheap to rebuild.
## What to do next (for the next animator) - **Verify, don't trust.** The source's claims (DuckDB HNSW full-deserialize-into-RAM, Orama "~few KB" footprint, concurrency behaviors) are **Gemini assertions** — turn them into measured benchmarks before they become decision-grade. Treat `engine-comparison.md` as a hypothesis table. - **Other engines to weigh** beyond Orama vs DuckDB-WASM: LokiJS / NeDB-style document stores with vector extensions; a pure JSON / B-tree index hand-rolled over the files; `hnswlib-wasm`; `voy` (Rust/WASM vector store); `vectra`; sqlite-wasm + sqlite-vec. Add rows as they're evaluated. - **Open question:** embedding cost/latency in the browser SW — is on-the-fly `all-MiniLM-L6-v2` fast enough, or must embeddings be precomputed in Node and shipped to the browser as part of the mirrored index? - **Output location:** comparison + recommendation stay in this bead's `notes/`. The eventual *implementation* (indexer, MCP server, SW search) moves to a deploying site path — NOT into this bead. - **Chat-log discipline:** write Stephen's verbatim prompts into `chats/` incrementally. - **Do not write** outside this bead's subdirectories.
## Anchor pointers - Source conversation (provenance): [`gemini-share-ed5cf391cad7.md`](https://redfish.acequia.io/guerin/.agents/43685e36-80bb-44dc-94be-89ea1c2fa205/2026-06-10/artifacts/gemini-share-ed5cf391cad7.md) - Contrast (DuckDB *fits* here): bead [`1c0f5851-...`](https://redfish.acequia.io/guerin/.agents/1c0f5851-d7f5-4ac4-846a-09b71feb82dc/) — DuckDB storm pipeline skill. - Orama vector search docs: https://docs.orama.com/docs/orama-js/search/vector-search - Browser storage substrate: `.ai/browser-hosting.md` (OPFS / IndexedDB / SW Cache).
## References (bead cross-links) - Bead: Storm Events Build · [canonical](https://redfish.acequia.io/guerin/.agents/1c0f5851-d7f5-4ac4-846a-09b71feb82dc/)