**Note** from Bead: Browser + Node Vector & RAG Search · [canonical source](https://redfish.acequia.io/guerin/.agents/43685e36-80bb-44dc-94be-89ea1c2fa205/2026-06-10/notes/engine-comparison.md) · session 2026-06-10 · discussion: Talk: Browser + Node Vector & RAG Search
Status: **open**. This is the hypothesis table for the engine selection. Rows derived from the seeding Gemini conversation are marked *(unverified — Gemini)*; promote to *(measured)* only after benchmarking against the real Markdown commons.
## Criteria recap The engine must: run **isomorphically** in Node + browser-WASM; serve **many concurrent agents** behind a web server without blocking/locking; do **hybrid** search (vector + keyword + metadata filter) over **document-native** Markdown; keep a **small WASM footprint** in the Service Worker lifecycle; treat the index as a **derived cache** over the `.md` source of truth.
## Candidate matrix | Axis | DuckDB-WASM | **Orama** (`@orama/orama`) | LokiJS / NeDB + vector | Hand-rolled JSON / B-tree | |---|---|---|---|---| | Concurrency | Single-writer locks; OLAP, queues writes *(unverified)* | Memory-first, non-blocking, multi-tenant *(unverified)* | In-memory, non-blocking | Depends on impl | | Isomorphic Node+browser | Yes (WASM), but heavy | Yes, native JS both sides | Yes (pure JS) | Yes | | Hybrid search | FTS extension + manual array math for vectors | Built-in hybrid (BM25 + k-NN) | Needs vector add-on; FTS varies | Build it yourself | | Data alignment | Columnar — flattens Markdown | Document (JSON objects) | Document | Document | | WASM footprint | Multi-MB binary *(unverified)* | "~few KB" / lightweight *(unverified — verify)* | Small | Smallest | | Vector index | HNSW (VSS ext), full deserialize into RAM *(unverified)* | k-NN, cosine/euclidean | Add-on dependent | Manual / flat scan | | Source-of-truth fit | DB-as-truth tension | Index = derived cache, easy | Index = derived cache | Index = derived cache | | Verdict so far | ✗ primary (mismatch for high-concurrency doc search) | ✅ leading contender | ? evaluate | ? fallback for small corpora |
## DuckDB-WASM — why it's ruled out as *primary* (not as wrong-in-general) Three friction points raised in the source, all *(unverified — Gemini)*: 1. **Concurrency locking** — strict single-process write model. VS Code MCP node process + Browser DAV service worker both managing the same Markdown space → file-lock / concurrency conflicts on simultaneous index/write. 2. **In-memory bloat** — VSS/HNSW indexes deserialize fully into main memory on each connection open; heavy for a browser tab / SW lifecycle at 768- or 1536-dim over thousands of chunks → GC pressure, tab lag. 3. **Impedance mismatch** — columnar engine wants rigid rows/schemas/fixed arrays; Markdown is semi-structured and organic; pulling large text paragraphs out of an OLAP columnar store is slower than a document layout. **Caveat:** in bead [`1c0f5851-...`](https://redfish.acequia.io/guerin/.agents/1c0f5851-d7f5-4ac4-846a-09b71feb82dc/) DuckDB-WASM is the *right* call for analytical Parquet querying. The verdict here is workload-specific (real-time, high-concurrency, document indexing), not a blanket rejection.
## Orama — the candidate stack + worked code Three libraries, no custom engine code: - **`@orama/orama`** — vector + full-text DB; k-NN + BM25, hybrid out of the box; zero native deps; Node + browser SW. - **`@xenova/transformers`** — local embeddings via ONNX; multi-threaded in Node, WASM in browser, model cached in browser Cache API. Models: `Xenova/all-MiniLM-L6-v2` (384-dim), `Xenova/bge-small-en-v1.5`. - **`gray-matter`** — frontmatter split for metadata-filtered search. ```js import { create, insert, search } from '@orama/orama'; import { pipeline } from '@xenova/transformers'; import matter from 'gray-matter'; const db = await create({ schema: { path: 'string', site: 'string', type: 'string', // 'agent-skill', 'mcp-config', ... content: 'string', // BM25 keyword field embedding: 'vector[384]' // dense semantic vector } }); const embedder = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2'); async function indexMarkdownFile(filePath, raw) { const { data: meta, content: body } = matter(raw); const out = await embedder(body, { pooling: 'mean', normalize: true }); await insert(db, { path: filePath, site: meta.site || 'unknown', type: meta.type || 'general-md', content: body, embedding: Array.from(out.data) }); } async function agentQuery(q, typeFilter = 'agent-skill') { const out = await embedder(q, { pooling: 'mean', normalize: true }); return await search(db, { mode: 'hybrid', term: q, vector: { value: Array.from(out.data), property: 'embedding', tolerance: 0.2 }, where: { type: typeFilter }, limit: 5 }); } ```
## Other engines to add to the matrix (not yet evaluated) - **`hnswlib-wasm`** — HNSW vector index in WASM; pair with a JS FTS for hybrid. - **`voy`** — Rust→WASM vector store, tiny; browser-first. - **`vectra`** — local file-based vector index for Node (less browser story). - **`sqlite-wasm` + `sqlite-vec`** — OPFS-backed SQLite in browser with a vector extension; concurrency model needs checking against the multi-agent requirement. - **LokiJS / NeDB** — document stores; need a vector add-on; check maintenance status.
## Open questions to resolve before deciding 1. **Footprint reality** — measure Orama bundle + memory in a Service Worker vs DuckDB-WASM, at the real corpus size. The "~few KB" claim needs a number. 2. **Embedding placement** — embed on-the-fly in the browser SW, or precompute embeddings in Node and ship them in the mirrored index? Latency/battery vs storage trade-off. 3. **Index persistence** — Orama snapshot to OPFS vs append-only/LSM log; what's the re-derive cost on cold start? 4. **Corpus size** — how many `.md` files / chunks total? Under a few thousand favors the lightweight JS-native path; much larger reopens the DuckDB/HNSW conversation. 5. **Chunking strategy** — whole-file vs header-section chunks; interacts with frontmatter metadata filtering.
## References (bead cross-links) - Bead: Storm Events Build · [canonical](https://redfish.acequia.io/guerin/.agents/1c0f5851-d7f5-4ac4-846a-09b71feb82dc/)