**Artifact** from Bead: Devops · [canonical source](https://redfish.acequia.io/guerin/.agents/64be6d29-d133-4ade-9dce-f62701003e37/2026-06-09/artifacts/devops-coas.md) · session 2026-06-09 · discussion: Talk: Devops
**Companion to** [devops-plan.md](devops-plan.md). The plan defines *what* (Phases 0–5) and recommends one sequence. This document offers **three distinct strategies for how to actually start**, so the choice is explicit rather than implied. Each is a complete path; they differ in which bet they make first. Pick one; they are mutually exclusive as *opening moves* (they reconverge by Phase 3–4). > COA = a complete, committed opening strategy, judged on a common frame (below). All three assume Phase 0 substrate primitives (`webdav-context`, `beads`) already exist.
## The decision being made The plan's phases are independently shippable, so the real question is **what to prove first** — and that forces three of the four open questions immediately: - **Q1 home of record** — stay on `simtable.acequia.io/dev/ai-team/` or migrate to `redfish`. - **Build axis** — *depth* (one project, whole loop) vs *breadth* (the reusable policy machine) vs *reach* (the multi-domain promotion spine). - **When to spend owner-run software** — Phase 1 needs none; Phases 2/5 need a Service Worker / a daemon. Each COA answers these differently. That's what makes them distinct rather than three flavors of "do Phase 1."
## COA A — Walking skeleton on one real project *(value-first, depth)* **The bet:** the file-state-machine discipline (immutable artifacts + pointer-promotion + ETag concurrency) is the thing most likely to be wrong in practice, so prove it end-to-end on something real before generalizing. **Sequence** 1. Pick **one existing live project** as the pilot (candidate: the `storm-event` app, or `geo.camera`, or the santafe catalog work — something with a real deploy and a willing owner). 2. Lay it out per Phase 1: `repo/ pipelines/ runs/ releases/ channels/{dev,stage,prod}.json`. 3. Wire the full loop **manually + in-browser**: run → record `runs/<id>/`, cut an immutable `releases/<id>/`, promote by editing `channels/prod.json`, roll back by rewriting the pointer. ETag + `If-Match` on every state write. 4. Add only a **thin taste of Phase 2**: one Service Worker that *gates the prod pointer write* (fresh-ETag + a hand-signed token), nothing more. No attestation, no observe, no multi-domain. **Defers:** attestation chain (Ph3), push/`observe` (Ph2 active field), multi-domain (Ph4), governance (Ph5). **Pros:** fastest time-to-first-real-promotion (days, not weeks); validates the one discipline everything depends on; tiny blast radius; almost no owner-run software. **Cons:** proves the *least novel* part of the thesis (single-domain pull DevOps exists elsewhere); the pilot's quirks may bias the general design; "leveraging domains" is untested. **First milestone (≈1–2 wks):** the pilot can promote dev→prod and roll back, fully auditable in the browser Network tab, with the prod write refused when the SW gate fails. **Forces:** Q1 (pick the pilot's home now) · Q2 (one SW for the pilot — per-project scope, decide later if shared).
## COA B — Reusable policy machine first *(safety-first, breadth)* **The bet:** the durable value is "who can push to prod" being a property of the *substrate*, not a per-project convention — so build the Service Worker policy monitor + attestation as a shared library before onboarding many projects, and migrate the home of record to `redfish` up front so the substrate has a clean tier. **Sequence** 1. **Migrate home of record to `redfish`** (answers Q1 yes) — clean separation of substrate from the `ai-team` prototype. 2. Build the **Phase 2 SW policy monitor** as a *mountable library* (per-origin, per the tier-isolation argument in Q2): intercepts `fetch()`, free writes pass on ETag, control writes fail closed without a valid signed attestation. 3. Build the **Phase 3 attestation chain** (`manifest → results → attestation`) with in-tab compute (WebContainers/esbuild-wasm) producing the evidence the SW verifies. 4. *Then* onboard 2–3 projects onto the finished guardrails (Phase 1 layout each). **Defers:** the active field/`observe` (can stay polling a while), multi-domain promotion (Ph4), governance/mayordomo (Ph5). **Pros:** every later project inherits enforcement; gets the security model right while the system is small and cheap to change; settles Q1/Q2/Q3 deliberately. **Cons:** slowest to first *visible* value (weeks of framework before a promotion anyone sees); risk of building guardrails ahead of demand — the plan itself warns against spending cost before traffic justifies it; a shared SW library is a non-trivial design commitment to make before real usage informs it. **First milestone (≈3–4 wks):** a reference project where prod promotion is *cryptographically* impossible without a passing, signed, in-tab-produced attestation — reusable by the next project with no new code. **Forces:** Q1 (migrate to redfish) · Q2 (per-origin SW) · Q3 (attestation signer / trust-root vs token-granting split) — all three answered early, by necessity.
## COA C — Multi-domain promotion spine first *(thesis-first, reach)* **The bet:** the genuinely novel, defensible claim is "*leveraging domains*" — cross-domain promotion as TPC + pointer flip, with the route declared to an owner-run mayordomo. Lead with that thin, because it's the part no off-the-shelf DevOps does, and the pieces already half-exist (geo.camera tier-subdomains; the `agentic-route-flip-plan`; cross-server COPY/MOVE in nephele). **Sequence** 1. Stand up **two tier domains** as separate origins (e.g. `nightly.<x>` and `<x>`), per the geo.camera pattern. 2. Build Phase 1 **minimally** on the dev domain — just enough `releases/` + `channels/` to have an immutable artifact worth promoting. 3. Wire **cross-domain promotion**: server-to-server `COPY` (CERN TPC, already proven on nephele) of a release dev→prod, then a pointer flip. 4. Wire the **Phase 5 ingress-as-mayordomo** thinly: the agent PUTs a `hey-you-its-me/1` route-declaration envelope into a mailbox; an owner-run Caddy-reloading daemon (per `agentic-route-flip-plan`) enacts it. LLM declares; ordinary software binds the socket (EULA-clean). **Defers:** rich attestation (Ph3 — start with a hand-signed token), the active field, the full policy ladder, prorrata accounting. **Pros:** proves the unique Acequia thesis earliest; produces the most compelling demo (a release flips across domains and the public route follows, all by file writes); exercises TPC + mayordomo, the parts the prototype has never reached. **Cons:** most moving parts at once (two origins, TPC, a daemon, capability tokens); depends on owner-run software from day one; thinnest safety story early (a hand-signed token guarding a *prod domain* flip is a lot of authority on a little proof). Highest ambition, highest risk. **First milestone (≈3–5 wks):** an authorized agent promotes a dev-domain release to a prod domain as a TPC + pointer flip, and the public ingress route follows — no agent ever holding a socket. **Forces:** Q1 (multi-domain ⇒ redfish-tier layout) · Q3 (need at least a minimal signer to guard the flip) · Q4 (push transport matters once two origins must stay consistent).
## Comparison | Axis | A — Walking skeleton | B — Policy machine | C — Multi-domain spine | |---|---|---|---| | First bet | the file-state discipline | substrate-level enforcement | the "leverage domains" thesis | | Time to first visible value | **days** | weeks | weeks | | Proves the *novel* claim | low | medium | **high** | | Blast radius if wrong | **tiny** (one project) | medium (framework) | large (prod domains) | | Owner-run software needed | minimal (1 SW gate) | SW library | SW + Caddy-reload daemon | | Safety maturity at milestone 1 | thin | **strong** | thin (token only) | | Demand-justified (plan's caution) | **yes** | partly speculative | speculative | | Reversibility | **high** | medium | lower | | Open Qs forced | Q1, Q2 | Q1, Q2, Q3 | Q1, Q3, Q4 | All three **reconverge**: A and C both grow *into* B's policy machine by Phase 3; B grows *into* C's multi-domain reach by Phase 4. The choice is only about **what to de-risk first**.
## Recommendation **Lead with COA A, with one borrowed element from C.** Ship the walking skeleton on a real pilot first — it de-risks the one discipline everything depends on (immutable artifacts + pointer-promotion), at the lowest cost, and it's the only option that is fully *demand-justified* today (the plan's own closing caution). Borrow C's instinct by choosing a pilot that *already has a tier subdomain* (e.g. geo.camera's nightly/beta/prod), so the eventual jump to multi-domain promotion is a short step, not a re-architecture. Then invest in B's policy machine **once there is real promotion traffic to protect** — building guardrails around observed behavior, not ahead of it. Defer C's full mayordomo spine to when more than one domain genuinely needs coordinated promotion. This is a recommendation, not a decision — the bet you want to make first is yours. If the priority is a *demo of the unique thesis* over a hardened internal loop, C is the honest choice; if it's getting the security model right while the system is small, B is. **Decision needed:** which opening COA — and, bundled into it, the answer to Q1 (home of record).
## COA D — mount real git in the browser *(substrate axis, added 2026-06-10)* A, B, C all assume the **bespoke file-state machine** (hand-rolled `releases/` + `channels/` pointers + a custom attestation chain). COA D reopens a question the upstream prototype already named and deferred. From `simtable.acequia.io/dev/ai-team/acequia-devops/shared/`: - *"Not replacing Git — browser-native operational state fabric ... with **optional Git interop at the edges**."* - The explicit open trade-off: *"**Snapshot manifests vs. Git history** — line-level merge, or is file-level enough?"* So the prototype built **git's model without git** (immutable artifacts + mutable pointers + signed attestations = commits + refs + signed tags) and leaned toward snapshot-manifests for merge simplicity. COA D takes the other fork: **The bet:** the durable substrate is **real git, run client-side** (`isomorphic-git` pure-JS, or `wasm-git`/libgit2) over the WebDAV surface (or OPFS, pushed to nephele over git's dumb-HTTP/WebDAV transport). The file-state machine is not re-derived; it is *mounted*. **What it buys** (beyond the bespoke machine): GitHub + git-tooling **interoperability** (the substrate stops being an island); real **history / diff / merge / blame**; **signed commits/tags = the attestations** (signed with the `#sovereign-webdav` key, so provenance is git-native); still **in-tab + ambient-auth + no server** (the browser authors, git semantics enforce, the SW still gates "who can push to prod"). **Cost:** line-level merge complexity (the exact thing the prototype flagged), and `isomorphic-git` perf/size in-tab. **Relation to A/B/C:** D is an *orthogonal substrate choice*, not a fourth "what to prove first." It can underlie any of them — A's walking skeleton could commit to real git from day one; B's policy machine gates git refs instead of JSON pointers. Given how much of the architecture is git re-derived, mounting git likely wins for the same reason we kept **nephele** over a custom WebDAV server and **STAC** over a bespoke catalog: don't reinvent a battle-tested content-addressed/signed/versioned substrate. **Decision needed (D):** bespoke snapshot-manifest state machine, or mount real git in-browser? See [#main-context/git-and-learning-from-repos.md](https://redfish.acequia.io/guerin/.agents/c38c1239-bfd3-44dd-8d97-1a0aa39ac8da/2026-06-10/notes/git-and-learning-from-repos.md).
## References (bead cross-links) - Bead: Incident Cataloging · [canonical](https://redfish.acequia.io/guerin/.agents/c38c1239-bfd3-44dd-8d97-1a0aa39ac8da/)