62 lines
2.4 KiB
Markdown
62 lines
2.4 KiB
Markdown
# ECP-0070: Relay-Native CAS Archival + NixOS Auto-Archive Service
|
|
|
|
## Summary
|
|
|
|
Add a first-party archival path for MoQ relay streams:
|
|
|
|
1. `ec-node wt-archive`:
|
|
- Subscribe to a relay broadcast over WebTransport/MoQ.
|
|
- Persist each received group payload as a content-addressed object (BLAKE3).
|
|
- Append per-track JSONL index records ("manifests") that reference CAS object hashes/paths.
|
|
|
|
2. Extend the NixOS `services.every-channel.ec-node` module with `archive.*` options:
|
|
- Poll `every.channel` public stream directory.
|
|
- Auto-start one `wt-archive` worker per discovered broadcast.
|
|
- Store large CAS blobs and manifest indices in separately configurable roots.
|
|
|
|
## Motivation
|
|
|
|
We need durable, low-duplication TV archive storage on dedicated storage hosts.
|
|
|
|
- Relay ingestion makes archive nodes location-agnostic.
|
|
- CAS enables deduplication and integrity-friendly storage semantics.
|
|
- Separate manifest/index storage allows faster metadata operations and simpler backup policy.
|
|
- Nix-managed host config keeps archival behavior immutable and repeatable.
|
|
|
|
## Decision
|
|
|
|
- Introduce `ec-node wt-archive` as the minimal relay subscriber for archival.
|
|
- Use BLAKE3 object IDs and a sharded filesystem layout (`<cas-root>/objects/blake3/aa/<hash>.bin`).
|
|
- Write append-only JSONL records per `broadcast/track` under a configurable manifest directory.
|
|
- Add `services.every-channel.ec-node.archive.*` in the Nix module to auto-discover and archive public streams.
|
|
|
|
Initial default archived tracks target current relay publisher output:
|
|
- `catalog.json`
|
|
- `init.mp4`
|
|
- `video0.m4s`
|
|
- `audio0.m4s`
|
|
|
|
## Consequences
|
|
|
|
Pros:
|
|
- Immediate archival path without requiring browser components.
|
|
- Works with current Cloudflare relay + WebTransport deployment.
|
|
- Fully declarative deployment via existing Nix input wiring.
|
|
|
|
Tradeoffs:
|
|
- First version archives raw group payloads and does not yet normalize across track schema variants.
|
|
- Discovery source is the web public stream list (not full control-topic gossip ingestion).
|
|
- Per-broadcast workers are process-based and best-effort supervised.
|
|
|
|
## Rollout
|
|
|
|
1. Ship `wt-archive` command in `ec-node`.
|
|
2. Extend Nix module with `archive.*`.
|
|
3. Enable on storage host(s) with:
|
|
- CAS root on large ZFS pool.
|
|
- manifest root on separate filesystem/dataset.
|
|
|
|
## Reversibility
|
|
|
|
- Disable `services.every-channel.ec-node.archive.enable`.
|
|
- Existing CAS/manifests remain inert on disk and can be pruned independently.
|