every.channel: sanitized baseline

evolution/proposals/ECP-0022-swarm-availability-anti-junk.md

@@ -0,0 +1,114 @@
+# ECP-0022: swarm availability, signed manifests, anti-junk controls
+
+Status: Draft
+
+## Problem / context
+
+We need to scale distribution without turning every node into a dump pipe. The primary threat is denial via junk data: a peer can join and send invalid chunks that waste bandwidth and compute. We also need a model for multi-party signing and divergent encodes without inventing a full blockchain.
+
+## Decision
+
+Adopt a pull-based swarm model centered on signed manifests and content-addressed chunks:
+
+- Chunks are immutable and referenced by hash.
+- A manifest commits to a chunk set via a Merkle root and is signed by one or more identities.
+- Peers only serve chunks that match a manifest they recognize and that a requester explicitly asks for.
+- Junk is rejected by hash before decode, and peers are rate-limited and penalized for bad data.
+
+Multi-root signing is handled through **attestations**: multiple signers may publish distinct manifest roots for the same stream/time slice, and clients decide which roots are acceptable based on policy.
+
+## Details
+
+### Objects
+
+- `ChunkId`:
+  - `stream_id`
+  - `epoch_id` (time window / slice)
+  - `chunk_index`
+  - `chunk_hash` (blake3 of raw chunk bytes)
+- `ManifestBody`:
+  - `stream_id`
+  - `epoch_id`
+  - `chunk_duration_ms`
+  - `total_chunks`
+  - `chunk_start_index`
+  - `encoder_profile_id`
+  - `merkle_root` (over ordered chunk_hash list)
+  - `created_unix_ms`
+  - `metadata` (optional, e.g. channel title, guide hints)
+  - `chunk_hashes` (ordered list for now; proofs can come later)
+- `Manifest`:
+  - `body`
+  - `manifest_id` = `blake3(body)`
+  - `signatures`: list of `(signer_id, sig_over_manifest_id)`
+
+### Pull-only data flow
+
+- A peer first obtains a manifest (via gossip, direct share, or local catalog).
+- The peer requests specific chunks by `ChunkId`.
+- Providers return bytes **only** if they can supply exact hash bytes and have quota available.
+
+### Anti-junk measures
+
+- Validate `chunk_hash` before decode; invalid bytes are dropped.
+- Require the requester to supply `manifest_id` + `chunk_index` + expected `chunk_hash`.
+- Token-bucket quotas per peer:
+  - max bytes/sec
+  - max concurrent chunk responses
+  - max outstanding requests
+- Penalize peers:
+  - invalid hash → immediate disconnect + temporary ban
+  - repeated invalid requests → throttle or blacklist
+
+### Multi-root signing (not blockchain)
+
+Multiple manifests may exist for the same `stream_id + epoch_id`. We treat them as **parallel attestations**, not consensus. Clients choose a policy, e.g.:
+
+- accept a manifest if signed by `>=1` trusted key
+- or `>=N` signatures from a trust set
+- or “self + any one other”
+
+If two manifests conflict (different `merkle_root`), clients may:
+
+- select the one with the most trusted signatures
+- fall back to local preference
+- display “split availability” without forcing global consensus
+
+### Relay behavior
+
+Relays cache hot chunks by `ChunkId` and serve on request only. They do not push data to peers. Relays enforce the same quota and validation rules as peers.
+
+### MoQ transport notes (initial)
+
+- Track `chunks` carries object payloads as in ECP-0012.
+- Track `manifests` carries JSON-encoded `Manifest` entries, one per group.
+- For live streams we start with **per-chunk manifests** (epoch size = 1) so every chunk can be validated immediately. Multi-chunk epochs and Merkle proofs can replace this later.
+
+### Chunk membership proofs (incremental)
+
+To avoid sending full `chunk_hashes` forever, object metadata may include a Merkle branch proving that `chunk_hash` is included in the manifest's `merkle_root` for the epoch.
+
+- `ObjectMeta.chunk_proof`: list of sibling hashes from leaf to root.
+- Verification uses the chunk offset within the epoch: `offset = chunk_index - chunk_start_index`.
+
+## Alternatives considered
+
+- Push-only streams: rejected (amplifies junk and DDoS risk).
+- Global consensus / blockchain for manifests: rejected as overkill for now.
+- Accepting arbitrary chunks without hashes: rejected (no integrity).
+
+## Rollout / teardown plan
+
+1. Define `ManifestBody`, `Manifest`, and `ChunkId` in `ec-core`.
+2. Implement signing + verification in `ec-crypto` (age/ssh).
+3. Add manifest publication and request protocol in `ec-moq`.
+4. Enforce per-peer quotas and ban lists in `ec-iroh` transport layer.
+5. Update catalog gossip to advertise manifests, not raw chunks.
+
+Teardown: revert to direct MoQ stream subscriptions (current behavior) if manifest flow blocks progress.
+
+## Open questions
+
+- What default trust policy should the UI use for manifests?
+- Should relays require proof-of-work for public manifest announcements?
+- Do we embed encoder profile parameters directly in `encoder_profile_id` or reference a registry?