every.channel/evolution/proposals/ECP-0022-swarm-availability-anti-junk.md

ECP-0022: swarm availability, signed manifests, anti-junk controls

Status: Draft

Problem / context

We need to scale distribution without turning every node into a dump pipe. The primary threat is denial via junk data: a peer can join and send invalid chunks that waste bandwidth and compute. We also need a model for multi-party signing and divergent encodes without inventing a full blockchain.

Decision

Adopt a pull-based swarm model centered on signed manifests and content-addressed chunks:

• Chunks are immutable and referenced by hash.
• A manifest commits to a chunk set via a Merkle root and is signed by one or more identities.
• Peers only serve chunks that match a manifest they recognize and that a requester explicitly asks for.
• Junk is rejected by hash before decode, and peers are rate-limited and penalized for bad data.

Multi-root signing is handled through attestations: multiple signers may publish distinct manifest roots for the same stream/time slice, and clients decide which roots are acceptable based on policy.

Details

Objects

• ChunkId:
  • stream_id
  • epoch_id (time window / slice)
  • chunk_index
  • chunk_hash (blake3 of raw chunk bytes)
• ManifestBody:
  • stream_id
  • epoch_id
  • chunk_duration_ms
  • total_chunks
  • chunk_start_index
  • encoder_profile_id
  • merkle_root (over ordered chunk_hash list)
  • created_unix_ms
  • metadata (optional, e.g. channel title, guide hints)
  • chunk_hashes (ordered list for now; proofs can come later)
• Manifest:
  • body
  • manifest_id = blake3(body)
  • signatures: list of (signer_id, sig_over_manifest_id)

Pull-only data flow

• A peer first obtains a manifest (via gossip, direct share, or local catalog).
• The peer requests specific chunks by ChunkId.
• Providers return bytes only if they can supply exact hash bytes and have quota available.

Anti-junk measures

• Validate chunk_hash before decode; invalid bytes are dropped.
• Require the requester to supply manifest_id + chunk_index + expected chunk_hash.
• Token-bucket quotas per peer:
  • max bytes/sec
  • max concurrent chunk responses
  • max outstanding requests
• Penalize peers:
  • invalid hash → immediate disconnect + temporary ban
  • repeated invalid requests → throttle or blacklist

Multi-root signing (not blockchain)

Multiple manifests may exist for the same stream_id + epoch_id. We treat them as parallel attestations, not consensus. Clients choose a policy, e.g.:

• accept a manifest if signed by >=1 trusted key
• or >=N signatures from a trust set
• or “self + any one other”

If two manifests conflict (different merkle_root), clients may:

• select the one with the most trusted signatures
• fall back to local preference
• display “split availability” without forcing global consensus

Relay behavior

Relays cache hot chunks by ChunkId and serve on request only. They do not push data to peers. Relays enforce the same quota and validation rules as peers.

MoQ transport notes (initial)

• Track chunks carries object payloads as in ECP-0012.
• Track manifests carries JSON-encoded Manifest entries, one per group.
• For live streams we start with per-chunk manifests (epoch size = 1) so every chunk can be validated immediately. Multi-chunk epochs and Merkle proofs can replace this later.

Chunk membership proofs (incremental)

To avoid sending full chunk_hashes forever, object metadata may include a Merkle branch proving that chunk_hash is included in the manifest's merkle_root for the epoch.

• ObjectMeta.chunk_proof: list of sibling hashes from leaf to root.
• Verification uses the chunk offset within the epoch: offset = chunk_index - chunk_start_index.

Alternatives considered

• Push-only streams: rejected (amplifies junk and DDoS risk).
• Global consensus / blockchain for manifests: rejected as overkill for now.
• Accepting arbitrary chunks without hashes: rejected (no integrity).

Rollout / teardown plan

1. Define ManifestBody, Manifest, and ChunkId in ec-core.
2. Implement signing + verification in ec-crypto (age/ssh).
3. Add manifest publication and request protocol in ec-moq.
4. Enforce per-peer quotas and ban lists in ec-iroh transport layer.
5. Update catalog gossip to advertise manifests, not raw chunks.

Teardown: revert to direct MoQ stream subscriptions (current behavior) if manifest flow blocks progress.

Open questions

• What default trust policy should the UI use for manifests?
• Should relays require proof-of-work for public manifest announcements?
• Do we embed encoder profile parameters directly in encoder_profile_id or reference a registry?