86 lines
3.5 KiB
Markdown
86 lines
3.5 KiB
Markdown
# ECP-0028: test strategy + coverage gates
|
|
|
|
Status: Draft
|
|
|
|
## Problem / context
|
|
|
|
every.channel is security and integrity sensitive. Before we scale ingestion and relaying, we need a test suite that makes protocol regressions obvious and makes junk injection difficult to accidentally enable.
|
|
|
|
We also want coverage as a forcing function to keep logic in testable libraries, not buried in binaries or UI glue.
|
|
|
|
## Decision
|
|
|
|
Adopt a layered test strategy and a coverage gate:
|
|
|
|
- **Unit tests** for pure logic (hashing, manifests, proofs, encryption, quotas, URL parsing).
|
|
- **Integration tests** for cross-crate invariants (manifest <-> object meta compatibility, Merkle membership validation, catalog entry semantics).
|
|
- **End to end tests** for “single node publish + subscribe” flows, with deterministic fixtures and explicit opt-in for tests that require external dependencies (ffmpeg headers, OS devices).
|
|
|
|
Coverage goals are defined per layer:
|
|
|
|
- `ec-core`, `ec-crypto`, `ec-moq`, `ec-iroh`, `ec-linux-iptv`: target **100% line coverage** (excluding third_party).
|
|
- Node runner + Tauri backend: target **high coverage** for shared logic modules; binary-only glue may be excluded, but the glue must be minimal by policy.
|
|
- UI: target **behavioral tests for state transitions** (not full DOM snapshots).
|
|
|
|
## Tooling
|
|
|
|
- Use `cargo llvm-cov` for coverage measurement.
|
|
- Coverage is run per-crate and per-workspace in the nix dev shell so `ac-ffmpeg` can find ffmpeg headers.
|
|
|
|
## Test matrix
|
|
|
|
Unit tests (must be deterministic)
|
|
|
|
- `ec-core`
|
|
- Manifest ID determinism and change sensitivity.
|
|
- Merkle root correctness.
|
|
- Merkle proof generation + verification (including tamper detection).
|
|
- `ec-crypto`
|
|
- Stream key/nonce derivation determinism.
|
|
- Encrypt/decrypt roundtrip and mismatch failures.
|
|
- Manifest signature sign/verify and allowlist behavior.
|
|
- `ec-moq`
|
|
- Object frame encode/decode roundtrip.
|
|
- Manifest frame encode/decode roundtrip.
|
|
- File relay sanitization stability.
|
|
- `ec-iroh`
|
|
- Token bucket throttling/refill behavior (no sleeps).
|
|
- `ec-linux-iptv`
|
|
- `channels.conf` parsing (unique/sorted, ignore comments).
|
|
- Default tune command construction.
|
|
|
|
Integration tests (cross-crate invariants)
|
|
|
|
- Manifest root validates object `chunk_hash` either by direct hash list or by Merkle proof.
|
|
- Encrypted objects preserve integrity checks (hash is over plaintext).
|
|
- Catalog entries carry manifest summaries consistently.
|
|
|
|
End to end tests (opt-in / platform dependent)
|
|
|
|
- `moq` publish/subscribe loopback tests with epoch manifests:
|
|
- publish N chunks, subscribe and verify acceptance/rejection paths.
|
|
- Linux DVB discovery tests:
|
|
- run only when `/dev/dvb` exists; otherwise skip.
|
|
|
|
## Policy implications
|
|
|
|
- New protocol logic should land in libraries, not binaries.
|
|
- Any feature that changes integrity behavior (hashing, proofs, signing, validation, quotas) must add tests proving:
|
|
- positive path (accept)
|
|
- negative path (reject)
|
|
- no panics on malformed inputs
|
|
|
|
## Alternatives considered
|
|
|
|
- “Only E2E tests”: rejected (slow and flaky).
|
|
- “Only unit tests”: rejected (cross-crate breakages are likely).
|
|
|
|
## Rollout / teardown
|
|
|
|
1. Add unit tests to core crates until 100% coverage is achievable.
|
|
2. Refactor binary logic into testable modules when coverage shows dead zones.
|
|
3. Add integration tests for manifest and object invariants.
|
|
4. Add opt-in E2E tests for ffmpeg and device-bound pipelines.
|
|
|
|
Teardown: if llvm-cov becomes too costly locally, keep tests and make coverage gates advisory, but retain per-crate coverage reports.
|
|
|