54 lines
2.3 KiB
Markdown
54 lines
2.3 KiB
Markdown
# ECP-0023: HLS ingest pipeline
|
||
|
||
Status: Draft
|
||
|
||
## Problem / context
|
||
|
||
Some broadcasters and community relays expose HLS playlists directly. If two peers ingest the same HLS origin and the bytes are identical, we can dedupe and swarm immediately. We need a first-class ingest path for HLS that plugs into the deterministic chunking pipeline.
|
||
|
||
## Decision
|
||
|
||
Add HLS as a supported ingest source. The ingest path:
|
||
|
||
- Uses ffmpeg/libav demuxing (no custom parsers).
|
||
- Normalizes input into our deterministic chunker.
|
||
- Treats identical origin bytes as a fast path for dedupe; if bytes vary, re-encode deterministically.
|
||
|
||
## Details
|
||
|
||
### Ingest modes
|
||
|
||
- **Pass-through**: If the HLS origin serves MPEG-TS segments that are already deterministic and stable, chunk the byte stream as-is.
|
||
- **Normalize**: If the origin uses fMP4 or inconsistent segmenting, remux via ffmpeg into a stable TS stream before chunking.
|
||
- **Deterministic re-encode**: If byte identity isn’t stable (e.g. ad insertion, timestamps), re-encode with the deterministic profile.
|
||
|
||
Initial implementation note: use ffmpeg (CLI) to output a continuous MPEG-TS stream which is then chunked by the ac-ffmpeg path. This keeps parsing inside ffmpeg and avoids custom HLS logic.
|
||
|
||
### Encryption
|
||
|
||
- If the HLS stream is encrypted with clear AES-128 and a key is provided, the ingest pipeline can decrypt and then re-encrypt using our stream encryption (stream id + chunk index + optional network secret).
|
||
- DRM (Widevine/PlayReady) is out of scope for now; ingest will skip or mark as protected.
|
||
- Our chunk encryption remains independent of HLS origin encryption.
|
||
|
||
### Identifiers
|
||
|
||
- HLS sources map to a `StreamKey` with `source.kind = "hls"` and `source.channel = <url-or-name>`.
|
||
- `StreamId` is derived as usual from `StreamKey`.
|
||
|
||
## Alternatives considered
|
||
|
||
- Directly serving HLS playlists to peers: rejected (pushes HLS semantics to every client).
|
||
- Custom HLS parser: rejected (ffmpeg already solves it).
|
||
|
||
## Rollout / teardown plan
|
||
|
||
1. Add `HlsSource` to `ec-node` ingest CLI and `ec-core` source typing.
|
||
2. Add ffmpeg/libav HLS ingest path to `ec-chopper`.
|
||
3. Wire HLS ingest into MoQ publisher and manifest pipeline.
|
||
|
||
Teardown: disable HLS ingest and fall back to HDHR/IPTV sources.
|
||
|
||
## Open questions
|
||
|
||
- Which HLS modes should default to remux vs re-encode?
|
||
- Should HLS playlist metadata (program info) be carried into stream metadata?
|