every-channel/every.channel
1
0
Fork 0
Code Releases Activity Actions 2
every.channel/evolution/proposals/ECP-0023-hls-ingest.md
every.channel 897e556bea
every.channel: sanitized baseline
2026-02-15 16:17:27 -05:00

2.3 KiB
Raw Permalink Blame History

ECP-0023: HLS ingest pipeline

Status: Draft

Problem / context

Some broadcasters and community relays expose HLS playlists directly. If two peers ingest the same HLS origin and the bytes are identical, we can dedupe and swarm immediately. We need a first-class ingest path for HLS that plugs into the deterministic chunking pipeline.

Decision

Add HLS as a supported ingest source. The ingest path:

  • Uses ffmpeg/libav demuxing (no custom parsers).
  • Normalizes input into our deterministic chunker.
  • Treats identical origin bytes as a fast path for dedupe; if bytes vary, re-encode deterministically.

Details

Ingest modes

  • Pass-through: If the HLS origin serves MPEG-TS segments that are already deterministic and stable, chunk the byte stream as-is.
  • Normalize: If the origin uses fMP4 or inconsistent segmenting, remux via ffmpeg into a stable TS stream before chunking.
  • Deterministic re-encode: If byte identity isn’t stable (e.g. ad insertion, timestamps), re-encode with the deterministic profile.

Initial implementation note: use ffmpeg (CLI) to output a continuous MPEG-TS stream which is then chunked by the ac-ffmpeg path. This keeps parsing inside ffmpeg and avoids custom HLS logic.

Encryption

  • If the HLS stream is encrypted with clear AES-128 and a key is provided, the ingest pipeline can decrypt and then re-encrypt using our stream encryption (stream id + chunk index + optional network secret).
  • DRM (Widevine/PlayReady) is out of scope for now; ingest will skip or mark as protected.
  • Our chunk encryption remains independent of HLS origin encryption.

Identifiers

  • HLS sources map to a StreamKey with source.kind = "hls" and source.channel = <url-or-name>.
  • StreamId is derived as usual from StreamKey.

Alternatives considered

  • Directly serving HLS playlists to peers: rejected (pushes HLS semantics to every client).
  • Custom HLS parser: rejected (ffmpeg already solves it).

Rollout / teardown plan

  1. Add HlsSource to ec-node ingest CLI and ec-core source typing.
  2. Add ffmpeg/libav HLS ingest path to ec-chopper.
  3. Wire HLS ingest into MoQ publisher and manifest pipeline.

Teardown: disable HLS ingest and fall back to HDHR/IPTV sources.

Open questions

  • Which HLS modes should default to remux vs re-encode?
  • Should HLS playlist metadata (program info) be carried into stream metadata?