every.channel/evolution/proposals/ECP-0156-duplicate-publisher-deterministic-data-layer.md

ECP-0156: Duplicate Publisher Deterministic Data Layer

Status: Draft

Context

Two publisher nodes may broadcast the same logical channel at the same time. The archive and relay layers need this for resilience, but duplicate publishers currently risk looking like conflicting streams instead of convergent copies of the same media.

Decision

Duplicate publishers are valid for a published channel. The data layer dedupes and verifies media by content identity, not by publisher envelope identity:

• CMAF init and media segment bytes for the same input, ladder profile, and chunk cadence must be byte-for-byte identical.
• BLAKE3 media hashes and per-rung Merkle roots are the shared data identity.
• Publisher manifests may carry different stream_id, epoch_id, created_unix_ms, signatures, locators, and manifest ids.
• The archive must treat matching media hashes from different publishers as corroborating sources.
• Archive records must carry source identity. Two copied buffers with the same source_node are not duplicate-publisher proof, even when their BLAKE3 hashes match.
• Divergent hashes for the same logical channel, rendition, and media time are misses that must be measured before the data is promoted as redundant.

Verification

The proof path has two stages:

1. Single-node duplicate-publisher tests produce the same ladder twice with different publisher identities and assert byte-for-byte BLAKE3 equality for every generated init and media segment. The duplicate_publishers_same_input_produce_identical_cmaf_ladder_bytes test is part of the default Rust test path when ffmpeg is present; it is not an ignored E2E.
2. Production verification runs the same channel on two real publishers long enough to measure duplicate media convergence, hash divergence, missing objects, and backfill behavior in Grafana.

The goal is not just "two publishers are online." Success requires elapsed production time behind the numbers and dashboards that show duplicate hits, misses, and archive repair.

Consequences

Manifest ids cannot be used as the archive dedupe key for duplicate publishers. Operators get a clear signal when two publishers produce identical bytes versus merely announcing the same channel. If encoder determinism changes, the single-node test fails before production redundancy silently degrades.

Alternatives considered

• Dedupe by manifest id. This preserves envelope identity but misses the resilience property because duplicate publishers necessarily produce different envelopes.
• Dedupe by logical channel and time only. This can hide encoder divergence and promote bad redundancy before byte-level media equality is proven.
• Disable duplicate publishers until the scheduler is perfect. This avoids conflict handling but weakens live resilience and leaves the archive data layer untested.

Rollout/teardown

Roll forward by landing the local deterministic test, adding miss/duplicate metrics to the archive scrape surface, then running two publishers for one logical channel in production. Roll back by disabling duplicate scheduling for that channel; existing content-addressed archive objects remain valid.

Implementation notes

The node-agent archive scrape now exposes duplicate-source and miss gauges without placing hashes in labels. Per node, role, broadcast, rendition, and track it reports duplicate matching hash sources, duplicate hash sequences, divergent hash sequences, and missing hash records. Grafana shows those next to archive ladder coverage so the production duplicate-publisher run has an operator-visible convergence and miss signal.

ec-node archive-convergence is the primary proof surface for duplicate media identity. It compares named archive manifest roots directly inside the Rust node binary, groups records by logical stream, rendition, track, and sequence, and only returns ok when every expected sequence has matching duplicate source hashes with no missing or divergent sequence. It also requires archive records to carry at least two distinct source_node values, so mirrored global-origin manifests cannot pass as independent publishers. This keeps the media-data invariant in the already-shipped Rust artifact instead of extending the Python node-agent. Rollout gates should use ec-node archive-convergence --require-ok; the command emits the JSON report either way, but --require-ok exits non-zero unless duplicate convergence is actually proven. ec-node archive-convergence --prometheus renders the same Rust convergence report as scrapeable every_channel_archive_* gauges for duplicate source records, duplicate sequences, divergent sequences, source-local divergence, missing hashes, missing source identity, media timing conflicts, record source count, and pass/fail state. This gives Grafana a Rust-owned proof metric path while the older node-agent ladder metrics remain available during migration. ec-node archive-convergence-serve keeps that proof path live for Prometheus: it serves /health and /metrics, recomputes convergence on each scrape, and emits scrape_ok=0 metrics instead of disappearing when manifests are missing or not ready. Production Grafana can therefore distinguish a healthy metrics target from an unproven duplicate-publisher run. The Nix services.every-channel.ec-node.archive.convergence.proofs option turns those Rust proof servers into named systemd units. Each proof must name at least two NAME=PATH sources and gets a dedicated listen address, so operators can add one Prometheus scrape target per duplicate channel without resurrecting the Python node-agent as the proof oracle. Forge enables an initial la-kcop-publisher-origin proof target on 127.0.0.1:7812 and Prometheus scrapes it alongside the other local every.channel targets. Until two real publisher manifest roots are mounted or fetched into Forge, the target intentionally uses the Forge manifest root as a placeholder peer and must report unproven convergence rather than green duplicate-publisher proof. Forge also exposes a static two-NUC la-kcet-remote-publisher-origin proof target once that channel is the live converged duplicate sample. Dynamic Headscale file-SD remains useful for discovery, but it can include relays and stale nodes; duplicate-publisher proof should use an explicit publisher pair or future scheduler group labels so unrelated agents do not turn a passing channel red. This static proof exports its own Rust convergence gauges rather than gating on broad legacy Prometheus aggregates, because older node-agent archive metrics do not yet carry enough proof-role labels to avoid summing stale divergence from unrelated scrape targets.

ec-node archive-convergence-measure is the primary production proof harness. It fetches named node-agent /v1/archive-manifest samples or direct manifest JSONL URLs, writes bounded temporary manifest roots, reuses the Rust archive-convergence report, and optionally queries Prometheus for the Grafana-facing duplicate/miss series. A production run only counts as complete when the report has elapsed samples, matching duplicate media hashes, zero divergent hash sequences, and live Prometheus series for the duplicate/miss gauges. The measurement groups records by archive record source identity, not by the URL used to fetch a manifest, and reports source identity failures when the sample is too weak to prove independent publisher data. The older scripts/measure-duplicate-publishers.py stays compatibility-only until live operators and Forge jobs are switched to the Rust command. The convergence report carries bounded divergent-sequence samples with per-source hash, byte size, receive time, source node/session, CAS path, and media timing when present, so a red proof is immediately actionable without fetching full manifests by hand. It also reports a non-blocking media-timing-missing count and Prometheus gauge; hash equality can still prove duplicate bytes, but missing timing means a divergent proof cannot yet classify whether the mismatch is a phase/windowing problem or an encoder byte problem. Publisher service builders must pass proof cadence explicitly. Both the node-agent publisher supervisor and Nix systemd publisher module set --publisher-archive-segment-duration-ms and --publisher-start-boundary-ms by default, so netbooted NUCs do not depend on stale hotpatch CLI defaults when aligning duplicate publisher proof windows.

ec-node archive-convergence-measure-serve turns that production proof harness into a live Prometheus target. Each /metrics scrape fetches one fresh sample from node-agent or direct JSONL manifest URLs, keeps a bounded in-memory sample window, and only reports measurement ok after the configured elapsed window has passed. This avoids blocking Prometheus scrapes for the measurement duration while still preventing two immediate samples from looking like a real production run. The service emits measurement-level gauges for fetch success, source record counts, invalid records, elapsed seconds, Prometheus series presence, reasons, and then appends the same every_channel_archive_* convergence gauges from the latest sample. The service can also read Prometheus file-SD JSON from Forge's Headscale node-agent discovery and turn each discovered target into a sampled node-agent manifest source. The Nix services.every-channel.ec-node.archive.convergence.remoteProofs option creates these remote proof services as systemd units from either static NAME=URL endpoints or dynamic file-SD inputs. Forge now exposes la-kcop-remote-publisher-origin on 127.0.0.1:7813 using the live /var/lib/prometheus/every-channel-node-agents.json inventory. It must stay red until that inventory contains at least two independent publisher node-agents whose publisher.m4s records converge.

When archive-serve ports are not reachable from the proof runner, the node-agent exposes a bounded, tailnet-authenticated /v1/archive-manifest sample endpoint. The harness can use that endpoint for each named publisher, compare local manifest records directly, and still require at least two elapsed samples before declaring success.

Production duplicate proof also requires archive-buffer freshness on each participating publisher. During mixed-generation rollouts, the current node-agent may supervise an older installed archive-hot-sync helper. The agent must probe helper flag support and omit optional arguments such as --link-mode when an older helper lacks them, because a silently failing archive-buffer sync can leave one publisher with healthy live streams but stale manifests.

The publisher buffer refresh is freshness-first: the node-managed sync must mirror full manifests without origin object fetch before running the slower cache fill/prune pass. This lets convergence checks, Grafana scrape surfaces, and demand fetch see current BLAKE3 indexes even when proactive CAS object backfill is still catching up.

wt-archive stamps each archive index record with source_node and source_session. The Nix archive launcher passes the runtime hostname as --source-node; explicit CLI users can override it. Older records without this identity continue to parse, but proof commands and production measurement mark them incomplete instead of accepting them as independent publisher evidence.

Publisher-origin proof must be captured before relay/archive mirroring can collapse source identity. When node-agent archive buffering is enabled, supervised wt-publish processes pass --publisher-archive-output-dir, --publisher-archive-manifest-dir, and --publisher-archive-source-node. wt-publish now supervises the Rust publisher-proof-archive-source worker for that archive track. The worker splits the MPEG-TS source by source-clock windows, fresh-encodes each bounded window with the deterministic proof profile, stores the resulting media fragments under publisher.m4s in the same CAS/index format, and stamps them with node-agent source identity. The relay playback encoder remains continuous for watchability, but it is no longer the BLAKE3 data identity for duplicate-publisher proof. The source identity is explicit override first, then hostname plus a short hash of machine-id, with boot-id only as a fallback; hostname alone is not enough because publisher images can share names like ec-node. Production duplicate verification can therefore compare publisher.m4s from two publisher buffers without treating copied relay-origin manifests as independent sources.

Proof tooling defaults to publisher.m4s. The relay video track 0.m4s is useful playback data, but it is not duplicate-publisher proof: a publisher buffer may hold relay/cache records on 0.m4s that have no publisher source identity. Production convergence checks that sample 0.m4s should be treated as playback/archive-cache diagnostics, not byte-for-byte duplicate publisher evidence.

The first live publisher-origin measurements on 2026-06-08 showed correct distinct source labels but zero matching duplicate sequences for la-nbc4, la-pbs-socal, and la-kcet. The failure is useful: independent wt-publish processes currently start their fragment sequence and encoder chunk phase at local process start, so sequence 0 from two publishers is not necessarily the same broadcast moment. Duplicate-publisher proof therefore requires a shared chunk clock or scheduler-controlled aligned encoder phase before byte-for-byte archive convergence can pass in production.

Publisher-origin publisher.m4s records now require timed fMP4 fragments for global proof and map those fragments onto observed wall-clock epoch buckets instead of local process counters. The Rust writer learns track timescales from the init moov box, reads fragment moof/traf/tfhd+tfdt decode timestamps to reject untimed proof when possible, then assigns group_sequence = observed_epoch_bucket * bucket_stride + fragment_slot. Fragments that lack usable timing still fall back to the previous local counter so publishing does not fail hard on malformed metadata, but duplicate-publisher proof should use timed fragments. The wt-publish ffmpeg path also preserves source timestamps and uses closed-GOP, single-threaded x264 settings with forced keyframe cadence so independent publishers have a real chance of producing identical bytes for the same media time window.

A later live run on 2026-06-08 found a stricter local invariant before cross-publisher byte equality: each publisher must produce at most one hash for a given source_node and group_sequence. Production publisher.m4s samples for la-kcop and la-ktla showed multiple hashes from the same source in the same sequence bucket because real fMP4 fragments can arrive faster than the configured proof segment duration, and the writer rounded decode time into repeated buckets. The writer now uses a fixed per-epoch bucket stride and increments an in-bucket fragment slot when multiple timed fragments arrive inside the same proof duration. This keeps source-local manifests unique while allowing independently restarted publishers to align on the same observed wall-clock bucket. ec-node archive-convergence reports this separately as source_local_divergent_sequences so operator tooling can distinguish a self-contradicting publisher from two publishers that simply disagree about the same sequence. Because bucket-strided proof sequences intentionally leave numeric gaps, archive convergence uses the observed sparse sequence union for publisher-origin manifests. Dense contiguous sequence ranges remain available in the simulation layer when a model explicitly expects every integer sequence.

The 2026-06-08 live la-kcet/publisher.m4s sample from Forge confirmed that both publishers now emit distinct source identities (ec-node-c3546fa5abc3 and ec-node-72cf1c3aa196) with no missing source identity records on the sampled publisher-origin manifests. It also confirmed the remaining bug: 156 shared publisher-origin sequences had zero byte-for-byte BLAKE3 matches and 156 divergent hashes. The next production fix must align the publisher chunk clock and encoded fMP4 byte stream, not merely improve scrape or Grafana plumbing.

After the wall-clock bucket hotpatch, the same live proof no longer has fake sparse-range missing IDs: la-kcet/publisher.m4s reported 376 observed proof sequences, zero missing source identities, zero source-local divergent sequences, and 234 divergent shared sequences. A byte-level sample for sequence 7287381184512 had different sizes, different BLAKE3 hashes, different tfdt base-media-decode-times (210210 versus 0), and different mdat payload prefixes. Across that sampled window there were zero common fragment hashes even when sequence IDs were ignored, proving that the remaining failure was independent-encoder media phase and fMP4 payload determinism, not an archive manifest identity bug.

A later la-kcop/publisher.m4s sample exposed a stricter live-source bug: source-window proof records were using unsynced MPEG-TS PCR chunk indexes as group_sequence when the OTA UTC clock was unavailable, causing restart-dependent jumps such as 93M, 135M, 341M, and 390M. The source-proof writer now uses the chunk UTC start only when the chopper reports synced timing, otherwise it falls back to the local wall-clock window start, and rewrites fMP4 tfdt onto that shared window before hashing. The live HTTP proof worker also retries transient source opens/reader failures in unbounded live mode, so a tuner 503 or malformed TS burst is skipped/retried instead of killing the publisher proof process.

The synced source-window clock must use the chopper's exact global chunk index, not integer UTC seconds. A 1001 ms proof cadence makes whole-second UTC start metadata lossy: adjacent source windows can share the same utc_start_unix, which caused one publisher to write several different hashes under the same source-local group_sequence. Synced chunks therefore use ChunkTiming.chunk_index directly; only unsynced chunks fall back to local wall-clock receipt. The live source-window proof writer also keeps subfragment slot allocation as stream state instead of per-chunk state. Real source windows can be emitted in more than one proof chunk for the same media timing sequence; resetting the slot counter for every chunk reused the same group_sequence and made one healthy publisher look self-divergent. The counter is bounded so the long-running live worker does not grow state unbounded.

wt-publish now has an explicit Unix-epoch start boundary, defaulting to the publisher-origin proof cadence. After relay setup and immediately before spawning ffmpeg it waits until the next boundary, so a newly restarted duplicate publisher starts its forced-keyframe clock on the same global cadence as already-running publishers. This does not by itself prove byte equality; it removes the local-process-start phase error from the live publisher path and gives rollout measurement a deterministic knob (--publisher-start-boundary-ms 0 disables it). The live ffmpeg argument plan is factored into a Rust unit-testable helper so future timestamp/keyframe changes are pinned in ec-node instead of being inferred from node-agent process strings or production samples.

The first post-start-clock live sample still failed duplicate byte identity: both publishers landed in the same wall-clock proof bucket, but one fragment carried tfdt=390390 while the other carried tfdt=30030, matching the staggered restart gap. Their mdat prefixes differed too, which means a continuous x264 encoder keeps enough local history that a later restart cannot prove byte equality merely by joining the same wall-clock cadence. The live profile therefore enables x264 stitchable=1 alongside closed GOP, no scenecut, no B-frames, no lookahead, and one thread. If that still does not converge in production, the next fix is a deliberately stateless per-fragment encode or a Rust-owned media clock/segmenter that resets encoder history at each proof boundary.

The follow-up production hotpatch moved the start-boundary wait to immediately before ffmpeg spawn, enabled stitchable=1, and restarted both publisher nodes in the same batch. The latest la-kcet sample still reported zero matching duplicate hashes with no missing source identity and no source-local divergence. A final sampled shared sequence differed by hundreds of milliseconds of receive time and by media size (439737 versus 270283 bytes for the video fragment), so the remaining mismatch is not just MP4 timestamp metadata. Production duplicate proof now needs a stateless fragment boundary: either encode each proof segment from the same bounded source window with fresh encoder state, or make the Rust media pipeline own exact frame-window capture before calling ffmpeg/x264.

Archive manifests now carry optional fMP4 media timing for publisher-origin fragments. The archive-convergence gate treats equal archive group sequence IDs with different media sequence or decode-time metadata as media_sequence_conflict, even if the byte hash happens to match. This keeps production proof aligned with the Rust simulation model: a duplicate publisher only proves the same broadcast moment when the archive sequence and media window agree.

The first stateless proof primitives are now in ec-node. publisher-proof-segment takes one bounded MPEG-TS source-clock window, runs a fresh deterministic x264/AAC fMP4 encode, splits the result into init bytes and media fragments, and emits BLAKE3 hashes for each. publisher-proof-windows uses the Rust MPEG-TS source-clock splitter first, then fresh-encodes each bounded window and reports per-window source TS, init, and media hashes. Proof windows carry explicit MPEG-TS decoder context with --preroll-packets, defaulting to the repo-owned WT_PUBLISH_PROOF_PREROLL_PACKETS budget, so mid-GOP windows do not silently depend on best-effort decoder recovery. Focused Rust tests fresh-encode the same bounded input and the same finite source-window campaign twice and assert byte-for-byte identical proof hashes.

publisher-proof-duplicates is the single-node duplicate-publisher gate for the stateless path. It runs publisher-proof-windows independently under at least two publisher identity labels, defaults to publisher-a and publisher-b, and compares source TS, init, and media fragment BLAKE3 hashes for every source-clock window. --require-ok exits non-zero unless every compared window matches, and duplicate publisher labels are rejected so the proof cannot accidentally collapse to one source identity. publisher-proof-compare is the cross-machine stateless proof gate: each publisher can run publisher-proof-windows against the same bounded source TS file locally, copy the JSON report back to the operator host, and compare the reports by named publisher. It rejects mismatched chunk cadence, missing windows, source TS hash mismatches, init hash mismatches, media fragment hash mismatches, and empty media windows.

publisher-proof-remote-compare is the production operator harness for that cross-machine gate. It copies one bounded .ts proof input to each named SSH target, runs ec-node publisher-proof-windows on the target, stores each returned JSON report under the local output directory, writes a compare.json, and returns the existing compare report with upload/proof timing. Remote labels use the same single-component validation as publisher identities, remote proof roots are constrained to /tmp/every-channel-*, and cleanup is opt-in so the generated proof files remain inspectable unless the operator explicitly requests removal. This keeps the live proof path in Rust without making the Python node-agent a new oracle. It proves the machine/runtime/compiler boundary without requiring the two NUCs to share a live tuner at the exact same instant.

publisher-proof-archive-source is the live archive implementation of the same proof model. It can read local source files directly, read plain HTTP MPEG-TS bodies directly for HDHomeRun-style sources, or fall back to an ffmpeg MPEG-TS copy reader for other inputs. Each emitted source-clock window is encoded with fresh proof state, archived as CAS-backed publisher.m4s records, and mapped to source-clock group sequences with explicit media timing metadata. A focused Rust regression now archives the same bounded TS input as two source nodes, then runs archive-convergence against the two manifest roots and requires full duplicate convergence with zero divergent or source-local divergent sequences.

Forge ci-gates now runs the publisher_proof and archive_convergence Rust filters before the distributed simulator campaign, so single-node byte-for-byte determinism, source-window archive proof semantics, and duplicate archive convergence are checked before production rollout evidence is considered. The next production step is to deploy the updated node binary and let fresh publisher.m4s source-window records age into the Grafana scrape window so live duplicate metrics can replace the older continuous-encoder divergence.