Wire HDHomeRun observations and recover Forge OP Stack
This commit is contained in:
every.channel
parent
8065860449
commit
0d86104762
18 changed files with 1613 additions and 58 deletions
|
|
@ -7,6 +7,8 @@ This repository owns deployment of `git.every.channel` (Hetzner 300TB host).
|
|||
- SSH access to `root@git.every.channel`.
|
||||
- Local key that matches host `authorized_keys` (default: `~/.ssh/id_ed25519`).
|
||||
- `nix` with flakes enabled.
|
||||
- For emergency Hetzner recovery, Robot Webservice credentials in 1Password item `Hetzner Robot`
|
||||
or `EVERY_CHANNEL_ROBOT_USER` / `EVERY_CHANNEL_ROBOT_PASSWORD`.
|
||||
|
||||
## Deploy
|
||||
|
||||
|
|
@ -38,6 +40,44 @@ nix run nixpkgs#nixos-rebuild -- \
|
|||
- `EVERY_CHANNEL_FORGE_BUILD_HOST` (default same as target)
|
||||
- `EVERY_CHANNEL_FORGE_SSH_IDENTITY` (default `~/.ssh/id_ed25519`)
|
||||
|
||||
## Emergency Robot recovery
|
||||
|
||||
Use this only when both Forge HTTPS and SSH are unreachable. The dedicated host is server
|
||||
`2800441` at `95.216.114.54`.
|
||||
|
||||
```sh
|
||||
./scripts/hetzner-robot-forge.sh probe
|
||||
```
|
||||
|
||||
If the probe confirms outage, sign in to 1Password CLI so the wrapper can read the existing Robot
|
||||
Webservice item at runtime:
|
||||
|
||||
```sh
|
||||
op signin
|
||||
./scripts/hetzner-robot-forge.sh status
|
||||
```
|
||||
|
||||
To boot the host into Hetzner Rescue and issue a hardware reset:
|
||||
|
||||
```sh
|
||||
./scripts/hetzner-robot-forge.sh recover
|
||||
./scripts/hetzner-robot-forge.sh wait-ssh
|
||||
```
|
||||
|
||||
The wrapper masks Robot-generated rescue passwords by default and tries to attach the local SSH key
|
||||
fingerprint when activating rescue. Set `EVERY_CHANNEL_ROBOT_AUTHORIZED_KEY_FINGERPRINT` if Robot
|
||||
uses a different uploaded key fingerprint. Set `EVERY_CHANNEL_ROBOT_PRINT_SENSITIVE=1` only when
|
||||
password-based rescue login is required.
|
||||
|
||||
If production boots but public SSH and HTTPS still time out, inspect the previous boot from Rescue.
|
||||
The known recovery check is host-wide VPN state: `mullvad-daemon.service` must not be active on
|
||||
`ecp-forge`, because its firewall policy can block public Forge ingress even when Robot and the
|
||||
NixOS firewall allow the ports. If a not-yet-redeployed generation still starts Mullvad and the
|
||||
mutable cached target state is rewritten to `secured`, back up `/boot/grub/grub.cfg`, append
|
||||
`systemd.mask=mullvad-daemon.service systemd.mask=mullvad-early-boot-blocking.service` to the
|
||||
default Linux line, and reboot production. After public SSH returns, deploy this repo's NixOS config
|
||||
so the bootloader is regenerated without the emergency mask.
|
||||
|
||||
## Verify
|
||||
|
||||
```sh
|
||||
|
|
|
|||
|
|
@ -98,7 +98,7 @@ ssh -o BatchMode=yes -o IdentityAgent=none -o IdentitiesOnly=yes -i ~/.ssh/id_ed
|
|||
Once the rollup RPC is live, deploy the observation rail to the L2 RPC:
|
||||
|
||||
```sh
|
||||
EVERY_CHANNEL_RPC_URL=http://127.0.0.1:8545 \
|
||||
EVERY_CHANNEL_RPC_URL=http://127.0.0.1:28545 \
|
||||
EVERY_CHANNEL_PRIVATE_KEY_FILE=/path/to/private-key \
|
||||
./scripts/op-stack/deploy-observation-ledger.sh
|
||||
```
|
||||
|
|
@ -106,5 +106,10 @@ EVERY_CHANNEL_PRIVATE_KEY_FILE=/path/to/private-key \
|
|||
## Notes
|
||||
|
||||
- `op-geth` and `op-node` RPC surfaces bind to `127.0.0.1` on `ecp-forge`.
|
||||
- The OP Stack L2 execution RPC defaults to `127.0.0.1:28545`, not `8545`; `8545` is reserved for
|
||||
the full Ethereum mainnet node on the same host.
|
||||
- `op-geth` P2P uses `28549`, not the Ethereum default `30303`; `30303` is reserved for the host
|
||||
full Ethereum node.
|
||||
- The public firewall opening is only for the `op-node` P2P port.
|
||||
- The bootstrap uses `op-deployer/v0.6.0-rc.3` by default and official OP Labs container images.
|
||||
- The bootstrap uses `op-deployer/v0.6.0-rc.3` by default and OP Labs runtime images aligned to the
|
||||
generated rollup schema.
|
||||
|
|
|
|||
|
|
@ -79,6 +79,18 @@ On Linux / forge hosts, the equivalent worker path lives in `ec-node`:
|
|||
- publish with
|
||||
`ec-node nbc-wt-publish --url https://cdn.moq.dev/anon --name forge-nbc-sports-philly --source-url 'https://www.nbc.com/live?brand=nbc-sports-philadelphia'`
|
||||
- for unattended hosts, persist the Chrome profile with `EVERY_CHANNEL_NBC_PROFILE_DIR=/path/to/profile`
|
||||
- to automate a Verizon popup on Linux / forge, pass MVPD credentials via env or file paths:
|
||||
`EVERY_CHANNEL_NBC_MVPD_USERNAME`, `EVERY_CHANNEL_NBC_MVPD_PASSWORD`,
|
||||
`EVERY_CHANNEL_NBC_MVPD_USERNAME_FILE`, `EVERY_CHANNEL_NBC_MVPD_PASSWORD_FILE`
|
||||
- the NixOS module can point the Linux worker at root-managed credential files with
|
||||
`services.every-channel.ec-node.nbc.mvpdUsernameFile` and
|
||||
`services.every-channel.ec-node.nbc.mvpdPasswordFile`
|
||||
- for forge-style isolation, the NixOS module can keep only the NBC publisher inside a rootless
|
||||
user+network namespace backed by `slirp4netns` with
|
||||
`services.every-channel.ec-node.nbc.isolateWithUserNetns = true`
|
||||
- pair that with `services.every-channel.ec-node.nbc.requireMullvad = true` to block worker startup
|
||||
until the host Mullvad daemon is connected; optionally pin a region/country family with
|
||||
`services.every-channel.ec-node.nbc.mullvadLocation = "USA"`
|
||||
- the NixOS module exposes `services.every-channel.ec-node.nbc.*` for a persistent Xvfb display plus
|
||||
an optional local-only VNC bridge so MVPD auth can be completed only when the session is cold
|
||||
- on Linux virtual displays, the worker disables Chrome GPU acceleration by default; only set
|
||||
|
|
@ -115,6 +127,47 @@ Requires Nix (so `ac-ffmpeg` finds FFmpeg headers):
|
|||
./scripts/e2e-hdhr.sh --host <HDHR_HOST> --channel <CHANNEL>
|
||||
```
|
||||
|
||||
## HDHomeRun + Observation Chain E2E Test
|
||||
|
||||
This runs a local Anvil chain, deploys the observation registry/ledger, publishes one HDHomeRun
|
||||
manifest epoch, and verifies that the manifest-derived observation finalizes on-chain.
|
||||
|
||||
Requires Nix, Foundry, and a reachable local HDHomeRun:
|
||||
|
||||
```sh
|
||||
./scripts/e2e-hdhr-blockchain.sh --host <HDHR_HOST> --channel <CHANNEL>
|
||||
```
|
||||
|
||||
## Local HDHomeRun Publisher Against Remote Observation Chain
|
||||
|
||||
The remote OP Stack RPC on `ecp-forge` is intentionally local-only. From the local publisher box,
|
||||
tunnel it first:
|
||||
|
||||
```sh
|
||||
ssh -N -L 9545:127.0.0.1:28545 root@git.every.channel
|
||||
```
|
||||
|
||||
Then run a local HDHomeRun publisher with observation submission enabled:
|
||||
|
||||
```sh
|
||||
cargo run -p ec-node -- moq-publish \
|
||||
--publish-manifests \
|
||||
--epoch-chunks 1 \
|
||||
--broadcast-name local-hdhr-8-1 \
|
||||
--observation-rpc-url http://127.0.0.1:9545 \
|
||||
--observation-ledger <OBSERVATION_LEDGER_ADDRESS> \
|
||||
--observation-private-key-file /path/to/witness.key \
|
||||
hdhr --host <HDHR_HOST> --channel <CHANNEL>
|
||||
```
|
||||
|
||||
Environment fallbacks are also supported:
|
||||
|
||||
- `EVERY_CHANNEL_OBSERVATION_RPC_URL`
|
||||
- `EVERY_CHANNEL_OBSERVATION_LEDGER`
|
||||
- `EVERY_CHANNEL_OBSERVATION_PRIVATE_KEY`
|
||||
- `EVERY_CHANNEL_OBSERVATION_PRIVATE_KEY_FILE`
|
||||
- `EVERY_CHANNEL_OBSERVATION_PARENT_HASH`
|
||||
|
||||
## Mesh E2E Test (Split Sources)
|
||||
|
||||
This runs two publishers over the same broadcast:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue