resolve: Gitea config backup push dead since 2026-08-08 — 92-commit backlog drained, script hardened to fail loudly

- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo
- Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved)
- Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB
- Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification,
  decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree,
  honest AgentMail/Telegram delivery reporting
- Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed
- Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated
- Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with
  'lost its durable fire claim ownership' and left the tree dirty)
This commit is contained in:
Tony0410
2026-11-07 05:28:39 +08:00
parent b820993ecc
commit 0b4ff25fd2
14 changed files with 370 additions and 110 deletions
+104
View File
@@ -2226,3 +2226,107 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted*
[[current-state]] (+frontmatter, +Recent Changes block), this index (+frontmatter, +summary, +incident
bullet), [[runbooks/job-radar]] (+concurrency subsection), this log, and the new incident page. Vault
mirror `/home/hermes/wiki` updated (both conventions).
## 2026-11-07 03:0x AWST - stocktake (wiki-stocktake-maintenance)
- **Script:** `wiki-maintain.sh` (repo, canonical) run clean — skills-index regenerated (**302 skills /
68 categories**), freshness report written to `raw/stocktake/freshness-20261107.txt`, Quartz rebuilt
(**769 files**) and deployed to `public/`, `hermes-wiki-serve.service` restarted.
- **Freshness:** **0 stale, 125 fresh, 3 no-date of 128 pages** — no page is past the 21-day threshold.
(`log.md` has no `updated` date and lands in the no-date bucket, which is correct for an append-only log.)
**Freshness is not accuracy**, so a live-vs-page diff was run against every page carrying live claims.
- **Tailscale (infrastructure/tailscale.md):** table rebuilt from `tailscale status --json`. **1 state
transition — `octop` (`100.92.163.122`) is back online** (peer record `Online: true`; `tailscale ping` →
`pong from octop via DERP(syd) in 98ms`); it is still **no LXC or VM on either hypervisor**, so it is an
orphaned tailnet peer, not an estate change. **9 offline ages refreshed (+1d each).** All **51 live IPv4
peers** match the table's IPs exactly — **no missing peers, no orphan rows, no IP mismatches** (52 rows =
51 peers + `hermes`; the ~23 IPv6-only `funnel-ingress-node` peers excluded by design). The CT460 Serve
route table still matches `tailscale serve status` **exactly (12 paths on :443)**; **both DEAD targets
re-confirmed dead** (no listener on `8769` `/agent-estate`, none on `9123` `/synergy-prep`), every other
Serve target has a live listener, and `/gitea` (`100.82.100.128:3000`) answers **HTTP 200**.
- **Proxmox (infrastructure/proxmox.md):** ⚠ **`m93p` runs 20 LXCs, not 19 — CT107 `mem0` was missing**
(running; `mem0-mcp.service` active on `:8890`; `pct exec 107 -- tailscale ip -4` → `100.73.94.53`).
Added to the table; the 2026-10-31 reading of 19 was already short for the same reason. Live now reads
**20 LXCs — 18 running, 2 stopped** (CT998, CT1264). `pve` still matches exactly: **17 LXCs + 4 VMs**.
Both hosts run the identical **`pve-manager 9.2.10`** build.
- **⚠ `m93p` uptime claim was wrong.** The page read *"up 1 day, 12:02 … rebooted ~2026-10-30 16:0x"*; live
`uptime -s` shows the host **booted 2026-11-05 21:19:52** (up **1 day, 7:18**, load **0.76 / 1.08 / 1.24** — normal).
The 2026-10-30 reboot was real but belonged to **`pve`** (boot **2026-10-30 16:24:00**).
- **Storage refreshed on both hosts** against live `pvesm status`: `pve` `ssd0` **76.43 → 78.29 %**
(~152 GiB free), `SSD1` **64.32 → 65.46 %**, `fred-qnap` **82.77 → 84.21 %**, `local` **79.66 → 81.62 %**,
`local-lvm` **13.75 → 14.25 %**; `m93p` iMac shares + `local` **60.25 → 76.21 %**, `nfs-proxmox-*`
**82.78 → 84.21 %**, `nfs-ssd0-39` **76.43 → 78.29 %**, `omb-internal` **57.92 → 64.66 %**.
- **Docker (infrastructure/docker-services.md):** set-wise diff against live `docker ps` on CT200/201/333 —
CT200 **61 running / 68 total** exact, CT201 **20 / 26** exact, CT333 **8 / 9** exact, **but CT333's
enumerated list still opened with `flamboyant_mahavira`, which is not running** (live name
**`nostalgic_keldysh`**, created 2026-11-05 22:15:57). Corrected. CT460 `ignis` running.
- **Cross-page corrections:** [[infrastructure/network-map]] `/gitea` was annotated *"Gitea (ct207 tailnet
IP)"* — it is **CT221 on `m93p`** (ct207 is `100.82.40.23`); also its `m93p` PVE version read *9.2.0*
(both hosts are 9.2.10). [[agents/hermes-chaos]] read *"`v0.22.3` … `5b147ab036d6` — the same build as
CT460"*, which contradicted itself: live is **`v0.22.5` … `bf0203a794a0`**, now genuinely identical to
CT460's (`hermes update --check` reports **v0.23.0** available). [[agents/hermes-production]]'s
*"2 cron jobs still pin `custom:omni`"* is **no longer true** — live `jobs.json` has **0** on the bare key
and **21** on `custom:omni-101`.
- **Fleet re-verified unchanged:** CT460 **29 jobs (26 enabled / 3 disabled)**; CT450 **25 jobs
(13 enabled / 12 disabled)** with **9 shared names** — confirmed by a live set-diff of both `jobs.json`.
A keyed `deepseek-paid-smart` chat probe against `omniroute-1` (`100.78.63.20:20128`) returns **HTTP 200**;
`config.yaml` still defaults to `custom:omni-101`.
- **Pages touched:** [[infrastructure/tailscale]], [[infrastructure/proxmox]], [[infrastructure/hosts]],
[[infrastructure/network-map]], [[infrastructure/docker-services]], [[agents/hermes-production]],
[[agents/hermes-chaos]], [[current-state]], [[index]], this log. No new incident page — no incident was
found; every drift above was a documentation drift corrected against a live probe.
## 2026-11-07 05:1x AWST - incident resolution: Gitea config backup push dead since 2026-08-08
- **Anthony's ask:** "fix this" on the failing backup alert — with a pointer to check Notion for the API key.
- **[[incidents/2026-08-08-gitea-config-backup-push-dead]] is now `status: resolved`.** The nightly
`e27da6c34bbb` config backup had been committing locally and failing to push **every night since
2026-08-08** — **92 commits** stranded — while its cron row read `last_status: ok`.
- **Four independent root causes, all confirmed against live systems:**
1. **No Gitea credential.** `origin` was HTTPS with no userinfo; the global `credential.helper=store`
file `~/.git-credentials` held only a `github.com` entry. The token in the Notion vault was
**valid** and byte-identical to the one the wiki repo carries in its remote URL — it simply was
never wired into the backup repo. Fixed with a host-scoped `git credential approve` entry (not
embedded in the URL); `git ls-remote origin` now authenticates with `GIT_TERMINAL_PROMPT=0`.
2. **`backup.sh` had no `exit` statement at all** — it always returned 0, so the scheduler's
`success = (returncode == 0)` recorded `ok` on every failed night. The scheduler was correct;
**the script was the liar**. This is why the outage ran 92 days without a job-level red.
3. **A 1.76 GB `checkpoints/` backlog** in the unpushed range (13 blobs > 100 MB) — junk that
`.gitignore` already excluded for *new* commits but that history still carried.
4. **Both secondary legs broken.** The `github` remote did not exist, and the fallback pushed a
branch's **full ancestry**, which contains a 7.45 GB tarball blob — so GitHub's 100 MB file
limit could never accept it. The AgentMail alert called a `python3 send_email.py` that no longer
exists, via a `chaos-agent@agentmail.to` inbox that no longer exists.
- **Backlog drained:** `git filter-repo --partial --refs origin/main..main --path checkpoints
--invert-paths --prune-empty never --force` stripped `checkpoints/` from the **unpushed range only**
(`origin/main` untouched, push stayed a fast-forward), then `checkpoints/` was untracked at the tip.
Push set **~2 GB → 264.8 MB**; 93 commits pushed in 11.6 s; the 7.45 GB tarball was already on
`origin/main` and never resent. Pre-rewrite history preserved under tag
`backup/pre-rewrite-20261107-032658`. **Token leakage scan over the HEAD tree, the working tree and
every unpushed blob: clean.**
- **Script rewritten** (`~/.hermes/config-backup/backup.sh`, via OpenCode + a correction pass): exits
**0 only on verified success**; stamps `.last_backup_ok` **only** after remote verification; decides
the push by **remote-vs-local** rather than by working-tree changes (the exact mechanism that created
the backlog); guards every `git ls-remote` with `|| true` so an unreachable remote reaches
`notify_failure` instead of dying at exit 128 *before* the alert; pushes GitHub as a **parentless
orphan snapshot** (`git commit-tree` from `HEAD^{tree}` — no history, no >100 MB blob, HEAD/branch/
index untouched) as a warning-only secondary target; and reports Telegram/AgentMail delivery
**honestly** (`curl --fail` + `"ok":true`; the real `agentmail` CLI, key read from `.env` at runtime
and passed via the environment, never argv).
- **Verified live 2026-11-07:** real run exit **0** in 47 s; `git rev-parse HEAD` ==
`git ls-remote origin refs/heads/main` == `a50a5170c1c0e2af5235fa63a95a732e2d780703`;
`rev-list --count origin/main..main` = **0**; Gitea API `main` = `a50a5170c1c0`, updated
`2026-11-07T05:10:46+08:00`; `.last_backup_ok` refreshed (was stale from 2026-11-06 11:00). GitHub
mirror now carries a **0-parent** orphan snapshot `faed9fb5c677` (30,086 files, largest blob 23.5 MB)
with `master` untouched at `c99c94e6b13b` — **no data lost**, since GitHub's tree was a strict subset
of local HEAD (only `BACKUP-INDEX.md` added). AgentMail and Telegram alert paths each exercised
directly (AgentMail self-test to the agent's own inbox; Telegram `getMe`, no message sent to Anthony).
Sandbox E2E: unreachable remote → exit 1 + alert + health **not** stamped; local-ahead-with-clean-tree
→ pushed + verified; already-up-to-date → no push, exit 0.
- **Pages touched:** [[incidents/2026-08-08-gitea-config-backup-push-dead]] (status → resolved, full
Resolution section added), [[infrastructure/backups]] (⚠ → ✅, re-verified 2026-11-07),
[[systems/scheduled-tasks]] (job row → RESOLVED), this log.
- **Note — unrelated pre-existing state:** the 03:00 `wiki-stocktake-maintenance` slot **failed** with
`RuntimeError: Cron job 'wiki-stocktake-maintenance' lost its durable fire claim ownership`
(failure_streak 2), leaving its stocktake edits uncommitted in the working tree. Those edits are
included in this commit so the tree is not left dirty.