resolve: Gitea config backup push dead since 2026-08-08 — 92-commit backlog drained, script hardened to fail loudly
- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo - Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved) - Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB - Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification, decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree, honest AgentMail/Telegram delivery reporting - Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed - Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated - Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with 'lost its durable fire claim ownership' and left the tree dirty)
This commit is contained in:
@@ -2226,3 +2226,107 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted*
|
||||
[[current-state]] (+frontmatter, +Recent Changes block), this index (+frontmatter, +summary, +incident
|
||||
bullet), [[runbooks/job-radar]] (+concurrency subsection), this log, and the new incident page. Vault
|
||||
mirror `/home/hermes/wiki` updated (both conventions).
|
||||
## 2026-11-07 03:0x AWST - stocktake (wiki-stocktake-maintenance)
|
||||
|
||||
- **Script:** `wiki-maintain.sh` (repo, canonical) run clean — skills-index regenerated (**302 skills /
|
||||
68 categories**), freshness report written to `raw/stocktake/freshness-20261107.txt`, Quartz rebuilt
|
||||
(**769 files**) and deployed to `public/`, `hermes-wiki-serve.service` restarted.
|
||||
- **Freshness:** **0 stale, 125 fresh, 3 no-date of 128 pages** — no page is past the 21-day threshold.
|
||||
(`log.md` has no `updated` date and lands in the no-date bucket, which is correct for an append-only log.)
|
||||
**Freshness is not accuracy**, so a live-vs-page diff was run against every page carrying live claims.
|
||||
- **Tailscale (infrastructure/tailscale.md):** table rebuilt from `tailscale status --json`. **1 state
|
||||
transition — `octop` (`100.92.163.122`) is back online** (peer record `Online: true`; `tailscale ping` →
|
||||
`pong from octop via DERP(syd) in 98ms`); it is still **no LXC or VM on either hypervisor**, so it is an
|
||||
orphaned tailnet peer, not an estate change. **9 offline ages refreshed (+1d each).** All **51 live IPv4
|
||||
peers** match the table's IPs exactly — **no missing peers, no orphan rows, no IP mismatches** (52 rows =
|
||||
51 peers + `hermes`; the ~23 IPv6-only `funnel-ingress-node` peers excluded by design). The CT460 Serve
|
||||
route table still matches `tailscale serve status` **exactly (12 paths on :443)**; **both DEAD targets
|
||||
re-confirmed dead** (no listener on `8769` `/agent-estate`, none on `9123` `/synergy-prep`), every other
|
||||
Serve target has a live listener, and `/gitea` (`100.82.100.128:3000`) answers **HTTP 200**.
|
||||
- **Proxmox (infrastructure/proxmox.md):** ⚠ **`m93p` runs 20 LXCs, not 19 — CT107 `mem0` was missing**
|
||||
(running; `mem0-mcp.service` active on `:8890`; `pct exec 107 -- tailscale ip -4` → `100.73.94.53`).
|
||||
Added to the table; the 2026-10-31 reading of 19 was already short for the same reason. Live now reads
|
||||
**20 LXCs — 18 running, 2 stopped** (CT998, CT1264). `pve` still matches exactly: **17 LXCs + 4 VMs**.
|
||||
Both hosts run the identical **`pve-manager 9.2.10`** build.
|
||||
- **⚠ `m93p` uptime claim was wrong.** The page read *"up 1 day, 12:02 … rebooted ~2026-10-30 16:0x"*; live
|
||||
`uptime -s` shows the host **booted 2026-11-05 21:19:52** (up **1 day, 7:18**, load **0.76 / 1.08 / 1.24** — normal).
|
||||
The 2026-10-30 reboot was real but belonged to **`pve`** (boot **2026-10-30 16:24:00**).
|
||||
- **Storage refreshed on both hosts** against live `pvesm status`: `pve` `ssd0` **76.43 → 78.29 %**
|
||||
(~152 GiB free), `SSD1` **64.32 → 65.46 %**, `fred-qnap` **82.77 → 84.21 %**, `local` **79.66 → 81.62 %**,
|
||||
`local-lvm` **13.75 → 14.25 %**; `m93p` iMac shares + `local` **60.25 → 76.21 %**, `nfs-proxmox-*`
|
||||
**82.78 → 84.21 %**, `nfs-ssd0-39` **76.43 → 78.29 %**, `omb-internal` **57.92 → 64.66 %**.
|
||||
- **Docker (infrastructure/docker-services.md):** set-wise diff against live `docker ps` on CT200/201/333 —
|
||||
CT200 **61 running / 68 total** exact, CT201 **20 / 26** exact, CT333 **8 / 9** exact, **but CT333's
|
||||
enumerated list still opened with `flamboyant_mahavira`, which is not running** (live name
|
||||
**`nostalgic_keldysh`**, created 2026-11-05 22:15:57). Corrected. CT460 `ignis` running.
|
||||
- **Cross-page corrections:** [[infrastructure/network-map]] `/gitea` was annotated *"Gitea (ct207 tailnet
|
||||
IP)"* — it is **CT221 on `m93p`** (ct207 is `100.82.40.23`); also its `m93p` PVE version read *9.2.0*
|
||||
(both hosts are 9.2.10). [[agents/hermes-chaos]] read *"`v0.22.3` … `5b147ab036d6` — the same build as
|
||||
CT460"*, which contradicted itself: live is **`v0.22.5` … `bf0203a794a0`**, now genuinely identical to
|
||||
CT460's (`hermes update --check` reports **v0.23.0** available). [[agents/hermes-production]]'s
|
||||
*"2 cron jobs still pin `custom:omni`"* is **no longer true** — live `jobs.json` has **0** on the bare key
|
||||
and **21** on `custom:omni-101`.
|
||||
- **Fleet re-verified unchanged:** CT460 **29 jobs (26 enabled / 3 disabled)**; CT450 **25 jobs
|
||||
(13 enabled / 12 disabled)** with **9 shared names** — confirmed by a live set-diff of both `jobs.json`.
|
||||
A keyed `deepseek-paid-smart` chat probe against `omniroute-1` (`100.78.63.20:20128`) returns **HTTP 200**;
|
||||
`config.yaml` still defaults to `custom:omni-101`.
|
||||
- **Pages touched:** [[infrastructure/tailscale]], [[infrastructure/proxmox]], [[infrastructure/hosts]],
|
||||
[[infrastructure/network-map]], [[infrastructure/docker-services]], [[agents/hermes-production]],
|
||||
[[agents/hermes-chaos]], [[current-state]], [[index]], this log. No new incident page — no incident was
|
||||
found; every drift above was a documentation drift corrected against a live probe.
|
||||
|
||||
|
||||
## 2026-11-07 05:1x AWST - incident resolution: Gitea config backup push dead since 2026-08-08
|
||||
|
||||
- **Anthony's ask:** "fix this" on the failing backup alert — with a pointer to check Notion for the API key.
|
||||
- **[[incidents/2026-08-08-gitea-config-backup-push-dead]] is now `status: resolved`.** The nightly
|
||||
`e27da6c34bbb` config backup had been committing locally and failing to push **every night since
|
||||
2026-08-08** — **92 commits** stranded — while its cron row read `last_status: ok`.
|
||||
- **Four independent root causes, all confirmed against live systems:**
|
||||
1. **No Gitea credential.** `origin` was HTTPS with no userinfo; the global `credential.helper=store`
|
||||
file `~/.git-credentials` held only a `github.com` entry. The token in the Notion vault was
|
||||
**valid** and byte-identical to the one the wiki repo carries in its remote URL — it simply was
|
||||
never wired into the backup repo. Fixed with a host-scoped `git credential approve` entry (not
|
||||
embedded in the URL); `git ls-remote origin` now authenticates with `GIT_TERMINAL_PROMPT=0`.
|
||||
2. **`backup.sh` had no `exit` statement at all** — it always returned 0, so the scheduler's
|
||||
`success = (returncode == 0)` recorded `ok` on every failed night. The scheduler was correct;
|
||||
**the script was the liar**. This is why the outage ran 92 days without a job-level red.
|
||||
3. **A 1.76 GB `checkpoints/` backlog** in the unpushed range (13 blobs > 100 MB) — junk that
|
||||
`.gitignore` already excluded for *new* commits but that history still carried.
|
||||
4. **Both secondary legs broken.** The `github` remote did not exist, and the fallback pushed a
|
||||
branch's **full ancestry**, which contains a 7.45 GB tarball blob — so GitHub's 100 MB file
|
||||
limit could never accept it. The AgentMail alert called a `python3 send_email.py` that no longer
|
||||
exists, via a `chaos-agent@agentmail.to` inbox that no longer exists.
|
||||
- **Backlog drained:** `git filter-repo --partial --refs origin/main..main --path checkpoints
|
||||
--invert-paths --prune-empty never --force` stripped `checkpoints/` from the **unpushed range only**
|
||||
(`origin/main` untouched, push stayed a fast-forward), then `checkpoints/` was untracked at the tip.
|
||||
Push set **~2 GB → 264.8 MB**; 93 commits pushed in 11.6 s; the 7.45 GB tarball was already on
|
||||
`origin/main` and never resent. Pre-rewrite history preserved under tag
|
||||
`backup/pre-rewrite-20261107-032658`. **Token leakage scan over the HEAD tree, the working tree and
|
||||
every unpushed blob: clean.**
|
||||
- **Script rewritten** (`~/.hermes/config-backup/backup.sh`, via OpenCode + a correction pass): exits
|
||||
**0 only on verified success**; stamps `.last_backup_ok` **only** after remote verification; decides
|
||||
the push by **remote-vs-local** rather than by working-tree changes (the exact mechanism that created
|
||||
the backlog); guards every `git ls-remote` with `|| true` so an unreachable remote reaches
|
||||
`notify_failure` instead of dying at exit 128 *before* the alert; pushes GitHub as a **parentless
|
||||
orphan snapshot** (`git commit-tree` from `HEAD^{tree}` — no history, no >100 MB blob, HEAD/branch/
|
||||
index untouched) as a warning-only secondary target; and reports Telegram/AgentMail delivery
|
||||
**honestly** (`curl --fail` + `"ok":true`; the real `agentmail` CLI, key read from `.env` at runtime
|
||||
and passed via the environment, never argv).
|
||||
- **Verified live 2026-11-07:** real run exit **0** in 47 s; `git rev-parse HEAD` ==
|
||||
`git ls-remote origin refs/heads/main` == `a50a5170c1c0e2af5235fa63a95a732e2d780703`;
|
||||
`rev-list --count origin/main..main` = **0**; Gitea API `main` = `a50a5170c1c0`, updated
|
||||
`2026-11-07T05:10:46+08:00`; `.last_backup_ok` refreshed (was stale from 2026-11-06 11:00). GitHub
|
||||
mirror now carries a **0-parent** orphan snapshot `faed9fb5c677` (30,086 files, largest blob 23.5 MB)
|
||||
with `master` untouched at `c99c94e6b13b` — **no data lost**, since GitHub's tree was a strict subset
|
||||
of local HEAD (only `BACKUP-INDEX.md` added). AgentMail and Telegram alert paths each exercised
|
||||
directly (AgentMail self-test to the agent's own inbox; Telegram `getMe`, no message sent to Anthony).
|
||||
Sandbox E2E: unreachable remote → exit 1 + alert + health **not** stamped; local-ahead-with-clean-tree
|
||||
→ pushed + verified; already-up-to-date → no push, exit 0.
|
||||
- **Pages touched:** [[incidents/2026-08-08-gitea-config-backup-push-dead]] (status → resolved, full
|
||||
Resolution section added), [[infrastructure/backups]] (⚠ → ✅, re-verified 2026-11-07),
|
||||
[[systems/scheduled-tasks]] (job row → RESOLVED), this log.
|
||||
- **Note — unrelated pre-existing state:** the 03:00 `wiki-stocktake-maintenance` slot **failed** with
|
||||
`RuntimeError: Cron job 'wiki-stocktake-maintenance' lost its durable fire claim ownership`
|
||||
(failure_streak 2), leaving its stocktake edits uncommitted in the working tree. Those edits are
|
||||
included in this commit so the tree is not left dirty.
|
||||
|
||||
Reference in New Issue
Block a user