resolve: Gitea config backup push dead since 2026-08-08 — 92-commit backlog drained, script hardened to fail loudly

- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo
- Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved)
- Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB
- Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification,
  decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree,
  honest AgentMail/Telegram delivery reporting
- Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed
- Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated
- Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with
  'lost its durable fire claim ownership' and left the tree dirty)
This commit is contained in:
Tony0410
2026-11-07 05:28:39 +08:00
parent b820993ecc
commit 0b4ff25fd2
14 changed files with 370 additions and 110 deletions
+39
View File
@@ -14,6 +14,45 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
# Current State
Updated: 2026-11-07
## Recent Changes (2026-11-07, stocktake; live probes 2026-11-07 03:1x-03:2x AWST)
- **The tailnet lost a peer and the m93p guest list lost a guest — both corrections came from live probes, not the age scan.**
[[infrastructure/tailscale]] was rebuilt from `tailscale status --json`: **`octop` (`100.92.163.122`) is back
online** (peer record `Online: true`, `tailscale ping` → `pong from octop via DERP(syd) in 98ms`; it remains
**no LXC or VM on either hypervisor** — an orphaned tailnet peer), and the **9 remaining offline ages** were
refreshed (+1d each). All **51 live IPv4 peers** match the table's IPs exactly, with no missing peers and no
orphan rows (**52 rows** = 51 peers + `hermes` itself). The CT460 Serve route table still matches
`tailscale serve status` **exactly (12 paths on :443)**; both previously-flagged dead targets were
**re-confirmed dead** — still no listener on `8769` (`/agent-estate`) or `9123` (`/synergy-prep`).
- **⚠ `m93p` actually runs 20 LXCs, not 19 — CT107 (`mem0`) was missing from the Proxmox page.**
Live `pct list` on `m93p` returns **20 LXCs — 18 running, 2 stopped** (CT998 `nightbridge`, CT1264
`openmausbot`). The omitted guest is **CT107 `mem0`** (`mem0-mcp.service` active on `:8890`,
`pct exec 107 -- tailscale ip -4` → `100.73.94.53`). Added to [[infrastructure/proxmox]]; the
**2026-10-31 reading of 19 was already short for the same reason**. `pve` still returns exactly
**17 LXCs + 4 VMs**, and both hosts still run the identical `pve-manager 9.2.10` build.
- **⚠ `m93p`'s uptime claim was wrong.** The page read *"up 1 day, 12:02 … rebooted ~2026-10-30 16:0x"*.
Live `uptime -s` shows the host **booted 2026-11-05 21:19:52** (up **1 day, 7:18** at probe time, load
**0.76 / 1.08 / 1.24** — normal). The 2026-10-30 reboot was real but belonged to **`pve`**
(boot **2026-10-30 16:24:00**), not `m93p`.
- **Storage percentages refreshed on both hosts** against live `pvesm status` — `pve`'s `ssd0` moved
**76.43 % → 78.29 %** (~152 GiB free), `fred-qnap` **82.77 % → 84.21 %**, `local` **79.66 % → 81.62 %**;
on `m93p` the iMac shares and `local` moved **60.25 % → 76.21 %** and `omb-internal`
**57.92 % → 64.66 %**.
- **Docker inventories diffed set-wise against live `docker ps` on all three hosts.** CT200 matches
**61 running / 68 total** exactly; CT201 matches **20 / 26** exactly; CT333 matches **8 / 9** — but its
**enumerated list still opened with `flamboyant_mahavira`, which is not running** (live name is
**`nostalgic_keldysh`**, created 2026-11-05 22:15:57). Corrected in [[infrastructure/docker-services]].
- **Two small cross-page corrections.** [[infrastructure/network-map]]'s `/gitea` route was annotated
*"Gitea (ct207 tailnet IP)"* — it is **CT221 on `m93p`** (`100.82.100.128`; ct207's own IP is
`100.82.40.23`); the target answers **HTTP 200**. [[agents/hermes-chaos]]'s version line read
*"`v0.22.3` … local `5b147ab036d6` — the same build as CT460"*, which contradicted itself: live
`hermes --version` on CT450 is **`v0.22.5` … `bf0203a794a0`**, now genuinely identical to CT460's.
[[agents/hermes-production]]'s claim that *2 cron jobs still pin `custom:omni`* is **no longer true** —
live `jobs.json` shows **0** jobs on the bare key and **21** on `custom:omni-101`.
- **Cron fleet re-verified unchanged:** **29 jobs (26 enabled / 3 disabled)**, matching the nightly audit
hours earlier. CT450's fleet is **25 jobs (13 enabled / 12 disabled)** with **9 shared names** — confirmed
by a live set-diff of both `jobs.json` files. → [[systems/scheduled-tasks]]
## Recent Changes (2026-11-07, nightly audit; live probes 2026-11-07 00:3x-00:5x AWST)
- **A red row with no alert: the Job Radar's 11:30 slot was discarded as stale while a concurrent fire