resolve: Gitea config backup push dead since 2026-08-08 — 92-commit backlog drained, script hardened to fail loudly
- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo - Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved) - Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB - Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification, decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree, honest AgentMail/Telegram delivery reporting - Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed - Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated - Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with 'lost its durable fire claim ownership' and left the tree dirty)
This commit is contained in:
@@ -14,6 +14,45 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
|
||||
# Current State
|
||||
Updated: 2026-11-07
|
||||
|
||||
## Recent Changes (2026-11-07, stocktake; live probes 2026-11-07 03:1x-03:2x AWST)
|
||||
|
||||
- **The tailnet lost a peer and the m93p guest list lost a guest — both corrections came from live probes, not the age scan.**
|
||||
[[infrastructure/tailscale]] was rebuilt from `tailscale status --json`: **`octop` (`100.92.163.122`) is back
|
||||
online** (peer record `Online: true`, `tailscale ping` → `pong from octop via DERP(syd) in 98ms`; it remains
|
||||
**no LXC or VM on either hypervisor** — an orphaned tailnet peer), and the **9 remaining offline ages** were
|
||||
refreshed (+1d each). All **51 live IPv4 peers** match the table's IPs exactly, with no missing peers and no
|
||||
orphan rows (**52 rows** = 51 peers + `hermes` itself). The CT460 Serve route table still matches
|
||||
`tailscale serve status` **exactly (12 paths on :443)**; both previously-flagged dead targets were
|
||||
**re-confirmed dead** — still no listener on `8769` (`/agent-estate`) or `9123` (`/synergy-prep`).
|
||||
- **⚠ `m93p` actually runs 20 LXCs, not 19 — CT107 (`mem0`) was missing from the Proxmox page.**
|
||||
Live `pct list` on `m93p` returns **20 LXCs — 18 running, 2 stopped** (CT998 `nightbridge`, CT1264
|
||||
`openmausbot`). The omitted guest is **CT107 `mem0`** (`mem0-mcp.service` active on `:8890`,
|
||||
`pct exec 107 -- tailscale ip -4` → `100.73.94.53`). Added to [[infrastructure/proxmox]]; the
|
||||
**2026-10-31 reading of 19 was already short for the same reason**. `pve` still returns exactly
|
||||
**17 LXCs + 4 VMs**, and both hosts still run the identical `pve-manager 9.2.10` build.
|
||||
- **⚠ `m93p`'s uptime claim was wrong.** The page read *"up 1 day, 12:02 … rebooted ~2026-10-30 16:0x"*.
|
||||
Live `uptime -s` shows the host **booted 2026-11-05 21:19:52** (up **1 day, 7:18** at probe time, load
|
||||
**0.76 / 1.08 / 1.24** — normal). The 2026-10-30 reboot was real but belonged to **`pve`**
|
||||
(boot **2026-10-30 16:24:00**), not `m93p`.
|
||||
- **Storage percentages refreshed on both hosts** against live `pvesm status` — `pve`'s `ssd0` moved
|
||||
**76.43 % → 78.29 %** (~152 GiB free), `fred-qnap` **82.77 % → 84.21 %**, `local` **79.66 % → 81.62 %**;
|
||||
on `m93p` the iMac shares and `local` moved **60.25 % → 76.21 %** and `omb-internal`
|
||||
**57.92 % → 64.66 %**.
|
||||
- **Docker inventories diffed set-wise against live `docker ps` on all three hosts.** CT200 matches
|
||||
**61 running / 68 total** exactly; CT201 matches **20 / 26** exactly; CT333 matches **8 / 9** — but its
|
||||
**enumerated list still opened with `flamboyant_mahavira`, which is not running** (live name is
|
||||
**`nostalgic_keldysh`**, created 2026-11-05 22:15:57). Corrected in [[infrastructure/docker-services]].
|
||||
- **Two small cross-page corrections.** [[infrastructure/network-map]]'s `/gitea` route was annotated
|
||||
*"Gitea (ct207 tailnet IP)"* — it is **CT221 on `m93p`** (`100.82.100.128`; ct207's own IP is
|
||||
`100.82.40.23`); the target answers **HTTP 200**. [[agents/hermes-chaos]]'s version line read
|
||||
*"`v0.22.3` … local `5b147ab036d6` — the same build as CT460"*, which contradicted itself: live
|
||||
`hermes --version` on CT450 is **`v0.22.5` … `bf0203a794a0`**, now genuinely identical to CT460's.
|
||||
[[agents/hermes-production]]'s claim that *2 cron jobs still pin `custom:omni`* is **no longer true** —
|
||||
live `jobs.json` shows **0** jobs on the bare key and **21** on `custom:omni-101`.
|
||||
- **Cron fleet re-verified unchanged:** **29 jobs (26 enabled / 3 disabled)**, matching the nightly audit
|
||||
hours earlier. CT450's fleet is **25 jobs (13 enabled / 12 disabled)** with **9 shared names** — confirmed
|
||||
by a live set-diff of both `jobs.json` files. → [[systems/scheduled-tasks]]
|
||||
|
||||
## Recent Changes (2026-11-07, nightly audit; live probes 2026-11-07 00:3x-00:5x AWST)
|
||||
|
||||
- **A red row with no alert: the Job Radar's 11:30 slot was discarded as stale while a concurrent fire
|
||||
|
||||
Reference in New Issue
Block a user