resolve: Gitea config backup push dead since 2026-08-08 — 92-commit backlog drained, script hardened to fail loudly

- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo
- Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved)
- Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB
- Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification,
  decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree,
  honest AgentMail/Telegram delivery reporting
- Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed
- Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated
- Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with
  'lost its durable fire claim ownership' and left the tree dirty)
This commit is contained in:
Tony0410
2026-11-07 05:28:39 +08:00
parent b820993ecc
commit 0b4ff25fd2
14 changed files with 370 additions and 110 deletions
+36 -26
View File
@@ -3,8 +3,8 @@ title: Backups
type: infrastructure
status: active
created: 2026-07-22
updated: 2026-11-04
verified_on: 2026-11-04
updated: 2026-11-07
verified_on: 2026-11-07
confidence: medium
tags: [infrastructure, backups]
sources: [raw/configs/hermes-config-sanitized.txt]
@@ -54,7 +54,7 @@ How Hermes's own configuration/state gets backed up, and what's missing.
local snapshot and no off-box copy** — the Gitea mirror below has been dead since 2026-08-08.
→ [[systems/hermes-agent]]
## ⚠ Scheduled config backup — the Gitea mirror has been dead since 2026-08-08
## ✅ Scheduled config backup — RESOLVED 2026-11-07 (was dead since 2026-08-08)
`e27da6c34bbb` (**Hermes Config Backup to Gitea**, daily `0 11 * * *`, `hermes-config-backup.sh`) **commits
locally every night and fails to push every night — while its cron row still reads `last_status: ok` and its
@@ -66,31 +66,41 @@ helper — so the non-interactive push dies on
`--- Backup complete: FAIL ---`. Full evidence and the proposed fix:
[[incidents/2026-08-08-gitea-config-backup-push-dead]].
**Consequence for this page: everything above describes a topology that has not carried `~/.hermes` off the box
since 2026-08-07.** The 2026-07-22 full zip and the wiki/authoring repos remain the only recent off-box copies.
Telegram failure notices *have* been firing daily (last: 2026-10-21 11:00:14, `message_id 8449`); the AgentMail
notification step failed.
> **RESOLVED 2026-11-07.** The 92-commit backlog is drained and pushed to Gitea, and the script has been
> hardened so the cron row can no longer report a false green. Full record:
> [[incidents/2026-08-08-gitea-config-backup-push-dead#Resolution 2026-11-07]].
**Re-verified 2026-11-04 — still dead, and the drift is now 89 commits.** Live `git -C ~/.hermes/config-backup`:
`origin` = `https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` (**unchanged** — still HTTPS with
no userinfo and no credential helper, so the credential root cause has not been addressed); `rev-list --count
origin/main..HEAD` = **89**; `log origin/main -1` = **2026-08-07 11:00:47 +0800 1e1eb208** (`🦝 nightly backup:
2026-08-07 11:00:29 AWST`) — i.e. **nothing has reached Gitea in 89 days**. `backup.log` confirms the nightly
failure is current, not historical: the last two runs read
`[2026-11-02 11:00:32 AWST] ✗ Push to Gitea FAILED` / `--- Backup complete: FAIL ---` and
`[2026-11-03 11:00:31 AWST] ✗ Push to Gitea FAILED` / `--- Backup complete: FAIL ---`.
**A second, separate push failure is now also present:** the script attempts a `github` remote that **does not
exist** — `fatal: 'github' does not appear to be a git repository` → `✗ Push to GitHub FAILED`. So both off-box
targets are broken; the GitHub leg is a missing-remote error rather than a credential error, and would need
`git remote add github …` with credentials before it can work at all.
**The cron row still lies:** `e27da6c34bbb` reads `enabled: true, last_status: ok` in `jobs.json` while every
nightly execution has failed the push — the classic green-row/dead-artifact pattern.
**What was wrong (four independent causes):** (1) no Gitea credential — `origin` was HTTPS and
`~/.git-credentials` held only a `github.com` entry, so the non-interactive push could not authenticate;
(2) **`backup.sh` had no `exit` statement at all** — it always returned 0, so the scheduler's
`success = (returncode == 0)` recorded `ok` on every failed night (the scheduler was correct; the script
was the liar); (3) a 1.76 GB `checkpoints/` backlog sat in the unpushed range, 13 blobs over 100 MB;
(4) both secondary legs were broken — the `github` remote did not exist, and its fallback pushed a
branch's full ancestry containing a 7.45 GB tarball blob that GitHub's 100 MB limit could never accept.
**Full-backup gap unchanged:** the one full `hermes backup` zip on disk is still
`/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, 22 Jul) — **now 105 days old**. No new full zip has
been taken. `crontab -l -u hermes` is still empty and there are still **0 user systemd timers**, so nothing
schedules the full zip; only `e5ed215fc91c` (**Mnemosyne DB Backup**, weekly `0 3 * * 0`) and `e27da6c34bbb`
(config backup, daily) exist in the cron fleet, and neither takes a full `hermes backup`.
**What was done:** host-scoped Gitea credential stored via `git credential approve`; `checkpoints/`
stripped from the unpushed range with `git filter-repo --partial` (push set ~2 GB → 264.8 MB, pre-rewrite
history preserved under tag `backup/pre-rewrite-20261107-032658`); the script rewritten to exit non-zero on
failure, stamp `.last_backup_ok` only after remote verification, decide the push by **remote-vs-local**
rather than by working-tree changes, guard every `ls-remote` with `|| true` so an unreachable remote still
reaches the alert path, push GitHub as a **parentless orphan snapshot**, and report Telegram/AgentMail
delivery honestly (`curl --fail` + `"ok":true`; real `agentmail` CLI).
**Re-verified 2026-11-07 — live and green.** Real run exit **0** in 47 s; `git rev-parse HEAD` and
`git ls-remote origin refs/heads/main` both `a50a5170c1c0e2af5235fa63a95a732e2d780703`;
`rev-list --count origin/main..main` = **0**; Gitea API confirms `main` at `a50a5170c1c0`, updated
`2026-11-07T05:10:46+08:00`; `.last_backup_ok` refreshed (was stale from 2026-11-06 11:00). GitHub mirror
now carries a parentless orphan snapshot (`faed9fb5c677`, 0 parents, 30,086 files, largest blob 23.5 MB) —
no data lost, since GitHub's tree was a strict subset of local HEAD. The AgentMail and Telegram alert
paths were each exercised directly and report honestly.
**Consequence for this page: the topology above carries `~/.hermes` off the box again.** The one full
`hermes backup` zip on disk is still `/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, 22 Jul) —
**now 108 days old**, and no new full zip has been taken. `crontab -l -u hermes` is still empty and there
are still **0 user systemd timers**, so nothing schedules the full zip; only `e5ed215fc91c`
(**Mnemosyne DB Backup**, weekly `0 3 * * 0`) and `e27da6c34bbb` (config backup, daily) exist in the cron
fleet, and neither takes a full `hermes backup`. The nightly config backup is a *config* mirror, not a
full-home backup — the gap on the full zip is unchanged.
## Relevant runbooks
- [[runbooks/backup-wiki]]