Files
agent-estate-wiki/infrastructure/backups.md
T
Tony0410 0b4ff25fd2 resolve: Gitea config backup push dead since 2026-08-08 — 92-commit backlog drained, script hardened to fail loudly
- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo
- Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved)
- Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB
- Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification,
  decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree,
  honest AgentMail/Telegram delivery reporting
- Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed
- Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated
- Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with
  'lost its durable fire claim ownership' and left the tree dirty)
2026-11-07 05:28:39 +08:00

7.6 KiB

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Backups infrastructure active 2026-07-22 2026-11-07 2026-11-07 medium
infrastructure
backups
raw/configs/hermes-config-sanitized.txt

Backups

Purpose

How Hermes's own configuration/state gets backed up, and what's missing.

Current mechanism

  • hermes backup CLI (verified via hermes backup --help): creates a zip of the entire Hermes config, skills, sessions, and data (excludes the hermes-agent codebase itself).
    • --quick / -q: fast snapshot of just critical state (config, state.db, .env, auth, cron)
    • --label / -l: label for a quick snapshot
    • --output / -o: custom output path (default ~/hermes-backup-<timestamp>.zip)
  • Config keys (updates: section of ~/.hermes/config.yaml):
    • pre_update_backup: false — Hermes does not automatically back up before self-updates
    • backup_keep: 5 — retention count (applies to config.yaml.bak* rotation seen during updates, e.g. in runbooks/update-hermes-safely)
    • non_interactive_local_changes: stash

Evidence on disk (2026-07-22)

  • One manual full backup exists: /home/hermes/hermes-backup-2026-07-22-180830.zip (~235 MB), same day as wiki creation.
  • /home/hermes/wiki-personal-backups/ holds an earlier snapshot of the wiki itself (SCHEMA.md, current-state.md, index.md, log.md from 17:34, before later same-day wiki edits) — see runbooks/backup-wiki.
  • /home/hermes/hermes-wiki/ exists but is empty — likely a staging/rename artifact from wiki setup, not an active backup target.
  • A dedicated memory-backup script exists: /home/hermes/.hermes/scripts/mnemosyne-backup.sh — see systems/mnemosyne.

Known gaps

  • No scheduled backup job found: crontab -l -u hermes returns empty, and no systemd timer references hermes or backup. The one full backup on disk appears to have been triggered manually, not on a schedule.

Re-verified 2026-10-20 — the gap is unchanged and the single backup is now 90 days old.

  • crontab -l -u hermes → "no crontab for hermes". systemctl --user list-timers --all → 0 timers. The only timer on the box is the system-level dpkg-db-backup.timer — nothing Hermes-related.
  • The one full backup is still /home/hermes/hermes-backup-2026-07-22-180830.zip (246 MB, dated 22 Jul). No new full backup has been taken in the 90 days since.
  • /home/hermes/hermes-wiki/ is still empty; /home/hermes/wiki-personal-backups/ still holds only the four original 22 Jul files (current-state.md, index.md, log.md, SCHEMA.md).
  • ~/.hermes/scripts/mnemosyne-backup.sh still exists (unchanged since 3 Jul).
  • updates: in config.yaml still reads pre_update_backup: false, backup_keep: 5, non_interactive_local_changes: stash.

The separate wiki backup is also unrun: /home/hermes/wiki-backups/ does not exist, so the tarball step in runbooks/backup-wiki has never been executed (the runbook's mkdir -p would create it on first use).

  • pre_update_backup: false means an in-place hermes update will not auto-snapshot first — the manual step in runbooks/update-hermes-safely step 1 is currently the only safety net. Exercised on 2026-10-21: the 22:47 hermes update (0.22.2 → 0.22.3) recorded its snapshot stage as skipped with reason disabled by updates.pre_update_backup (mode: off), so that update shipped with no local snapshot and no off-box copy — the Gitea mirror below has been dead since 2026-08-08. → systems/hermes-agent

✅ Scheduled config backup — RESOLVED 2026-11-07 (was dead since 2026-08-08)

e27da6c34bbb (Hermes Config Backup to Gitea, daily 0 11 * * *, hermes-config-backup.sh) commits locally every night and fails to push every night — while its cron row still reads last_status: ok and its execution row reads completed. Last commit that reached Gitea: 2026-08-07 11:00:47; the local repo is 89 commits ahead of origin/main (was 76 on 2026-10-22 — 13 more nights of local-only commits since). Root cause is a credential: origin is https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git — HTTPS, no userinfo, no credential helper — so the non-interactive push dies on fatal: could not read Username … No such device or address → ✗ Push to Gitea FAILED → --- Backup complete: FAIL ---. Full evidence and the proposed fix: incidents/2026-08-08-gitea-config-backup-push-dead.

RESOLVED 2026-11-07. The 92-commit backlog is drained and pushed to Gitea, and the script has been hardened so the cron row can no longer report a false green. Full record: incidents/2026-08-08-gitea-config-backup-push-dead#Resolution 2026-11-07.

What was wrong (four independent causes): (1) no Gitea credential — origin was HTTPS and ~/.git-credentials held only a github.com entry, so the non-interactive push could not authenticate; (2) backup.sh had no exit statement at all — it always returned 0, so the scheduler's success = (returncode == 0) recorded ok on every failed night (the scheduler was correct; the script was the liar); (3) a 1.76 GB checkpoints/ backlog sat in the unpushed range, 13 blobs over 100 MB; (4) both secondary legs were broken — the github remote did not exist, and its fallback pushed a branch's full ancestry containing a 7.45 GB tarball blob that GitHub's 100 MB limit could never accept.

What was done: host-scoped Gitea credential stored via git credential approve; checkpoints/ stripped from the unpushed range with git filter-repo --partial (push set ~2 GB → 264.8 MB, pre-rewrite history preserved under tag backup/pre-rewrite-20261107-032658); the script rewritten to exit non-zero on failure, stamp .last_backup_ok only after remote verification, decide the push by remote-vs-local rather than by working-tree changes, guard every ls-remote with || true so an unreachable remote still reaches the alert path, push GitHub as a parentless orphan snapshot, and report Telegram/AgentMail delivery honestly (curl --fail + "ok":true; real agentmail CLI).

Re-verified 2026-11-07 — live and green. Real run exit 0 in 47 s; git rev-parse HEAD and git ls-remote origin refs/heads/main both a50a5170c1c0e2af5235fa63a95a732e2d780703; rev-list --count origin/main..main = 0; Gitea API confirms main at a50a5170c1c0, updated 2026-11-07T05:10:46+08:00; .last_backup_ok refreshed (was stale from 2026-11-06 11:00). GitHub mirror now carries a parentless orphan snapshot (faed9fb5c677, 0 parents, 30,086 files, largest blob 23.5 MB) — no data lost, since GitHub's tree was a strict subset of local HEAD. The AgentMail and Telegram alert paths were each exercised directly and report honestly.

Consequence for this page: the topology above carries ~/.hermes off the box again. The one full hermes backup zip on disk is still /home/hermes/hermes-backup-2026-07-22-180830.zip (246 MB, 22 Jul) — now 108 days old, and no new full zip has been taken. crontab -l -u hermes is still empty and there are still 0 user systemd timers, so nothing schedules the full zip; only e5ed215fc91c (Mnemosyne DB Backup, weekly 0 3 * * 0) and e27da6c34bbb (config backup, daily) exist in the cron fleet, and neither takes a full hermes backup. The nightly config backup is a config mirror, not a full-home backup — the gap on the full zip is unchanged.

Relevant runbooks