- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo - Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved) - Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB - Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification, decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree, honest AgentMail/Telegram delivery reporting - Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed - Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated - Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with 'lost its durable fire claim ownership' and left the tree dirty)
7.6 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | |||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Backups | infrastructure | active | 2026-07-22 | 2026-11-07 | 2026-11-07 | medium |
|
|
Backups
Purpose
How Hermes's own configuration/state gets backed up, and what's missing.
Current mechanism
hermes backupCLI (verified viahermes backup --help): creates a zip of the entire Hermes config, skills, sessions, and data (excludes the hermes-agent codebase itself).--quick/-q: fast snapshot of just critical state (config,state.db,.env,auth,cron)--label/-l: label for a quick snapshot--output/-o: custom output path (default~/hermes-backup-<timestamp>.zip)
- Config keys (
updates:section of~/.hermes/config.yaml):pre_update_backup: false— Hermes does not automatically back up before self-updatesbackup_keep: 5— retention count (applies to config.yaml.bak* rotation seen during updates, e.g. in runbooks/update-hermes-safely)non_interactive_local_changes: stash
Evidence on disk (2026-07-22)
- One manual full backup exists:
/home/hermes/hermes-backup-2026-07-22-180830.zip(~235 MB), same day as wiki creation. /home/hermes/wiki-personal-backups/holds an earlier snapshot of the wiki itself (SCHEMA.md,current-state.md,index.md,log.mdfrom 17:34, before later same-day wiki edits) — see runbooks/backup-wiki./home/hermes/hermes-wiki/exists but is empty — likely a staging/rename artifact from wiki setup, not an active backup target.- A dedicated memory-backup script exists:
/home/hermes/.hermes/scripts/mnemosyne-backup.sh— see systems/mnemosyne.
Known gaps
- No scheduled backup job found:
crontab -l -u hermesreturns empty, and no systemd timer referenceshermesorbackup. The one full backup on disk appears to have been triggered manually, not on a schedule.
Re-verified 2026-10-20 — the gap is unchanged and the single backup is now 90 days old.
crontab -l -u hermes→ "no crontab for hermes".systemctl --user list-timers --all→ 0 timers. The only timer on the box is the system-leveldpkg-db-backup.timer— nothing Hermes-related.- The one full backup is still
/home/hermes/hermes-backup-2026-07-22-180830.zip(246 MB, dated 22 Jul). No new full backup has been taken in the 90 days since./home/hermes/hermes-wiki/is still empty;/home/hermes/wiki-personal-backups/still holds only the four original 22 Jul files (current-state.md,index.md,log.md,SCHEMA.md).~/.hermes/scripts/mnemosyne-backup.shstill exists (unchanged since 3 Jul).updates:inconfig.yamlstill readspre_update_backup: false,backup_keep: 5,non_interactive_local_changes: stash.The separate wiki backup is also unrun:
/home/hermes/wiki-backups/does not exist, so the tarball step in runbooks/backup-wiki has never been executed (the runbook'smkdir -pwould create it on first use).
pre_update_backup: falsemeans an in-placehermes updatewill not auto-snapshot first — the manual step in runbooks/update-hermes-safely step 1 is currently the only safety net. Exercised on 2026-10-21: the 22:47hermes update(0.22.2 → 0.22.3) recorded itssnapshotstage asskippedwith reasondisabled by updates.pre_update_backup (mode: off), so that update shipped with no local snapshot and no off-box copy — the Gitea mirror below has been dead since 2026-08-08. → systems/hermes-agent
✅ Scheduled config backup — RESOLVED 2026-11-07 (was dead since 2026-08-08)
e27da6c34bbb (Hermes Config Backup to Gitea, daily 0 11 * * *, hermes-config-backup.sh) commits
locally every night and fails to push every night — while its cron row still reads last_status: ok and its
execution row reads completed. Last commit that reached Gitea: 2026-08-07 11:00:47; the local repo is
89 commits ahead of origin/main (was 76 on 2026-10-22 — 13 more nights of local-only commits since). Root cause is a credential: origin is
https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git — HTTPS, no userinfo, no credential
helper — so the non-interactive push dies on
fatal: could not read Username … No such device or address → ✗ Push to Gitea FAILED →
--- Backup complete: FAIL ---. Full evidence and the proposed fix:
incidents/2026-08-08-gitea-config-backup-push-dead.
RESOLVED 2026-11-07. The 92-commit backlog is drained and pushed to Gitea, and the script has been hardened so the cron row can no longer report a false green. Full record: incidents/2026-08-08-gitea-config-backup-push-dead#Resolution 2026-11-07.
What was wrong (four independent causes): (1) no Gitea credential — origin was HTTPS and
~/.git-credentials held only a github.com entry, so the non-interactive push could not authenticate;
(2) backup.sh had no exit statement at all — it always returned 0, so the scheduler's
success = (returncode == 0) recorded ok on every failed night (the scheduler was correct; the script
was the liar); (3) a 1.76 GB checkpoints/ backlog sat in the unpushed range, 13 blobs over 100 MB;
(4) both secondary legs were broken — the github remote did not exist, and its fallback pushed a
branch's full ancestry containing a 7.45 GB tarball blob that GitHub's 100 MB limit could never accept.
What was done: host-scoped Gitea credential stored via git credential approve; checkpoints/
stripped from the unpushed range with git filter-repo --partial (push set ~2 GB → 264.8 MB, pre-rewrite
history preserved under tag backup/pre-rewrite-20261107-032658); the script rewritten to exit non-zero on
failure, stamp .last_backup_ok only after remote verification, decide the push by remote-vs-local
rather than by working-tree changes, guard every ls-remote with || true so an unreachable remote still
reaches the alert path, push GitHub as a parentless orphan snapshot, and report Telegram/AgentMail
delivery honestly (curl --fail + "ok":true; real agentmail CLI).
Re-verified 2026-11-07 — live and green. Real run exit 0 in 47 s; git rev-parse HEAD and
git ls-remote origin refs/heads/main both a50a5170c1c0e2af5235fa63a95a732e2d780703;
rev-list --count origin/main..main = 0; Gitea API confirms main at a50a5170c1c0, updated
2026-11-07T05:10:46+08:00; .last_backup_ok refreshed (was stale from 2026-11-06 11:00). GitHub mirror
now carries a parentless orphan snapshot (faed9fb5c677, 0 parents, 30,086 files, largest blob 23.5 MB) —
no data lost, since GitHub's tree was a strict subset of local HEAD. The AgentMail and Telegram alert
paths were each exercised directly and report honestly.
Consequence for this page: the topology above carries ~/.hermes off the box again. The one full
hermes backup zip on disk is still /home/hermes/hermes-backup-2026-07-22-180830.zip (246 MB, 22 Jul) —
now 108 days old, and no new full zip has been taken. crontab -l -u hermes is still empty and there
are still 0 user systemd timers, so nothing schedules the full zip; only e5ed215fc91c
(Mnemosyne DB Backup, weekly 0 3 * * 0) and e27da6c34bbb (config backup, daily) exist in the cron
fleet, and neither takes a full hermes backup. The nightly config backup is a config mirror, not a
full-home backup — the gap on the full zip is unchanged.