Files
agent-estate-wiki/infrastructure/backups.md
T
Tony0410 0b4ff25fd2 resolve: Gitea config backup push dead since 2026-08-08 — 92-commit backlog drained, script hardened to fail loudly
- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo
- Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved)
- Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB
- Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification,
  decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree,
  honest AgentMail/Telegram delivery reporting
- Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed
- Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated
- Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with
  'lost its durable fire claim ownership' and left the tree dirty)
2026-11-07 05:28:39 +08:00

112 lines
7.6 KiB
Markdown

---
title: Backups
type: infrastructure
status: active
created: 2026-07-22
updated: 2026-11-07
verified_on: 2026-11-07
confidence: medium
tags: [infrastructure, backups]
sources: [raw/configs/hermes-config-sanitized.txt]
---
# Backups
## Purpose
How Hermes's own configuration/state gets backed up, and what's missing.
## Current mechanism
- **`hermes backup` CLI** (verified via `hermes backup --help`): creates a zip of the entire Hermes config, skills, sessions, and data (excludes the hermes-agent codebase itself).
- `--quick` / `-q`: fast snapshot of just critical state (`config`, `state.db`, `.env`, `auth`, `cron`)
- `--label` / `-l`: label for a quick snapshot
- `--output` / `-o`: custom output path (default `~/hermes-backup-<timestamp>.zip`)
- **Config keys** (`updates:` section of `~/.hermes/config.yaml`):
- `pre_update_backup: false` — Hermes does **not** automatically back up before self-updates
- `backup_keep: 5` — retention count (applies to config.yaml.bak* rotation seen during updates, e.g. in [[runbooks/update-hermes-safely]])
- `non_interactive_local_changes: stash`
## Evidence on disk (2026-07-22)
- One manual full backup exists: `/home/hermes/hermes-backup-2026-07-22-180830.zip` (~235 MB), same day as wiki creation.
- `/home/hermes/wiki-personal-backups/` holds an earlier snapshot of the wiki itself (`SCHEMA.md`, `current-state.md`, `index.md`, `log.md` from 17:34, before later same-day wiki edits) — see [[runbooks/backup-wiki]].
- `/home/hermes/hermes-wiki/` exists but is empty — likely a staging/rename artifact from wiki setup, not an active backup target.
- A dedicated memory-backup script exists: `/home/hermes/.hermes/scripts/mnemosyne-backup.sh` — see [[systems/mnemosyne]].
## Known gaps
- **No scheduled backup job found:** `crontab -l -u hermes` returns empty, and no systemd timer references `hermes` or `backup`. The one full backup on disk appears to have been triggered manually, not on a schedule.
> **Re-verified 2026-10-20 — the gap is unchanged and the single backup is now 90 days old.**
> - `crontab -l -u hermes` → "no crontab for hermes". `systemctl --user list-timers --all` → **0 timers**.
> The only timer on the box is the system-level `dpkg-db-backup.timer` — nothing Hermes-related.
> - The one full backup is still `/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, dated
> 22 Jul). **No new full backup has been taken in the 90 days since.**
> - `/home/hermes/hermes-wiki/` is still **empty**; `/home/hermes/wiki-personal-backups/` still holds
> only the four original 22 Jul files (`current-state.md`, `index.md`, `log.md`, `SCHEMA.md`).
> - `~/.hermes/scripts/mnemosyne-backup.sh` still exists (unchanged since 3 Jul).
> - `updates:` in `config.yaml` still reads `pre_update_backup: false`, `backup_keep: 5`,
> `non_interactive_local_changes: stash`.
>
> **The separate wiki backup is also unrun:** `/home/hermes/wiki-backups/` does not exist, so the
> tarball step in [[runbooks/backup-wiki]] has never been executed (the runbook's `mkdir -p` would
> create it on first use).
- `pre_update_backup: false` means an in-place `hermes update` will not auto-snapshot first — the manual step in [[runbooks/update-hermes-safely]] step 1 is currently the only safety net.
**Exercised on 2026-10-21:** the 22:47 `hermes update` (0.22.2 → 0.22.3) recorded its `snapshot` stage as
`skipped` with reason `disabled by updates.pre_update_backup (mode: off)`, so that update shipped with **no
local snapshot and no off-box copy** — the Gitea mirror below has been dead since 2026-08-08.
→ [[systems/hermes-agent]]
## ✅ Scheduled config backup — RESOLVED 2026-11-07 (was dead since 2026-08-08)
`e27da6c34bbb` (**Hermes Config Backup to Gitea**, daily `0 11 * * *`, `hermes-config-backup.sh`) **commits
locally every night and fails to push every night — while its cron row still reads `last_status: ok` and its
execution row reads `completed`.** Last commit that reached Gitea: **2026-08-07 11:00:47**; the local repo is
**89 commits ahead of `origin/main`** (was 76 on 2026-10-22 — **13 more nights of local-only commits** since). Root cause is a credential: `origin` is
`https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` — HTTPS, no userinfo, no credential
helper — so the non-interactive push dies on
`fatal: could not read Username … No such device or address` → `✗ Push to Gitea FAILED` →
`--- Backup complete: FAIL ---`. Full evidence and the proposed fix:
[[incidents/2026-08-08-gitea-config-backup-push-dead]].
> **RESOLVED 2026-11-07.** The 92-commit backlog is drained and pushed to Gitea, and the script has been
> hardened so the cron row can no longer report a false green. Full record:
> [[incidents/2026-08-08-gitea-config-backup-push-dead#Resolution 2026-11-07]].
**What was wrong (four independent causes):** (1) no Gitea credential — `origin` was HTTPS and
`~/.git-credentials` held only a `github.com` entry, so the non-interactive push could not authenticate;
(2) **`backup.sh` had no `exit` statement at all** — it always returned 0, so the scheduler's
`success = (returncode == 0)` recorded `ok` on every failed night (the scheduler was correct; the script
was the liar); (3) a 1.76 GB `checkpoints/` backlog sat in the unpushed range, 13 blobs over 100 MB;
(4) both secondary legs were broken — the `github` remote did not exist, and its fallback pushed a
branch's full ancestry containing a 7.45 GB tarball blob that GitHub's 100 MB limit could never accept.
**What was done:** host-scoped Gitea credential stored via `git credential approve`; `checkpoints/`
stripped from the unpushed range with `git filter-repo --partial` (push set ~2 GB → 264.8 MB, pre-rewrite
history preserved under tag `backup/pre-rewrite-20261107-032658`); the script rewritten to exit non-zero on
failure, stamp `.last_backup_ok` only after remote verification, decide the push by **remote-vs-local**
rather than by working-tree changes, guard every `ls-remote` with `|| true` so an unreachable remote still
reaches the alert path, push GitHub as a **parentless orphan snapshot**, and report Telegram/AgentMail
delivery honestly (`curl --fail` + `"ok":true`; real `agentmail` CLI).
**Re-verified 2026-11-07 — live and green.** Real run exit **0** in 47 s; `git rev-parse HEAD` and
`git ls-remote origin refs/heads/main` both `a50a5170c1c0e2af5235fa63a95a732e2d780703`;
`rev-list --count origin/main..main` = **0**; Gitea API confirms `main` at `a50a5170c1c0`, updated
`2026-11-07T05:10:46+08:00`; `.last_backup_ok` refreshed (was stale from 2026-11-06 11:00). GitHub mirror
now carries a parentless orphan snapshot (`faed9fb5c677`, 0 parents, 30,086 files, largest blob 23.5 MB) —
no data lost, since GitHub's tree was a strict subset of local HEAD. The AgentMail and Telegram alert
paths were each exercised directly and report honestly.
**Consequence for this page: the topology above carries `~/.hermes` off the box again.** The one full
`hermes backup` zip on disk is still `/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, 22 Jul) —
**now 108 days old**, and no new full zip has been taken. `crontab -l -u hermes` is still empty and there
are still **0 user systemd timers**, so nothing schedules the full zip; only `e5ed215fc91c`
(**Mnemosyne DB Backup**, weekly `0 3 * * 0`) and `e27da6c34bbb` (config backup, daily) exist in the cron
fleet, and neither takes a full `hermes backup`. The nightly config backup is a *config* mirror, not a
full-home backup — the gap on the full zip is unchanged.
## Relevant runbooks
- [[runbooks/backup-wiki]]
- [[runbooks/update-hermes-safely]]
## Related
- [[systems/mnemosyne]]
- [[systems/hermes-agent]]