102 lines
6.8 KiB
Markdown
102 lines
6.8 KiB
Markdown
---
|
|
title: Backups
|
|
type: infrastructure
|
|
status: active
|
|
created: 2026-07-22
|
|
updated: 2026-11-04
|
|
verified_on: 2026-11-04
|
|
confidence: medium
|
|
tags: [infrastructure, backups]
|
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
|
---
|
|
|
|
# Backups
|
|
|
|
## Purpose
|
|
How Hermes's own configuration/state gets backed up, and what's missing.
|
|
|
|
## Current mechanism
|
|
- **`hermes backup` CLI** (verified via `hermes backup --help`): creates a zip of the entire Hermes config, skills, sessions, and data (excludes the hermes-agent codebase itself).
|
|
- `--quick` / `-q`: fast snapshot of just critical state (`config`, `state.db`, `.env`, `auth`, `cron`)
|
|
- `--label` / `-l`: label for a quick snapshot
|
|
- `--output` / `-o`: custom output path (default `~/hermes-backup-<timestamp>.zip`)
|
|
- **Config keys** (`updates:` section of `~/.hermes/config.yaml`):
|
|
- `pre_update_backup: false` — Hermes does **not** automatically back up before self-updates
|
|
- `backup_keep: 5` — retention count (applies to config.yaml.bak* rotation seen during updates, e.g. in [[runbooks/update-hermes-safely]])
|
|
- `non_interactive_local_changes: stash`
|
|
|
|
## Evidence on disk (2026-07-22)
|
|
- One manual full backup exists: `/home/hermes/hermes-backup-2026-07-22-180830.zip` (~235 MB), same day as wiki creation.
|
|
- `/home/hermes/wiki-personal-backups/` holds an earlier snapshot of the wiki itself (`SCHEMA.md`, `current-state.md`, `index.md`, `log.md` from 17:34, before later same-day wiki edits) — see [[runbooks/backup-wiki]].
|
|
- `/home/hermes/hermes-wiki/` exists but is empty — likely a staging/rename artifact from wiki setup, not an active backup target.
|
|
- A dedicated memory-backup script exists: `/home/hermes/.hermes/scripts/mnemosyne-backup.sh` — see [[systems/mnemosyne]].
|
|
|
|
## Known gaps
|
|
- **No scheduled backup job found:** `crontab -l -u hermes` returns empty, and no systemd timer references `hermes` or `backup`. The one full backup on disk appears to have been triggered manually, not on a schedule.
|
|
|
|
> **Re-verified 2026-10-20 — the gap is unchanged and the single backup is now 90 days old.**
|
|
> - `crontab -l -u hermes` → "no crontab for hermes". `systemctl --user list-timers --all` → **0 timers**.
|
|
> The only timer on the box is the system-level `dpkg-db-backup.timer` — nothing Hermes-related.
|
|
> - The one full backup is still `/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, dated
|
|
> 22 Jul). **No new full backup has been taken in the 90 days since.**
|
|
> - `/home/hermes/hermes-wiki/` is still **empty**; `/home/hermes/wiki-personal-backups/` still holds
|
|
> only the four original 22 Jul files (`current-state.md`, `index.md`, `log.md`, `SCHEMA.md`).
|
|
> - `~/.hermes/scripts/mnemosyne-backup.sh` still exists (unchanged since 3 Jul).
|
|
> - `updates:` in `config.yaml` still reads `pre_update_backup: false`, `backup_keep: 5`,
|
|
> `non_interactive_local_changes: stash`.
|
|
>
|
|
> **The separate wiki backup is also unrun:** `/home/hermes/wiki-backups/` does not exist, so the
|
|
> tarball step in [[runbooks/backup-wiki]] has never been executed (the runbook's `mkdir -p` would
|
|
> create it on first use).
|
|
- `pre_update_backup: false` means an in-place `hermes update` will not auto-snapshot first — the manual step in [[runbooks/update-hermes-safely]] step 1 is currently the only safety net.
|
|
**Exercised on 2026-10-21:** the 22:47 `hermes update` (0.22.2 → 0.22.3) recorded its `snapshot` stage as
|
|
`skipped` with reason `disabled by updates.pre_update_backup (mode: off)`, so that update shipped with **no
|
|
local snapshot and no off-box copy** — the Gitea mirror below has been dead since 2026-08-08.
|
|
→ [[systems/hermes-agent]]
|
|
|
|
## ⚠ Scheduled config backup — the Gitea mirror has been dead since 2026-08-08
|
|
|
|
`e27da6c34bbb` (**Hermes Config Backup to Gitea**, daily `0 11 * * *`, `hermes-config-backup.sh`) **commits
|
|
locally every night and fails to push every night — while its cron row still reads `last_status: ok` and its
|
|
execution row reads `completed`.** Last commit that reached Gitea: **2026-08-07 11:00:47**; the local repo is
|
|
**89 commits ahead of `origin/main`** (was 76 on 2026-10-22 — **13 more nights of local-only commits** since). Root cause is a credential: `origin` is
|
|
`https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` — HTTPS, no userinfo, no credential
|
|
helper — so the non-interactive push dies on
|
|
`fatal: could not read Username … No such device or address` → `✗ Push to Gitea FAILED` →
|
|
`--- Backup complete: FAIL ---`. Full evidence and the proposed fix:
|
|
[[incidents/2026-08-08-gitea-config-backup-push-dead]].
|
|
|
|
**Consequence for this page: everything above describes a topology that has not carried `~/.hermes` off the box
|
|
since 2026-08-07.** The 2026-07-22 full zip and the wiki/authoring repos remain the only recent off-box copies.
|
|
Telegram failure notices *have* been firing daily (last: 2026-10-21 11:00:14, `message_id 8449`); the AgentMail
|
|
notification step failed.
|
|
|
|
**Re-verified 2026-11-04 — still dead, and the drift is now 89 commits.** Live `git -C ~/.hermes/config-backup`:
|
|
`origin` = `https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` (**unchanged** — still HTTPS with
|
|
no userinfo and no credential helper, so the credential root cause has not been addressed); `rev-list --count
|
|
origin/main..HEAD` = **89**; `log origin/main -1` = **2026-08-07 11:00:47 +0800 1e1eb208** (`🦝 nightly backup:
|
|
2026-08-07 11:00:29 AWST`) — i.e. **nothing has reached Gitea in 89 days**. `backup.log` confirms the nightly
|
|
failure is current, not historical: the last two runs read
|
|
`[2026-11-02 11:00:32 AWST] ✗ Push to Gitea FAILED` / `--- Backup complete: FAIL ---` and
|
|
`[2026-11-03 11:00:31 AWST] ✗ Push to Gitea FAILED` / `--- Backup complete: FAIL ---`.
|
|
**A second, separate push failure is now also present:** the script attempts a `github` remote that **does not
|
|
exist** — `fatal: 'github' does not appear to be a git repository` → `✗ Push to GitHub FAILED`. So both off-box
|
|
targets are broken; the GitHub leg is a missing-remote error rather than a credential error, and would need
|
|
`git remote add github …` with credentials before it can work at all.
|
|
**The cron row still lies:** `e27da6c34bbb` reads `enabled: true, last_status: ok` in `jobs.json` while every
|
|
nightly execution has failed the push — the classic green-row/dead-artifact pattern.
|
|
|
|
**Full-backup gap unchanged:** the one full `hermes backup` zip on disk is still
|
|
`/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, 22 Jul) — **now 105 days old**. No new full zip has
|
|
been taken. `crontab -l -u hermes` is still empty and there are still **0 user systemd timers**, so nothing
|
|
schedules the full zip; only `e5ed215fc91c` (**Mnemosyne DB Backup**, weekly `0 3 * * 0`) and `e27da6c34bbb`
|
|
(config backup, daily) exist in the cron fleet, and neither takes a full `hermes backup`.
|
|
|
|
## Relevant runbooks
|
|
- [[runbooks/backup-wiki]]
|
|
- [[runbooks/update-hermes-safely]]
|
|
|
|
## Related
|
|
- [[systems/mnemosyne]]
|
|
- [[systems/hermes-agent]]
|