--- title: Hermes Config Backup to Gitea — Push Was Dead 2026-08-08 → 2026-11-07 While the Job Read `ok` type: incident status: resolved created: 2026-10-21 updated: 2026-11-07 resolved_on: 2026-11-07 verified_on: 2026-11-07 review_after: 2027-02-07 confidence: high tags: [backup, gitea, cron, git, green-vs-actual] sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki, gitea-api, github-api, sandbox-e2e-test] --- # Hermes Config Backup to Gitea — Off-Box Config Backup Dead Since 2026-08-08 > **RESOLVED 2026-11-07.** The push is live again, the backlog is drained, and the script now exits > non-zero on failure so the cron row cannot lie. See [Resolution 2026-11-07](#resolution-2026-11-07) > at the foot of this page for the full record. The original investigation is preserved below. ## Headline The daily **Hermes Config Backup to Gitea** job (`e27da6c34bbb`, `0 11 * * *`, no-agent, `hermes-config-backup.sh`) **commits locally every day and fails to push every day — and its cron row still reads `last_status: ok`.** The last commit that actually reached Gitea is **2026-08-07 11:00:47**; every nightly commit since **2026-08-08** is sitting in the local repo only. ## Evidence (live, 2026-10-21 AWST) | Probe | Result | |---|---| | `git log origin/main -1` in `~/.hermes/config-backup` | **`2026-08-07 11:00:47 +0800 1e1eb208 🦝 nightly backup: 2026-08-07 11:00:29 AWST`** | | first unpushed commit | `2026-08-08 11:00:43 c7d25358 🦝 nightly backup: 2026-08-08 11:00:59 AWST` | | `git rev-list --count origin/main..main` | **75 commits ahead of the last known remote commit** | | `git remote -v` | `origin` = `https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` — **HTTPS, no userinfo, no credential helper** | | `backup.log` tail (2026-10-20 11:00:17) | `fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address` → `✗ Push to Gitea FAILED` | | same run, second remote | `fatal: 'github' does not appear to be a git repository` → `✗ Push to GitHub FAILED` | | same run, notification step | AgentMail: `send failed`; **Telegram: delivered** (`🚨 BACKUP FAILED — 2026-10-20 11:00:17 AWST`, `message_id 8447`) | | same run, final line | `Your branch is ahead of 'origin/main' by 75 commits` / `--- Backup complete: FAIL ---` | | `.backup_history` | last 30 entries **all `FAIL`** (the file keeps 30 — the true streak is ~75) | | `jobs.json` entry `e27da6c34bbb` | `last_status: ok`, `last_error: null`, enabled | | executions ledger, 2026-10-20 11:00:14 | `status: completed` | **This is the second instance of the pattern in [[systems/scheduled-tasks]]'s notes: a green cron row is not evidence the job's work happened.** (The first was `c0a9fa314542`, GitHub Trending Radar, green while its delivery token was revoked.) ## Root cause The remote is HTTPS with the credential expected to come from git's credential machinery, and the cron worker has no credential helper, no `~/.git-credentials` entry for that host, and no userinfo in the URL — so a non-interactive push cannot authenticate and dies with `could not read Username`. Nothing in the script treats a failed push as failure of the job (it still commits, still sends the Telegram notify, and exits 0), so the scheduler records `completed` / `ok`. Contrast with the wiki repo, which pushes reliably because it carries the token **in the remote URL** (`http://@gitea:3000/Anthony/agent-estate-wiki.git`). The secondary failures are cosmetic by comparison: the `github` remote named in the script does not exist in this repo, and the AgentMail notify step also fails while the Telegram notify succeeds. ## Impact - **No configuration state has left the box since 2026-08-07.** A host loss would take `~/.hermes/config.yaml`, the cron job definitions, the skills tree and the dashboard server sources with it — only a 2026-07-22 full zip and the wiki repo survive off-box (see [[infrastructure/backups]]). - The **2026-10-20 20:06 `dashboard.basic_auth.secret` change and the 2026-10-20 `model.default` change are among the 75 unpushed commits** — the config history the wiki cites for those decisions lives only locally. - Daily failure notifications have been arriving in Telegram for ~75 days, so the signal exists; what is missing is the *job-level* red. ## Proposed fix (not applied — needs Anthony's go-ahead) 1. Give `origin` a credential the cron worker can use, the same way the wiki repo does — either userinfo in the URL or a `GIT_ASKPASS`/helper wired into the script's environment. 2. Push the 75-commit backlog **after** confirming the token is still valid (a 75-day-old URL token may be stale). 3. Delete or create the `github` remote so the script stops erroring on it. 4. Make the script **exit non-zero when the push fails**, so `last_status` stops lying. 5. Re-run and confirm: `git rev-list --count origin/main..main` → `0` and a fresh `git log origin/main -1` timestamp. ## Verification command set (for the next audit) ``` git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s' git -C ~/.hermes/config-backup rev-list --count origin/main..main tail -20 ~/.hermes/config-backup/backup.log tail -3 ~/.hermes/config-backup/.backup_history ``` ## Update 2026-10-21 (nightly run — the backlog grew again) | Probe | Result | |---|---| | 2026-10-21 11:00 run | commit **`a8486a20`** created (`🦝 nightly backup: 2026-10-21 11:00:14 AWST`) — **push failed again**, `fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address` | | `git rev-list --count origin/main..main` | **76** (was 75) — the backlog grows by exactly one commit per night | | Telegram notice | delivered: `🚨 BACKUP FAILED — 2026-10-21 11:00:14 AWST`, **`message_id 8449`**; the AgentMail step failed again | | GitHub fallback | `fatal: 'github' does not appear to be a git repository`; its "pushing clean branch" recovery (`github-clean-1792551629`) also failed and was deleted | | `.backup_history` | `2026-10-19 … FAIL`, `2026-10-20 … FAIL`, **`2026-10-21 … FAIL`** | | cron row `e27da6c34bbb` | still `last_status: ok` — the green-vs-actual gap is now **74 days** wide | **Compounding factor found the same night:** the `hermes update` that ran at **2026-10-21 22:47 AWST** recorded its own `snapshot` stage as **`skipped`** (`disabled by updates.pre_update_backup (mode: off)`), so for that change as well the *only* off-box copy path was this dead push → [[infrastructure/backups]]. ## Related - [[infrastructure/backups]] - [[systems/scheduled-tasks]] - [[systems/hermes-agent]] --- # Resolution 2026-11-07 Anthony asked to fix the failing push (and to check the Notion vault for the API key). Resolved in one session: the push is live, the backlog is drained, and the script can no longer report a false green. ## Root causes (four, all independent) 1. **No Gitea credential.** `origin` was HTTPS with no userinfo, and the global `credential.helper=store` file `~/.git-credentials` held only a `github.com` entry — so the cron worker could not authenticate non-interactively. The token in the Notion vault was valid and identical to the one the wiki repo carries in its remote URL; it simply was never wired into the backup repo. 2. **`backup.sh` had no `exit` statement at all.** It always returned 0, so the scheduler (`success = (returncode == 0)` in `cron/scheduler_script.py`) recorded `last_status: ok` on every failed night. This is why the outage went unnoticed for 92 days — it was a *false green*, not a missing signal. (The incident's earlier note blamed the scheduler; the scheduler was correct. The script was the liar.) 3. **A 1.76 GB `checkpoints/` backlog** sat in the unpushed range — 13 blobs over 100 MB, all junk that `.gitignore` already excluded for new commits but that history still carried. 4. **Two broken secondary legs.** The `github` remote did not exist, and the fallback `_github_clean_push` pushed a branch's full ancestry — which contains a 7.45 GB tarball blob, so GitHub's 100 MB file limit could never accept it. The AgentMail alert called a `python3 send_email.py` that no longer exists, via a `chaos-agent@agentmail.to` inbox that no longer exists. ## Fix applied - **Credential:** stored a host-scoped Gitea entry via `git credential approve` (not embedded in the remote URL). `git ls-remote origin` now authenticates with `GIT_TERMINAL_PROMPT=0`. - **Backlog:** `git filter-repo --partial --refs origin/main..main --path checkpoints --invert-paths --prune-empty never --force` stripped `checkpoints/` from the unpushed range only (so `origin/main` was untouched and the push stayed a fast-forward), then `checkpoints/` was untracked at the tip. Pre-rewrite history is preserved under the tag `backup/pre-rewrite-20261107-032658`. **Push set: ~2 GB → 264.8 MB.** The 7.45 GB tarball was already on `origin/main` and was never resent. - **Script hardening** (`~/.hermes/config-backup/backup.sh`, rewritten via OpenCode + a correction pass): - exits **0 only on verified success**, `1` otherwise — the cron row can no longer lie; - `.last_backup_ok` is stamped **only** after the remote is confirmed to hold the commit; - the push decision compares **remote vs local**, not "were there new working-tree changes" — a local-ahead-of-remote with a clean tree now still pushes (this is exactly the mechanism that created the 92-commit backlog); - every `git ls-remote` is guarded with `|| true`, so an unreachable remote reaches the alert path instead of killing the script at exit 128 before `notify_failure` fires; - the GitHub leg now pushes a **parentless orphan snapshot** built from `HEAD^{tree}` via `git commit-tree` — no history, no >100 MB blob, HEAD/branch/index untouched — and is a warning-only secondary target that can never fail the job; - the AgentMail leg uses the real CLI (`agentmail inboxes:messages send --inbox-id `) with the key read from `.env` at runtime and passed via the environment, never argv; - the Telegram leg uses `curl --fail` and requires `"ok":true` in the body, so it reports `sent` only on a real delivery. ## Verification (live, 2026-11-07 AWST) | Check | Result | |---|---| | Real `backup.sh` run | exit **0**, 47 s | | `git rev-parse HEAD` vs `git ls-remote origin refs/heads/main` | identical — `a50a5170c1c0e2af5235fa63a95a732e2d780703` | | `git rev-list --count origin/main..main` | **0** (was 92) | | Gitea API repo `Anthony/hermes-config-backup` | HTTP 200, default branch `main`, updated `2026-11-07T05:10:46+08:00`, 8.1 MB | | Gitea API `branches/main` | `a50a5170c1c0` — `🦝 nightly backup: 2026-11-07 05:10:33 AWST` | | `.last_backup_ok` | refreshed to the run timestamp (was stale from 2026-11-06 11:00) | | GitHub `main` | `faed9fb5c677`, **0 parents** (orphan snapshot confirmed), 30,086 files, largest blob 23.5 MB | | GitHub `master` branch | untouched at `c99c94e6b13b` | | GitHub force-push data loss | none — GitHub's tree was a strict **subset** of local HEAD (only `BACKUP-INDEX.md` added) | | Token leakage scan (HEAD tree, working tree, all unpushed blobs) | **clean** — token absent everywhere | | AgentMail leg (self-test to the agent's own inbox) | delivered, `message_id` returned | | Telegram leg (`getMe`, no message sent to Anthony) | `"ok":true` detected → would report `sent` | | Sandbox E2E: unreachable remote | exit **1**, `notify_failure` fired, health file **not** stamped | | Sandbox E2E: local ahead of remote, clean working tree | pushed + verified, exit **0**, health stamped | | Sandbox E2E: already up to date | no push, exit **0**, health stamped | ## Verification command set (for the next audit) ``` git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s' git -C ~/.hermes/config-backup rev-list --count origin/main..main tail -20 ~/.hermes/config-backup/backup.log tail -3 ~/.hermes/config-backup/.backup_history ``` ## Still open (not part of this incident) - Gitea host SSH login is denied for `anthonymau` — free disk headroom on the Gitea box is still unverified (the Gitea API reports repo size, not host free space). - `hermes-gateway-runtime-health.service` still fails every 5 min because the Mattermost host `100.98.116.14` has been offline since ~Sept 2026. Separate incident. - The `checkpoints/` directory is still excluded from backups by policy; the live snapshots remain at `~/.hermes/checkpoints/` on-box only.