Files
agent-estate-wiki/incidents/2026-08-08-gitea-config-backup-push-dead.md
T
Tony0410 0b4ff25fd2 resolve: Gitea config backup push dead since 2026-08-08 — 92-commit backlog drained, script hardened to fail loudly
- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo
- Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved)
- Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB
- Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification,
  decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree,
  honest AgentMail/Telegram delivery reporting
- Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed
- Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated
- Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with
  'lost its durable fire claim ownership' and left the tree dirty)
2026-11-07 05:28:39 +08:00

12 KiB

title, type, status, created, updated, resolved_on, verified_on, review_after, confidence, tags, sources
title type status created updated resolved_on verified_on review_after confidence tags sources
Hermes Config Backup to Gitea — Push Was Dead 2026-08-08 → 2026-11-07 While the Job Read `ok` incident resolved 2026-10-21 2026-11-07 2026-11-07 2026-11-07 2027-02-07 high
backup
gitea
cron
git
green-vs-actual
live-git-config-backup
live-cron-backup-log
live-jobs-json
live-git-wiki
gitea-api
github-api
sandbox-e2e-test

Hermes Config Backup to Gitea — Off-Box Config Backup Dead Since 2026-08-08

RESOLVED 2026-11-07. The push is live again, the backlog is drained, and the script now exits non-zero on failure so the cron row cannot lie. See Resolution 2026-11-07 at the foot of this page for the full record. The original investigation is preserved below.

Headline

The daily Hermes Config Backup to Gitea job (e27da6c34bbb, 0 11 * * *, no-agent, hermes-config-backup.sh) commits locally every day and fails to push every day — and its cron row still reads last_status: ok. The last commit that actually reached Gitea is 2026-08-07 11:00:47; every nightly commit since 2026-08-08 is sitting in the local repo only.

Evidence (live, 2026-10-21 AWST)

Probe Result
git log origin/main -1 in ~/.hermes/config-backup 2026-08-07 11:00:47 +0800 1e1eb208 🦝 nightly backup: 2026-08-07 11:00:29 AWST
first unpushed commit 2026-08-08 11:00:43 c7d25358 🦝 nightly backup: 2026-08-08 11:00:59 AWST
git rev-list --count origin/main..main 75 commits ahead of the last known remote commit
git remote -v origin = https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git — HTTPS, no userinfo, no credential helper
backup.log tail (2026-10-20 11:00:17) fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address → ✗ Push to Gitea FAILED
same run, second remote fatal: 'github' does not appear to be a git repository → ✗ Push to GitHub FAILED
same run, notification step AgentMail: send failed; Telegram: delivered (🚨 BACKUP FAILED — 2026-10-20 11:00:17 AWST, message_id 8447)
same run, final line Your branch is ahead of 'origin/main' by 75 commits / --- Backup complete: FAIL ---
.backup_history last 30 entries all FAIL (the file keeps 30 — the true streak is ~75)
jobs.json entry e27da6c34bbb last_status: ok, last_error: null, enabled
executions ledger, 2026-10-20 11:00:14 status: completed

This is the second instance of the pattern in systems/scheduled-tasks's notes: a green cron row is not evidence the job's work happened. (The first was c0a9fa314542, GitHub Trending Radar, green while its delivery token was revoked.)

Root cause

The remote is HTTPS with the credential expected to come from git's credential machinery, and the cron worker has no credential helper, no ~/.git-credentials entry for that host, and no userinfo in the URL — so a non-interactive push cannot authenticate and dies with could not read Username. Nothing in the script treats a failed push as failure of the job (it still commits, still sends the Telegram notify, and exits 0), so the scheduler records completed / ok.

Contrast with the wiki repo, which pushes reliably because it carries the token in the remote URL (http://<token>@gitea:3000/Anthony/agent-estate-wiki.git).

The secondary failures are cosmetic by comparison: the github remote named in the script does not exist in this repo, and the AgentMail notify step also fails while the Telegram notify succeeds.

Impact

  • No configuration state has left the box since 2026-08-07. A host loss would take ~/.hermes/config.yaml, the cron job definitions, the skills tree and the dashboard server sources with it — only a 2026-07-22 full zip and the wiki repo survive off-box (see infrastructure/backups).
  • The 2026-10-20 20:06 dashboard.basic_auth.secret change and the 2026-10-20 model.default change are among the 75 unpushed commits — the config history the wiki cites for those decisions lives only locally.
  • Daily failure notifications have been arriving in Telegram for ~75 days, so the signal exists; what is missing is the job-level red.

Proposed fix (not applied — needs Anthony's go-ahead)

  1. Give origin a credential the cron worker can use, the same way the wiki repo does — either userinfo in the URL or a GIT_ASKPASS/helper wired into the script's environment.
  2. Push the 75-commit backlog after confirming the token is still valid (a 75-day-old URL token may be stale).
  3. Delete or create the github remote so the script stops erroring on it.
  4. Make the script exit non-zero when the push fails, so last_status stops lying.
  5. Re-run and confirm: git rev-list --count origin/main..main → 0 and a fresh git log origin/main -1 timestamp.

Verification command set (for the next audit)

git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s'
git -C ~/.hermes/config-backup rev-list --count origin/main..main
tail -20 ~/.hermes/config-backup/backup.log
tail -3 ~/.hermes/config-backup/.backup_history

Update 2026-10-21 (nightly run — the backlog grew again)

Probe Result
2026-10-21 11:00 run commit a8486a20 created (🦝 nightly backup: 2026-10-21 11:00:14 AWST) — push failed again, fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address
git rev-list --count origin/main..main 76 (was 75) — the backlog grows by exactly one commit per night
Telegram notice delivered: 🚨 BACKUP FAILED — 2026-10-21 11:00:14 AWST, message_id 8449; the AgentMail step failed again
GitHub fallback fatal: 'github' does not appear to be a git repository; its "pushing clean branch" recovery (github-clean-1792551629) also failed and was deleted
.backup_history 2026-10-19 … FAIL, 2026-10-20 … FAIL, 2026-10-21 … FAIL
cron row e27da6c34bbb still last_status: ok — the green-vs-actual gap is now 74 days wide

Compounding factor found the same night: the hermes update that ran at 2026-10-21 22:47 AWST recorded its own snapshot stage as skipped (disabled by updates.pre_update_backup (mode: off)), so for that change as well the only off-box copy path was this dead push → infrastructure/backups.


Resolution 2026-11-07

Anthony asked to fix the failing push (and to check the Notion vault for the API key). Resolved in one session: the push is live, the backlog is drained, and the script can no longer report a false green.

Root causes (four, all independent)

  1. No Gitea credential. origin was HTTPS with no userinfo, and the global credential.helper=store file ~/.git-credentials held only a github.com entry — so the cron worker could not authenticate non-interactively. The token in the Notion vault was valid and identical to the one the wiki repo carries in its remote URL; it simply was never wired into the backup repo.
  2. backup.sh had no exit statement at all. It always returned 0, so the scheduler (success = (returncode == 0) in cron/scheduler_script.py) recorded last_status: ok on every failed night. This is why the outage went unnoticed for 92 days — it was a false green, not a missing signal. (The incident's earlier note blamed the scheduler; the scheduler was correct. The script was the liar.)
  3. A 1.76 GB checkpoints/ backlog sat in the unpushed range — 13 blobs over 100 MB, all junk that .gitignore already excluded for new commits but that history still carried.
  4. Two broken secondary legs. The github remote did not exist, and the fallback _github_clean_push pushed a branch's full ancestry — which contains a 7.45 GB tarball blob, so GitHub's 100 MB file limit could never accept it. The AgentMail alert called a python3 send_email.py that no longer exists, via a chaos-agent@agentmail.to inbox that no longer exists.

Fix applied

  • Credential: stored a host-scoped Gitea entry via git credential approve (not embedded in the remote URL). git ls-remote origin now authenticates with GIT_TERMINAL_PROMPT=0.
  • Backlog: git filter-repo --partial --refs origin/main..main --path checkpoints --invert-paths --prune-empty never --force stripped checkpoints/ from the unpushed range only (so origin/main was untouched and the push stayed a fast-forward), then checkpoints/ was untracked at the tip. Pre-rewrite history is preserved under the tag backup/pre-rewrite-20261107-032658. Push set: ~2 GB → 264.8 MB. The 7.45 GB tarball was already on origin/main and was never resent.
  • Script hardening (~/.hermes/config-backup/backup.sh, rewritten via OpenCode + a correction pass):
    • exits 0 only on verified success, 1 otherwise — the cron row can no longer lie;
    • .last_backup_ok is stamped only after the remote is confirmed to hold the commit;
    • the push decision compares remote vs local, not "were there new working-tree changes" — a local-ahead-of-remote with a clean tree now still pushes (this is exactly the mechanism that created the 92-commit backlog);
    • every git ls-remote is guarded with || true, so an unreachable remote reaches the alert path instead of killing the script at exit 128 before notify_failure fires;
    • the GitHub leg now pushes a parentless orphan snapshot built from HEAD^{tree} via git commit-tree — no history, no >100 MB blob, HEAD/branch/index untouched — and is a warning-only secondary target that can never fail the job;
    • the AgentMail leg uses the real CLI (agentmail inboxes:messages send --inbox-id <addr>) with the key read from .env at runtime and passed via the environment, never argv;
    • the Telegram leg uses curl --fail and requires "ok":true in the body, so it reports sent only on a real delivery.

Verification (live, 2026-11-07 AWST)

Check Result
Real backup.sh run exit 0, 47 s
git rev-parse HEAD vs git ls-remote origin refs/heads/main identical — a50a5170c1c0e2af5235fa63a95a732e2d780703
git rev-list --count origin/main..main 0 (was 92)
Gitea API repo Anthony/hermes-config-backup HTTP 200, default branch main, updated 2026-11-07T05:10:46+08:00, 8.1 MB
Gitea API branches/main a50a5170c1c0 — 🦝 nightly backup: 2026-11-07 05:10:33 AWST
.last_backup_ok refreshed to the run timestamp (was stale from 2026-11-06 11:00)
GitHub main faed9fb5c677, 0 parents (orphan snapshot confirmed), 30,086 files, largest blob 23.5 MB
GitHub master branch untouched at c99c94e6b13b
GitHub force-push data loss none — GitHub's tree was a strict subset of local HEAD (only BACKUP-INDEX.md added)
Token leakage scan (HEAD tree, working tree, all unpushed blobs) clean — token absent everywhere
AgentMail leg (self-test to the agent's own inbox) delivered, message_id returned
Telegram leg (getMe, no message sent to Anthony) "ok":true detected → would report sent
Sandbox E2E: unreachable remote exit 1, notify_failure fired, health file not stamped
Sandbox E2E: local ahead of remote, clean working tree pushed + verified, exit 0, health stamped
Sandbox E2E: already up to date no push, exit 0, health stamped

Verification command set (for the next audit)

git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s'
git -C ~/.hermes/config-backup rev-list --count origin/main..main
tail -20 ~/.hermes/config-backup/backup.log
tail -3 ~/.hermes/config-backup/.backup_history

Still open (not part of this incident)

  • Gitea host SSH login is denied for anthonymau — free disk headroom on the Gitea box is still unverified (the Gitea API reports repo size, not host free space).
  • hermes-gateway-runtime-health.service still fails every 5 min because the Mattermost host 100.98.116.14 has been offline since ~Sept 2026. Separate incident.
  • The checkpoints/ directory is still excluded from backups by policy; the live snapshots remain at ~/.hermes/checkpoints/ on-box only.