Files
agent-estate-wiki/incidents/2026-08-08-gitea-config-backup-push-dead.md
T

4.9 KiB

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Hermes Config Backup to Gitea — Push Has Been Dead Since 2026-08-08 While the Job Reads `ok` incident active 2026-10-21 2026-10-21 2026-10-21 2026-10-28 high
backup
gitea
cron
git
green-vs-actual
live-git-config-backup
live-cron-backup-log
live-jobs-json
live-git-wiki

Hermes Config Backup to Gitea — Off-Box Config Backup Dead Since 2026-08-08

Headline

The daily Hermes Config Backup to Gitea job (e27da6c34bbb, 0 11 * * *, no-agent, hermes-config-backup.sh) commits locally every day and fails to push every day — and its cron row still reads last_status: ok. The last commit that actually reached Gitea is 2026-08-07 11:00:47; every nightly commit since 2026-08-08 is sitting in the local repo only.

Evidence (live, 2026-10-21 AWST)

Probe Result
git log origin/main -1 in ~/.hermes/config-backup 2026-08-07 11:00:47 +0800 1e1eb208 🦝 nightly backup: 2026-08-07 11:00:29 AWST
first unpushed commit 2026-08-08 11:00:43 c7d25358 🦝 nightly backup: 2026-08-08 11:00:59 AWST
git rev-list --count origin/main..main 75 commits ahead of the last known remote commit
git remote -v origin = https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git — HTTPS, no userinfo, no credential helper
backup.log tail (2026-10-20 11:00:17) fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address → ✗ Push to Gitea FAILED
same run, second remote fatal: 'github' does not appear to be a git repository → ✗ Push to GitHub FAILED
same run, notification step AgentMail: send failed; Telegram: delivered (🚨 BACKUP FAILED — 2026-10-20 11:00:17 AWST, message_id 8447)
same run, final line Your branch is ahead of 'origin/main' by 75 commits / --- Backup complete: FAIL ---
.backup_history last 30 entries all FAIL (the file keeps 30 — the true streak is ~75)
jobs.json entry e27da6c34bbb last_status: ok, last_error: null, enabled
executions ledger, 2026-10-20 11:00:14 status: completed

This is the second instance of the pattern in systems/scheduled-tasks's notes: a green cron row is not evidence the job's work happened. (The first was c0a9fa314542, GitHub Trending Radar, green while its delivery token was revoked.)

Root cause

The remote is HTTPS with the credential expected to come from git's credential machinery, and the cron worker has no credential helper, no ~/.git-credentials entry for that host, and no userinfo in the URL — so a non-interactive push cannot authenticate and dies with could not read Username. Nothing in the script treats a failed push as failure of the job (it still commits, still sends the Telegram notify, and exits 0), so the scheduler records completed / ok.

Contrast with the wiki repo, which pushes reliably because it carries the token in the remote URL (http://<token>@gitea:3000/Anthony/agent-estate-wiki.git).

The secondary failures are cosmetic by comparison: the github remote named in the script does not exist in this repo, and the AgentMail notify step also fails while the Telegram notify succeeds.

Impact

  • No configuration state has left the box since 2026-08-07. A host loss would take ~/.hermes/config.yaml, the cron job definitions, the skills tree and the dashboard server sources with it — only a 2026-07-22 full zip and the wiki repo survive off-box (see infrastructure/backups).
  • The 2026-10-20 20:06 dashboard.basic_auth.secret change and the 2026-10-20 model.default change are among the 75 unpushed commits — the config history the wiki cites for those decisions lives only locally.
  • Daily failure notifications have been arriving in Telegram for ~75 days, so the signal exists; what is missing is the job-level red.

Proposed fix (not applied — needs Anthony's go-ahead)

  1. Give origin a credential the cron worker can use, the same way the wiki repo does — either userinfo in the URL or a GIT_ASKPASS/helper wired into the script's environment.
  2. Push the 75-commit backlog after confirming the token is still valid (a 75-day-old URL token may be stale).
  3. Delete or create the github remote so the script stops erroring on it.
  4. Make the script exit non-zero when the push fails, so last_status stops lying.
  5. Re-run and confirm: git rev-list --count origin/main..main → 0 and a fresh git log origin/main -1 timestamp.

Verification command set (for the next audit)

git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s'
git -C ~/.hermes/config-backup rev-list --count origin/main..main
tail -20 ~/.hermes/config-backup/backup.log
tail -3 ~/.hermes/config-backup/.backup_history