4.9 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Hermes Config Backup to Gitea — Push Has Been Dead Since 2026-08-08 While the Job Reads `ok` | incident | active | 2026-10-21 | 2026-10-21 | 2026-10-21 | 2026-10-28 | high |
|
|
Hermes Config Backup to Gitea — Off-Box Config Backup Dead Since 2026-08-08
Headline
The daily Hermes Config Backup to Gitea job (e27da6c34bbb, 0 11 * * *, no-agent, hermes-config-backup.sh)
commits locally every day and fails to push every day — and its cron row still reads last_status: ok.
The last commit that actually reached Gitea is 2026-08-07 11:00:47; every nightly commit since
2026-08-08 is sitting in the local repo only.
Evidence (live, 2026-10-21 AWST)
| Probe | Result |
|---|---|
git log origin/main -1 in ~/.hermes/config-backup |
2026-08-07 11:00:47 +0800 1e1eb208 🦝 nightly backup: 2026-08-07 11:00:29 AWST |
| first unpushed commit | 2026-08-08 11:00:43 c7d25358 🦝 nightly backup: 2026-08-08 11:00:59 AWST |
git rev-list --count origin/main..main |
75 commits ahead of the last known remote commit |
git remote -v |
origin = https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git — HTTPS, no userinfo, no credential helper |
backup.log tail (2026-10-20 11:00:17) |
fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address → ✗ Push to Gitea FAILED |
| same run, second remote | fatal: 'github' does not appear to be a git repository → ✗ Push to GitHub FAILED |
| same run, notification step | AgentMail: send failed; Telegram: delivered (🚨 BACKUP FAILED — 2026-10-20 11:00:17 AWST, message_id 8447) |
| same run, final line | Your branch is ahead of 'origin/main' by 75 commits / --- Backup complete: FAIL --- |
.backup_history |
last 30 entries all FAIL (the file keeps 30 — the true streak is ~75) |
jobs.json entry e27da6c34bbb |
last_status: ok, last_error: null, enabled |
| executions ledger, 2026-10-20 11:00:14 | status: completed |
This is the second instance of the pattern in systems/scheduled-tasks's notes: a green cron row is not
evidence the job's work happened. (The first was c0a9fa314542, GitHub Trending Radar, green while its
delivery token was revoked.)
Root cause
The remote is HTTPS with the credential expected to come from git's credential machinery, and the cron worker
has no credential helper, no ~/.git-credentials entry for that host, and no userinfo in the URL — so a
non-interactive push cannot authenticate and dies with could not read Username. Nothing in the script treats
a failed push as failure of the job (it still commits, still sends the Telegram notify, and exits 0), so the
scheduler records completed / ok.
Contrast with the wiki repo, which pushes reliably because it carries the token in the remote URL
(http://<token>@gitea:3000/Anthony/agent-estate-wiki.git).
The secondary failures are cosmetic by comparison: the github remote named in the script does not exist in
this repo, and the AgentMail notify step also fails while the Telegram notify succeeds.
Impact
- No configuration state has left the box since 2026-08-07. A host loss would take
~/.hermes/config.yaml, the cron job definitions, the skills tree and the dashboard server sources with it — only a 2026-07-22 full zip and the wiki repo survive off-box (see infrastructure/backups). - The 2026-10-20 20:06
dashboard.basic_auth.secretchange and the 2026-10-20model.defaultchange are among the 75 unpushed commits — the config history the wiki cites for those decisions lives only locally. - Daily failure notifications have been arriving in Telegram for ~75 days, so the signal exists; what is missing is the job-level red.
Proposed fix (not applied — needs Anthony's go-ahead)
- Give
origina credential the cron worker can use, the same way the wiki repo does — either userinfo in the URL or aGIT_ASKPASS/helper wired into the script's environment. - Push the 75-commit backlog after confirming the token is still valid (a 75-day-old URL token may be stale).
- Delete or create the
githubremote so the script stops erroring on it. - Make the script exit non-zero when the push fails, so
last_statusstops lying. - Re-run and confirm:
git rev-list --count origin/main..main→0and a freshgit log origin/main -1timestamp.
Verification command set (for the next audit)
git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s'
git -C ~/.hermes/config-backup rev-list --count origin/main..main
tail -20 ~/.hermes/config-backup/backup.log
tail -3 ~/.hermes/config-backup/.backup_history