- new: incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md - systems/hermes-agent.md: version -> 0.22.3 (5b147ab036), 22:47-22:57 update, snapshot stage skipped, advisories unactioned - systems/scheduled-tasks.md: 2026-10-21 = 171 completed / 1 failed; non-clean stays 9 with 08eb86f02d37 out, dca8482e4f76 in - systems/omniroute.md: one pin, three signatures in eight days - incidents/2026-10-16-omniroute-503-resource-pressure.md: window update, stays monitoring - incidents/2026-08-08-gitea-config-backup-push-dead.md: 2026-10-21 run, backlog 75 -> 76, message_id 8449 - infrastructure/backups.md: 76 commits; pre_update_backup:false now exercised - current-state.md, index.md, log.md updated
6.2 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Hermes Config Backup to Gitea — Push Has Been Dead Since 2026-08-08 While the Job Reads `ok` | incident | active | 2026-10-21 | 2026-10-22 | 2026-10-22 | 2026-10-29 | high |
|
|
Hermes Config Backup to Gitea — Off-Box Config Backup Dead Since 2026-08-08
Headline
The daily Hermes Config Backup to Gitea job (e27da6c34bbb, 0 11 * * *, no-agent, hermes-config-backup.sh)
commits locally every day and fails to push every day — and its cron row still reads last_status: ok.
The last commit that actually reached Gitea is 2026-08-07 11:00:47; every nightly commit since
2026-08-08 is sitting in the local repo only.
Evidence (live, 2026-10-21 AWST)
| Probe | Result |
|---|---|
git log origin/main -1 in ~/.hermes/config-backup |
2026-08-07 11:00:47 +0800 1e1eb208 🦝 nightly backup: 2026-08-07 11:00:29 AWST |
| first unpushed commit | 2026-08-08 11:00:43 c7d25358 🦝 nightly backup: 2026-08-08 11:00:59 AWST |
git rev-list --count origin/main..main |
75 commits ahead of the last known remote commit |
git remote -v |
origin = https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git — HTTPS, no userinfo, no credential helper |
backup.log tail (2026-10-20 11:00:17) |
fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address → ✗ Push to Gitea FAILED |
| same run, second remote | fatal: 'github' does not appear to be a git repository → ✗ Push to GitHub FAILED |
| same run, notification step | AgentMail: send failed; Telegram: delivered (🚨 BACKUP FAILED — 2026-10-20 11:00:17 AWST, message_id 8447) |
| same run, final line | Your branch is ahead of 'origin/main' by 75 commits / --- Backup complete: FAIL --- |
.backup_history |
last 30 entries all FAIL (the file keeps 30 — the true streak is ~75) |
jobs.json entry e27da6c34bbb |
last_status: ok, last_error: null, enabled |
| executions ledger, 2026-10-20 11:00:14 | status: completed |
This is the second instance of the pattern in systems/scheduled-tasks's notes: a green cron row is not
evidence the job's work happened. (The first was c0a9fa314542, GitHub Trending Radar, green while its
delivery token was revoked.)
Root cause
The remote is HTTPS with the credential expected to come from git's credential machinery, and the cron worker
has no credential helper, no ~/.git-credentials entry for that host, and no userinfo in the URL — so a
non-interactive push cannot authenticate and dies with could not read Username. Nothing in the script treats
a failed push as failure of the job (it still commits, still sends the Telegram notify, and exits 0), so the
scheduler records completed / ok.
Contrast with the wiki repo, which pushes reliably because it carries the token in the remote URL
(http://<token>@gitea:3000/Anthony/agent-estate-wiki.git).
The secondary failures are cosmetic by comparison: the github remote named in the script does not exist in
this repo, and the AgentMail notify step also fails while the Telegram notify succeeds.
Impact
- No configuration state has left the box since 2026-08-07. A host loss would take
~/.hermes/config.yaml, the cron job definitions, the skills tree and the dashboard server sources with it — only a 2026-07-22 full zip and the wiki repo survive off-box (see infrastructure/backups). - The 2026-10-20 20:06
dashboard.basic_auth.secretchange and the 2026-10-20model.defaultchange are among the 75 unpushed commits — the config history the wiki cites for those decisions lives only locally. - Daily failure notifications have been arriving in Telegram for ~75 days, so the signal exists; what is missing is the job-level red.
Proposed fix (not applied — needs Anthony's go-ahead)
- Give
origina credential the cron worker can use, the same way the wiki repo does — either userinfo in the URL or aGIT_ASKPASS/helper wired into the script's environment. - Push the 75-commit backlog after confirming the token is still valid (a 75-day-old URL token may be stale).
- Delete or create the
githubremote so the script stops erroring on it. - Make the script exit non-zero when the push fails, so
last_statusstops lying. - Re-run and confirm:
git rev-list --count origin/main..main→0and a freshgit log origin/main -1timestamp.
Verification command set (for the next audit)
git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s'
git -C ~/.hermes/config-backup rev-list --count origin/main..main
tail -20 ~/.hermes/config-backup/backup.log
tail -3 ~/.hermes/config-backup/.backup_history
Update 2026-10-21 (nightly run — the backlog grew again)
| Probe | Result |
|---|---|
| 2026-10-21 11:00 run | commit a8486a20 created (🦝 nightly backup: 2026-10-21 11:00:14 AWST) — push failed again, fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address |
git rev-list --count origin/main..main |
76 (was 75) — the backlog grows by exactly one commit per night |
| Telegram notice | delivered: 🚨 BACKUP FAILED — 2026-10-21 11:00:14 AWST, message_id 8449; the AgentMail step failed again |
| GitHub fallback | fatal: 'github' does not appear to be a git repository; its "pushing clean branch" recovery (github-clean-1792551629) also failed and was deleted |
.backup_history |
2026-10-19 … FAIL, 2026-10-20 … FAIL, 2026-10-21 … FAIL |
cron row e27da6c34bbb |
still last_status: ok — the green-vs-actual gap is now 74 days wide |
Compounding factor found the same night: the hermes update that ran at 2026-10-21 22:47 AWST recorded
its own snapshot stage as skipped (disabled by updates.pre_update_backup (mode: off)), so for that
change as well the only off-box copy path was this dead push → infrastructure/backups.