- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo - Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved) - Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB - Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification, decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree, honest AgentMail/Telegram delivery reporting - Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed - Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated - Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with 'lost its durable fire claim ownership' and left the tree dirty)
12 KiB
title, type, status, created, updated, resolved_on, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | resolved_on | verified_on | review_after | confidence | tags | sources | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Hermes Config Backup to Gitea — Push Was Dead 2026-08-08 → 2026-11-07 While the Job Read `ok` | incident | resolved | 2026-10-21 | 2026-11-07 | 2026-11-07 | 2026-11-07 | 2027-02-07 | high |
|
|
Hermes Config Backup to Gitea — Off-Box Config Backup Dead Since 2026-08-08
RESOLVED 2026-11-07. The push is live again, the backlog is drained, and the script now exits non-zero on failure so the cron row cannot lie. See Resolution 2026-11-07 at the foot of this page for the full record. The original investigation is preserved below.
Headline
The daily Hermes Config Backup to Gitea job (e27da6c34bbb, 0 11 * * *, no-agent, hermes-config-backup.sh)
commits locally every day and fails to push every day — and its cron row still reads last_status: ok.
The last commit that actually reached Gitea is 2026-08-07 11:00:47; every nightly commit since
2026-08-08 is sitting in the local repo only.
Evidence (live, 2026-10-21 AWST)
| Probe | Result |
|---|---|
git log origin/main -1 in ~/.hermes/config-backup |
2026-08-07 11:00:47 +0800 1e1eb208 🦝 nightly backup: 2026-08-07 11:00:29 AWST |
| first unpushed commit | 2026-08-08 11:00:43 c7d25358 🦝 nightly backup: 2026-08-08 11:00:59 AWST |
git rev-list --count origin/main..main |
75 commits ahead of the last known remote commit |
git remote -v |
origin = https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git — HTTPS, no userinfo, no credential helper |
backup.log tail (2026-10-20 11:00:17) |
fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address → ✗ Push to Gitea FAILED |
| same run, second remote | fatal: 'github' does not appear to be a git repository → ✗ Push to GitHub FAILED |
| same run, notification step | AgentMail: send failed; Telegram: delivered (🚨 BACKUP FAILED — 2026-10-20 11:00:17 AWST, message_id 8447) |
| same run, final line | Your branch is ahead of 'origin/main' by 75 commits / --- Backup complete: FAIL --- |
.backup_history |
last 30 entries all FAIL (the file keeps 30 — the true streak is ~75) |
jobs.json entry e27da6c34bbb |
last_status: ok, last_error: null, enabled |
| executions ledger, 2026-10-20 11:00:14 | status: completed |
This is the second instance of the pattern in systems/scheduled-tasks's notes: a green cron row is not
evidence the job's work happened. (The first was c0a9fa314542, GitHub Trending Radar, green while its
delivery token was revoked.)
Root cause
The remote is HTTPS with the credential expected to come from git's credential machinery, and the cron worker
has no credential helper, no ~/.git-credentials entry for that host, and no userinfo in the URL — so a
non-interactive push cannot authenticate and dies with could not read Username. Nothing in the script treats
a failed push as failure of the job (it still commits, still sends the Telegram notify, and exits 0), so the
scheduler records completed / ok.
Contrast with the wiki repo, which pushes reliably because it carries the token in the remote URL
(http://<token>@gitea:3000/Anthony/agent-estate-wiki.git).
The secondary failures are cosmetic by comparison: the github remote named in the script does not exist in
this repo, and the AgentMail notify step also fails while the Telegram notify succeeds.
Impact
- No configuration state has left the box since 2026-08-07. A host loss would take
~/.hermes/config.yaml, the cron job definitions, the skills tree and the dashboard server sources with it — only a 2026-07-22 full zip and the wiki repo survive off-box (see infrastructure/backups). - The 2026-10-20 20:06
dashboard.basic_auth.secretchange and the 2026-10-20model.defaultchange are among the 75 unpushed commits — the config history the wiki cites for those decisions lives only locally. - Daily failure notifications have been arriving in Telegram for ~75 days, so the signal exists; what is missing is the job-level red.
Proposed fix (not applied — needs Anthony's go-ahead)
- Give
origina credential the cron worker can use, the same way the wiki repo does — either userinfo in the URL or aGIT_ASKPASS/helper wired into the script's environment. - Push the 75-commit backlog after confirming the token is still valid (a 75-day-old URL token may be stale).
- Delete or create the
githubremote so the script stops erroring on it. - Make the script exit non-zero when the push fails, so
last_statusstops lying. - Re-run and confirm:
git rev-list --count origin/main..main→0and a freshgit log origin/main -1timestamp.
Verification command set (for the next audit)
git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s'
git -C ~/.hermes/config-backup rev-list --count origin/main..main
tail -20 ~/.hermes/config-backup/backup.log
tail -3 ~/.hermes/config-backup/.backup_history
Update 2026-10-21 (nightly run — the backlog grew again)
| Probe | Result |
|---|---|
| 2026-10-21 11:00 run | commit a8486a20 created (🦝 nightly backup: 2026-10-21 11:00:14 AWST) — push failed again, fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address |
git rev-list --count origin/main..main |
76 (was 75) — the backlog grows by exactly one commit per night |
| Telegram notice | delivered: 🚨 BACKUP FAILED — 2026-10-21 11:00:14 AWST, message_id 8449; the AgentMail step failed again |
| GitHub fallback | fatal: 'github' does not appear to be a git repository; its "pushing clean branch" recovery (github-clean-1792551629) also failed and was deleted |
.backup_history |
2026-10-19 … FAIL, 2026-10-20 … FAIL, 2026-10-21 … FAIL |
cron row e27da6c34bbb |
still last_status: ok — the green-vs-actual gap is now 74 days wide |
Compounding factor found the same night: the hermes update that ran at 2026-10-21 22:47 AWST recorded
its own snapshot stage as skipped (disabled by updates.pre_update_backup (mode: off)), so for that
change as well the only off-box copy path was this dead push → infrastructure/backups.
Related
Resolution 2026-11-07
Anthony asked to fix the failing push (and to check the Notion vault for the API key). Resolved in one session: the push is live, the backlog is drained, and the script can no longer report a false green.
Root causes (four, all independent)
- No Gitea credential.
originwas HTTPS with no userinfo, and the globalcredential.helper=storefile~/.git-credentialsheld only agithub.comentry — so the cron worker could not authenticate non-interactively. The token in the Notion vault was valid and identical to the one the wiki repo carries in its remote URL; it simply was never wired into the backup repo. backup.shhad noexitstatement at all. It always returned 0, so the scheduler (success = (returncode == 0)incron/scheduler_script.py) recordedlast_status: okon every failed night. This is why the outage went unnoticed for 92 days — it was a false green, not a missing signal. (The incident's earlier note blamed the scheduler; the scheduler was correct. The script was the liar.)- A 1.76 GB
checkpoints/backlog sat in the unpushed range — 13 blobs over 100 MB, all junk that.gitignorealready excluded for new commits but that history still carried. - Two broken secondary legs. The
githubremote did not exist, and the fallback_github_clean_pushpushed a branch's full ancestry — which contains a 7.45 GB tarball blob, so GitHub's 100 MB file limit could never accept it. The AgentMail alert called apython3 send_email.pythat no longer exists, via achaos-agent@agentmail.toinbox that no longer exists.
Fix applied
- Credential: stored a host-scoped Gitea entry via
git credential approve(not embedded in the remote URL).git ls-remote originnow authenticates withGIT_TERMINAL_PROMPT=0. - Backlog:
git filter-repo --partial --refs origin/main..main --path checkpoints --invert-paths --prune-empty never --forcestrippedcheckpoints/from the unpushed range only (soorigin/mainwas untouched and the push stayed a fast-forward), thencheckpoints/was untracked at the tip. Pre-rewrite history is preserved under the tagbackup/pre-rewrite-20261107-032658. Push set: ~2 GB → 264.8 MB. The 7.45 GB tarball was already onorigin/mainand was never resent. - Script hardening (
~/.hermes/config-backup/backup.sh, rewritten via OpenCode + a correction pass):- exits 0 only on verified success,
1otherwise — the cron row can no longer lie; .last_backup_okis stamped only after the remote is confirmed to hold the commit;- the push decision compares remote vs local, not "were there new working-tree changes" — a local-ahead-of-remote with a clean tree now still pushes (this is exactly the mechanism that created the 92-commit backlog);
- every
git ls-remoteis guarded with|| true, so an unreachable remote reaches the alert path instead of killing the script at exit 128 beforenotify_failurefires; - the GitHub leg now pushes a parentless orphan snapshot built from
HEAD^{tree}viagit commit-tree— no history, no >100 MB blob, HEAD/branch/index untouched — and is a warning-only secondary target that can never fail the job; - the AgentMail leg uses the real CLI (
agentmail inboxes:messages send --inbox-id <addr>) with the key read from.envat runtime and passed via the environment, never argv; - the Telegram leg uses
curl --failand requires"ok":truein the body, so it reportssentonly on a real delivery.
- exits 0 only on verified success,
Verification (live, 2026-11-07 AWST)
| Check | Result |
|---|---|
Real backup.sh run |
exit 0, 47 s |
git rev-parse HEAD vs git ls-remote origin refs/heads/main |
identical — a50a5170c1c0e2af5235fa63a95a732e2d780703 |
git rev-list --count origin/main..main |
0 (was 92) |
Gitea API repo Anthony/hermes-config-backup |
HTTP 200, default branch main, updated 2026-11-07T05:10:46+08:00, 8.1 MB |
Gitea API branches/main |
a50a5170c1c0 — 🦝 nightly backup: 2026-11-07 05:10:33 AWST |
.last_backup_ok |
refreshed to the run timestamp (was stale from 2026-11-06 11:00) |
GitHub main |
faed9fb5c677, 0 parents (orphan snapshot confirmed), 30,086 files, largest blob 23.5 MB |
GitHub master branch |
untouched at c99c94e6b13b |
| GitHub force-push data loss | none — GitHub's tree was a strict subset of local HEAD (only BACKUP-INDEX.md added) |
| Token leakage scan (HEAD tree, working tree, all unpushed blobs) | clean — token absent everywhere |
| AgentMail leg (self-test to the agent's own inbox) | delivered, message_id returned |
Telegram leg (getMe, no message sent to Anthony) |
"ok":true detected → would report sent |
| Sandbox E2E: unreachable remote | exit 1, notify_failure fired, health file not stamped |
| Sandbox E2E: local ahead of remote, clean working tree | pushed + verified, exit 0, health stamped |
| Sandbox E2E: already up to date | no push, exit 0, health stamped |
Verification command set (for the next audit)
git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s'
git -C ~/.hermes/config-backup rev-list --count origin/main..main
tail -20 ~/.hermes/config-backup/backup.log
tail -3 ~/.hermes/config-backup/.backup_history
Still open (not part of this incident)
- Gitea host SSH login is denied for
anthonymau— free disk headroom on the Gitea box is still unverified (the Gitea API reports repo size, not host free space). hermes-gateway-runtime-health.servicestill fails every 5 min because the Mattermost host100.98.116.14has been offline since ~Sept 2026. Separate incident.- The
checkpoints/directory is still excluded from backups by policy; the live snapshots remain at~/.hermes/checkpoints/on-box only.