- new: incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md - systems/hermes-agent.md: version -> 0.22.3 (5b147ab036), 22:47-22:57 update, snapshot stage skipped, advisories unactioned - systems/scheduled-tasks.md: 2026-10-21 = 171 completed / 1 failed; non-clean stays 9 with 08eb86f02d37 out, dca8482e4f76 in - systems/omniroute.md: one pin, three signatures in eight days - incidents/2026-10-16-omniroute-503-resource-pressure.md: window update, stays monitoring - incidents/2026-08-08-gitea-config-backup-push-dead.md: 2026-10-21 run, backlog 75 -> 76, message_id 8449 - infrastructure/backups.md: 76 commits; pre_update_backup:false now exercised - current-state.md, index.md, log.md updated
106 lines
6.2 KiB
Markdown
106 lines
6.2 KiB
Markdown
---
|
|
title: Hermes Config Backup to Gitea — Push Has Been Dead Since 2026-08-08 While the Job Reads `ok`
|
|
type: incident
|
|
status: active
|
|
created: 2026-10-21
|
|
updated: 2026-10-22
|
|
verified_on: 2026-10-22
|
|
review_after: 2026-10-29
|
|
confidence: high
|
|
tags: [backup, gitea, cron, git, green-vs-actual]
|
|
sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki]
|
|
---
|
|
|
|
# Hermes Config Backup to Gitea — Off-Box Config Backup Dead Since 2026-08-08
|
|
|
|
## Headline
|
|
|
|
The daily **Hermes Config Backup to Gitea** job (`e27da6c34bbb`, `0 11 * * *`, no-agent, `hermes-config-backup.sh`)
|
|
**commits locally every day and fails to push every day — and its cron row still reads `last_status: ok`.**
|
|
The last commit that actually reached Gitea is **2026-08-07 11:00:47**; every nightly commit since
|
|
**2026-08-08** is sitting in the local repo only.
|
|
|
|
## Evidence (live, 2026-10-21 AWST)
|
|
|
|
| Probe | Result |
|
|
|---|---|
|
|
| `git log origin/main -1` in `~/.hermes/config-backup` | **`2026-08-07 11:00:47 +0800 1e1eb208 🦝 nightly backup: 2026-08-07 11:00:29 AWST`** |
|
|
| first unpushed commit | `2026-08-08 11:00:43 c7d25358 🦝 nightly backup: 2026-08-08 11:00:59 AWST` |
|
|
| `git rev-list --count origin/main..main` | **75 commits ahead of the last known remote commit** |
|
|
| `git remote -v` | `origin` = `https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` — **HTTPS, no userinfo, no credential helper** |
|
|
| `backup.log` tail (2026-10-20 11:00:17) | `fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address` → `✗ Push to Gitea FAILED` |
|
|
| same run, second remote | `fatal: 'github' does not appear to be a git repository` → `✗ Push to GitHub FAILED` |
|
|
| same run, notification step | AgentMail: `send failed`; **Telegram: delivered** (`🚨 BACKUP FAILED — 2026-10-20 11:00:17 AWST`, `message_id 8447`) |
|
|
| same run, final line | `Your branch is ahead of 'origin/main' by 75 commits` / `--- Backup complete: FAIL ---` |
|
|
| `.backup_history` | last 30 entries **all `FAIL`** (the file keeps 30 — the true streak is ~75) |
|
|
| `jobs.json` entry `e27da6c34bbb` | `last_status: ok`, `last_error: null`, enabled |
|
|
| executions ledger, 2026-10-20 11:00:14 | `status: completed` |
|
|
|
|
**This is the second instance of the pattern in [[systems/scheduled-tasks]]'s notes: a green cron row is not
|
|
evidence the job's work happened.** (The first was `c0a9fa314542`, GitHub Trending Radar, green while its
|
|
delivery token was revoked.)
|
|
|
|
## Root cause
|
|
|
|
The remote is HTTPS with the credential expected to come from git's credential machinery, and the cron worker
|
|
has no credential helper, no `~/.git-credentials` entry for that host, and no userinfo in the URL — so a
|
|
non-interactive push cannot authenticate and dies with `could not read Username`. Nothing in the script treats
|
|
a failed push as failure of the job (it still commits, still sends the Telegram notify, and exits 0), so the
|
|
scheduler records `completed` / `ok`.
|
|
|
|
Contrast with the wiki repo, which pushes reliably because it carries the token **in the remote URL**
|
|
(`http://<token>@gitea:3000/Anthony/agent-estate-wiki.git`).
|
|
|
|
The secondary failures are cosmetic by comparison: the `github` remote named in the script does not exist in
|
|
this repo, and the AgentMail notify step also fails while the Telegram notify succeeds.
|
|
|
|
## Impact
|
|
|
|
- **No configuration state has left the box since 2026-08-07.** A host loss would take `~/.hermes/config.yaml`,
|
|
the cron job definitions, the skills tree and the dashboard server sources with it — only a 2026-07-22 full
|
|
zip and the wiki repo survive off-box (see [[infrastructure/backups]]).
|
|
- The **2026-10-20 20:06 `dashboard.basic_auth.secret` change and the 2026-10-20 `model.default` change are
|
|
among the 75 unpushed commits** — the config history the wiki cites for those decisions lives only locally.
|
|
- Daily failure notifications have been arriving in Telegram for ~75 days, so the signal exists; what is
|
|
missing is the *job-level* red.
|
|
|
|
## Proposed fix (not applied — needs Anthony's go-ahead)
|
|
|
|
1. Give `origin` a credential the cron worker can use, the same way the wiki repo does — either userinfo in
|
|
the URL or a `GIT_ASKPASS`/helper wired into the script's environment.
|
|
2. Push the 75-commit backlog **after** confirming the token is still valid (a 75-day-old URL token may be stale).
|
|
3. Delete or create the `github` remote so the script stops erroring on it.
|
|
4. Make the script **exit non-zero when the push fails**, so `last_status` stops lying.
|
|
5. Re-run and confirm: `git rev-list --count origin/main..main` → `0` and a fresh
|
|
`git log origin/main -1` timestamp.
|
|
|
|
## Verification command set (for the next audit)
|
|
|
|
```
|
|
git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s'
|
|
git -C ~/.hermes/config-backup rev-list --count origin/main..main
|
|
tail -20 ~/.hermes/config-backup/backup.log
|
|
tail -3 ~/.hermes/config-backup/.backup_history
|
|
```
|
|
|
|
## Update 2026-10-21 (nightly run — the backlog grew again)
|
|
|
|
| Probe | Result |
|
|
|---|---|
|
|
| 2026-10-21 11:00 run | commit **`a8486a20`** created (`🦝 nightly backup: 2026-10-21 11:00:14 AWST`) — **push failed again**, `fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address` |
|
|
| `git rev-list --count origin/main..main` | **76** (was 75) — the backlog grows by exactly one commit per night |
|
|
| Telegram notice | delivered: `🚨 BACKUP FAILED — 2026-10-21 11:00:14 AWST`, **`message_id 8449`**; the AgentMail step failed again |
|
|
| GitHub fallback | `fatal: 'github' does not appear to be a git repository`; its "pushing clean branch" recovery (`github-clean-1792551629`) also failed and was deleted |
|
|
| `.backup_history` | `2026-10-19 … FAIL`, `2026-10-20 … FAIL`, **`2026-10-21 … FAIL`** |
|
|
| cron row `e27da6c34bbb` | still `last_status: ok` — the green-vs-actual gap is now **74 days** wide |
|
|
|
|
**Compounding factor found the same night:** the `hermes update` that ran at **2026-10-21 22:47 AWST** recorded
|
|
its own `snapshot` stage as **`skipped`** (`disabled by updates.pre_update_backup (mode: off)`), so for that
|
|
change as well the *only* off-box copy path was this dead push → [[infrastructure/backups]].
|
|
|
|
## Related
|
|
|
|
- [[infrastructure/backups]]
|
|
- [[systems/scheduled-tasks]]
|
|
- [[systems/hermes-agent]]
|