resolve: Gitea config backup push dead since 2026-08-08 — 92-commit backlog drained, script hardened to fail loudly
- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo - Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved) - Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB - Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification, decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree, honest AgentMail/Telegram delivery reporting - Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed - Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated - Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with 'lost its durable fire claim ownership' and left the tree dirty)
This commit is contained in:
@@ -1,18 +1,23 @@
|
||||
---
|
||||
title: Hermes Config Backup to Gitea — Push Has Been Dead Since 2026-08-08 While the Job Reads `ok`
|
||||
title: Hermes Config Backup to Gitea — Push Was Dead 2026-08-08 → 2026-11-07 While the Job Read `ok`
|
||||
type: incident
|
||||
status: active
|
||||
status: resolved
|
||||
created: 2026-10-21
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
review_after: 2026-10-29
|
||||
updated: 2026-11-07
|
||||
resolved_on: 2026-11-07
|
||||
verified_on: 2026-11-07
|
||||
review_after: 2027-02-07
|
||||
confidence: high
|
||||
tags: [backup, gitea, cron, git, green-vs-actual]
|
||||
sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki]
|
||||
sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki, gitea-api, github-api, sandbox-e2e-test]
|
||||
---
|
||||
|
||||
# Hermes Config Backup to Gitea — Off-Box Config Backup Dead Since 2026-08-08
|
||||
|
||||
> **RESOLVED 2026-11-07.** The push is live again, the backlog is drained, and the script now exits
|
||||
> non-zero on failure so the cron row cannot lie. See [Resolution 2026-11-07](#resolution-2026-11-07)
|
||||
> at the foot of this page for the full record. The original investigation is preserved below.
|
||||
|
||||
## Headline
|
||||
|
||||
The daily **Hermes Config Backup to Gitea** job (`e27da6c34bbb`, `0 11 * * *`, no-agent, `hermes-config-backup.sh`)
|
||||
@@ -103,3 +108,92 @@ change as well the *only* off-box copy path was this dead push → [[infrastruct
|
||||
- [[infrastructure/backups]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/hermes-agent]]
|
||||
|
||||
---
|
||||
|
||||
# Resolution 2026-11-07
|
||||
|
||||
Anthony asked to fix the failing push (and to check the Notion vault for the API key). Resolved in
|
||||
one session: the push is live, the backlog is drained, and the script can no longer report a false green.
|
||||
|
||||
## Root causes (four, all independent)
|
||||
|
||||
1. **No Gitea credential.** `origin` was HTTPS with no userinfo, and the global `credential.helper=store`
|
||||
file `~/.git-credentials` held only a `github.com` entry — so the cron worker could not authenticate
|
||||
non-interactively. The token in the Notion vault was valid and identical to the one the wiki repo
|
||||
carries in its remote URL; it simply was never wired into the backup repo.
|
||||
2. **`backup.sh` had no `exit` statement at all.** It always returned 0, so the scheduler
|
||||
(`success = (returncode == 0)` in `cron/scheduler_script.py`) recorded `last_status: ok` on every
|
||||
failed night. This is why the outage went unnoticed for 92 days — it was a *false green*, not a
|
||||
missing signal. (The incident's earlier note blamed the scheduler; the scheduler was correct. The
|
||||
script was the liar.)
|
||||
3. **A 1.76 GB `checkpoints/` backlog** sat in the unpushed range — 13 blobs over 100 MB, all junk
|
||||
that `.gitignore` already excluded for new commits but that history still carried.
|
||||
4. **Two broken secondary legs.** The `github` remote did not exist, and the fallback
|
||||
`_github_clean_push` pushed a branch's full ancestry — which contains a 7.45 GB tarball blob, so
|
||||
GitHub's 100 MB file limit could never accept it. The AgentMail alert called a
|
||||
`python3 send_email.py` that no longer exists, via a `chaos-agent@agentmail.to` inbox that no
|
||||
longer exists.
|
||||
|
||||
## Fix applied
|
||||
|
||||
- **Credential:** stored a host-scoped Gitea entry via `git credential approve` (not embedded in the
|
||||
remote URL). `git ls-remote origin` now authenticates with `GIT_TERMINAL_PROMPT=0`.
|
||||
- **Backlog:** `git filter-repo --partial --refs origin/main..main --path checkpoints --invert-paths
|
||||
--prune-empty never --force` stripped `checkpoints/` from the unpushed range only (so `origin/main`
|
||||
was untouched and the push stayed a fast-forward), then `checkpoints/` was untracked at the tip.
|
||||
Pre-rewrite history is preserved under the tag `backup/pre-rewrite-20261107-032658`.
|
||||
**Push set: ~2 GB → 264.8 MB.** The 7.45 GB tarball was already on `origin/main` and was never resent.
|
||||
- **Script hardening** (`~/.hermes/config-backup/backup.sh`, rewritten via OpenCode + a correction pass):
|
||||
- exits **0 only on verified success**, `1` otherwise — the cron row can no longer lie;
|
||||
- `.last_backup_ok` is stamped **only** after the remote is confirmed to hold the commit;
|
||||
- the push decision compares **remote vs local**, not "were there new working-tree changes" — a
|
||||
local-ahead-of-remote with a clean tree now still pushes (this is exactly the mechanism that
|
||||
created the 92-commit backlog);
|
||||
- every `git ls-remote` is guarded with `|| true`, so an unreachable remote reaches the alert path
|
||||
instead of killing the script at exit 128 before `notify_failure` fires;
|
||||
- the GitHub leg now pushes a **parentless orphan snapshot** built from `HEAD^{tree}` via
|
||||
`git commit-tree` — no history, no >100 MB blob, HEAD/branch/index untouched — and is a
|
||||
warning-only secondary target that can never fail the job;
|
||||
- the AgentMail leg uses the real CLI (`agentmail inboxes:messages send --inbox-id <addr>`) with the
|
||||
key read from `.env` at runtime and passed via the environment, never argv;
|
||||
- the Telegram leg uses `curl --fail` and requires `"ok":true` in the body, so it reports `sent`
|
||||
only on a real delivery.
|
||||
|
||||
## Verification (live, 2026-11-07 AWST)
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Real `backup.sh` run | exit **0**, 47 s |
|
||||
| `git rev-parse HEAD` vs `git ls-remote origin refs/heads/main` | identical — `a50a5170c1c0e2af5235fa63a95a732e2d780703` |
|
||||
| `git rev-list --count origin/main..main` | **0** (was 92) |
|
||||
| Gitea API repo `Anthony/hermes-config-backup` | HTTP 200, default branch `main`, updated `2026-11-07T05:10:46+08:00`, 8.1 MB |
|
||||
| Gitea API `branches/main` | `a50a5170c1c0` — `🦝 nightly backup: 2026-11-07 05:10:33 AWST` |
|
||||
| `.last_backup_ok` | refreshed to the run timestamp (was stale from 2026-11-06 11:00) |
|
||||
| GitHub `main` | `faed9fb5c677`, **0 parents** (orphan snapshot confirmed), 30,086 files, largest blob 23.5 MB |
|
||||
| GitHub `master` branch | untouched at `c99c94e6b13b` |
|
||||
| GitHub force-push data loss | none — GitHub's tree was a strict **subset** of local HEAD (only `BACKUP-INDEX.md` added) |
|
||||
| Token leakage scan (HEAD tree, working tree, all unpushed blobs) | **clean** — token absent everywhere |
|
||||
| AgentMail leg (self-test to the agent's own inbox) | delivered, `message_id` returned |
|
||||
| Telegram leg (`getMe`, no message sent to Anthony) | `"ok":true` detected → would report `sent` |
|
||||
| Sandbox E2E: unreachable remote | exit **1**, `notify_failure` fired, health file **not** stamped |
|
||||
| Sandbox E2E: local ahead of remote, clean working tree | pushed + verified, exit **0**, health stamped |
|
||||
| Sandbox E2E: already up to date | no push, exit **0**, health stamped |
|
||||
|
||||
## Verification command set (for the next audit)
|
||||
|
||||
```
|
||||
git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s'
|
||||
git -C ~/.hermes/config-backup rev-list --count origin/main..main
|
||||
tail -20 ~/.hermes/config-backup/backup.log
|
||||
tail -3 ~/.hermes/config-backup/.backup_history
|
||||
```
|
||||
|
||||
## Still open (not part of this incident)
|
||||
|
||||
- Gitea host SSH login is denied for `anthonymau` — free disk headroom on the Gitea box is still
|
||||
unverified (the Gitea API reports repo size, not host free space).
|
||||
- `hermes-gateway-runtime-health.service` still fails every 5 min because the Mattermost host
|
||||
`100.98.116.14` has been offline since ~Sept 2026. Separate incident.
|
||||
- The `checkpoints/` directory is still excluded from backups by policy; the live snapshots remain
|
||||
at `~/.hermes/checkpoints/` on-box only.
|
||||
|
||||
Reference in New Issue
Block a user