🦉 wiki audit 2026-10-21: gateway unclean-exit orphaned run (new), Gitea config-backup push dead since 2026-08-08 (new), model default → nous, 503 back to monitoring, canned-substitute trigger corrected

This commit is contained in:
Tony0410
2026-10-21 00:11:44 +08:00
parent fbd4f297eb
commit b662254569
10 changed files with 386 additions and 25 deletions
@@ -0,0 +1,90 @@
---
title: Hermes Config Backup to Gitea — Push Has Been Dead Since 2026-08-08 While the Job Reads `ok`
type: incident
status: active
created: 2026-10-21
updated: 2026-10-21
verified_on: 2026-10-21
review_after: 2026-10-28
confidence: high
tags: [backup, gitea, cron, git, green-vs-actual]
sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki]
---
# Hermes Config Backup to Gitea — Off-Box Config Backup Dead Since 2026-08-08
## Headline
The daily **Hermes Config Backup to Gitea** job (`e27da6c34bbb`, `0 11 * * *`, no-agent, `hermes-config-backup.sh`)
**commits locally every day and fails to push every day — and its cron row still reads `last_status: ok`.**
The last commit that actually reached Gitea is **2026-08-07 11:00:47**; every nightly commit since
**2026-08-08** is sitting in the local repo only.
## Evidence (live, 2026-10-21 AWST)
| Probe | Result |
|---|---|
| `git log origin/main -1` in `~/.hermes/config-backup` | **`2026-08-07 11:00:47 +0800 1e1eb208 🦝 nightly backup: 2026-08-07 11:00:29 AWST`** |
| first unpushed commit | `2026-08-08 11:00:43 c7d25358 🦝 nightly backup: 2026-08-08 11:00:59 AWST` |
| `git rev-list --count origin/main..main` | **75 commits ahead of the last known remote commit** |
| `git remote -v` | `origin` = `https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` — **HTTPS, no userinfo, no credential helper** |
| `backup.log` tail (2026-10-20 11:00:17) | `fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address` → `✗ Push to Gitea FAILED` |
| same run, second remote | `fatal: 'github' does not appear to be a git repository` → `✗ Push to GitHub FAILED` |
| same run, notification step | AgentMail: `send failed`; **Telegram: delivered** (`🚨 BACKUP FAILED — 2026-10-20 11:00:17 AWST`, `message_id 8447`) |
| same run, final line | `Your branch is ahead of 'origin/main' by 75 commits` / `--- Backup complete: FAIL ---` |
| `.backup_history` | last 30 entries **all `FAIL`** (the file keeps 30 — the true streak is ~75) |
| `jobs.json` entry `e27da6c34bbb` | `last_status: ok`, `last_error: null`, enabled |
| executions ledger, 2026-10-20 11:00:14 | `status: completed` |
**This is the second instance of the pattern in [[systems/scheduled-tasks]]'s notes: a green cron row is not
evidence the job's work happened.** (The first was `c0a9fa314542`, GitHub Trending Radar, green while its
delivery token was revoked.)
## Root cause
The remote is HTTPS with the credential expected to come from git's credential machinery, and the cron worker
has no credential helper, no `~/.git-credentials` entry for that host, and no userinfo in the URL — so a
non-interactive push cannot authenticate and dies with `could not read Username`. Nothing in the script treats
a failed push as failure of the job (it still commits, still sends the Telegram notify, and exits 0), so the
scheduler records `completed` / `ok`.
Contrast with the wiki repo, which pushes reliably because it carries the token **in the remote URL**
(`http://<token>@gitea:3000/Anthony/agent-estate-wiki.git`).
The secondary failures are cosmetic by comparison: the `github` remote named in the script does not exist in
this repo, and the AgentMail notify step also fails while the Telegram notify succeeds.
## Impact
- **No configuration state has left the box since 2026-08-07.** A host loss would take `~/.hermes/config.yaml`,
the cron job definitions, the skills tree and the dashboard server sources with it — only a 2026-07-22 full
zip and the wiki repo survive off-box (see [[infrastructure/backups]]).
- The **2026-10-20 20:06 `dashboard.basic_auth.secret` change and the 2026-10-20 `model.default` change are
among the 75 unpushed commits** — the config history the wiki cites for those decisions lives only locally.
- Daily failure notifications have been arriving in Telegram for ~75 days, so the signal exists; what is
missing is the *job-level* red.
## Proposed fix (not applied — needs Anthony's go-ahead)
1. Give `origin` a credential the cron worker can use, the same way the wiki repo does — either userinfo in
the URL or a `GIT_ASKPASS`/helper wired into the script's environment.
2. Push the 75-commit backlog **after** confirming the token is still valid (a 75-day-old URL token may be stale).
3. Delete or create the `github` remote so the script stops erroring on it.
4. Make the script **exit non-zero when the push fails**, so `last_status` stops lying.
5. Re-run and confirm: `git rev-list --count origin/main..main` → `0` and a fresh
`git log origin/main -1` timestamp.
## Verification command set (for the next audit)
```
git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s'
git -C ~/.hermes/config-backup rev-list --count origin/main..main
tail -20 ~/.hermes/config-backup/backup.log
tail -3 ~/.hermes/config-backup/.backup_history
```
## Related
- [[infrastructure/backups]]
- [[systems/scheduled-tasks]]
- [[systems/hermes-agent]]
@@ -1,11 +1,11 @@
---
title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts
type: incident
status: active
status: monitoring
created: 2026-10-18
updated: 2026-10-20
verified_on: 2026-10-20
review_after: 2026-10-27
updated: 2026-10-21
verified_on: 2026-10-21
review_after: 2026-10-28
confidence: high
tags: [cron, router, provider, infrastructure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
@@ -13,10 +13,21 @@ sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-jso
# OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts (burst 1: 2026-10-16 → 10-17; burst 2: 2026-10-19)
**Status: ACTIVE — this is a recurrence, not a closed incident.** The failure mode came back **twice in four
**Status: `MONITORING` as of 2026-10-21 — not closed.** The failure mode came back **twice in four
days** (10-16 → 10-17 and 10-19) with an identical signature on an identical node, and the **exact same
9-model-call blast radius** each time. The node was healthy before, between and after both bursts.
> **2026-10-21 update — burst 2 is over; the page returns to `monitoring` on the 18–24 h rule it set itself.**
> **2026-10-20 ran 168 `completed` / 0 `failed`** (plus one `unknown` orphan — see
> [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]), the **first zero-failure day since 2026-10-12**,
> and **no execution row carries the resource-pressure string** (the last such failure remains 2026-10-19).
> **4 incident rows closed on 10-20** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`,
> `ecc1fc_21a72aa11cc4`), taking unclosed rows **13 → 9** (8 `alerted` / 1 `detected`). **Nothing was changed on
> the node** — the recovery is self-clearing, exactly as in burst 1, and the underlying exposure is untouched:
> the faults are still invisible to `/api/health`, still single-node, and a fresh burst would still take out
> most of the fleet. Two bursts four days apart, both self-clearing, is the pattern to watch; this is a standing
> risk, not a resolved incident.
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven
@@ -188,6 +199,27 @@ a second provider pin on a subset of jobs, or router-side failover between `omni
cost of leaving it open: **two of the last four days took double-digit failures from this one node**, and every
model-calling cron job still has exactly one route.
## Update 2026-10-21 (audit of the 2026-10-20 window): the burst did **not** resume
The 2026-10-19 burst was followed by **2026-10-20's 168 `completed` / 0 `failed`** — a clean day and the
first zero-failure day since 2026-10-12, with **zero** rows carrying the resource-pressure string and **zero
new incident rows** opened for the whole window. The 503 fault's last recorded failure is therefore
**2026-10-19**, and per [[systems/scheduled-tasks]] the page status returns to **`monitoring`** (18–24 h of
clean running, not a fix).
**What did *not* change, and matters more than the clean day:**
- **No failover was added.** 20 of 28 jobs still pin `custom:omni-101` (`smart-route` ×11, `free-stack` ×6,
`deepseek-paid-smart` ×1) or `custom:omni` ×2. One quiet day on a single-route topology is luck, not
margin, and this page has already been wrong once about the burst being over (it was downgraded 44 h
before burst 2).
- **The 8 jobs with no model pin no longer ride the router at all.** The global provider default moved to
`nous` direct (see below), so the fleet is now split across **two** upstream paths for the first time —
worth watching as a separate risk, since a Nous-side fault now has its own blast radius.
- **Global default changed between 2026-10-09 and 2026-10-20:** `provider: custom` →
`nous` (`https://inference-api.nousresearch.com/v1`), `model.default: smart-route` →
`meituan/longcat-2.5-preview:free`. Pinned jobs are unaffected; unpinned jobs are. → [[systems/current-profile]]
## Related
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — the TCP-refusal storm, two days earlier, same node.
@@ -59,8 +59,34 @@ The delivered message, stored verbatim as the run's assistant output:
`custom:omni-101` alias family that produced the 2026-10-14 → 10-15 connection-refused storm and the
2026-10-16 → 10-17 resource-pressure 503 burst. Single-node topology unchanged.
- **Second recorded surface: an interactive session, 2026-10-20 18:37 AWST — and a different guard fired.**
Session `20261020_183748_6d40ea` (`source: desktop`, `model: minimax/minimax-m3.1-flash:free`, 2 messages,
0 tool calls, `end_reason: agent_close`) holds **one user question** — *“i want to use the nous oauth dashboard
aut for this gateway how do i do that. can i also set up a gateway token of …”* — and one assistant reply: the
**identical canned substitute**. The turn-end log names the trigger:
`Rebuilt-message restart limit (3) exceeded; ending turn instead of refunding the iteration budget
indefinitely.` → `reason=rebuilt_restart_limit_exceeded model=qwen3.5:397b api_calls=1/150 tool_turns=0
response_len=0` (single occurrence in the retained logs). **No incident row was created** — the ledger relays
*cron* failures only, so a user's unanswered question never enters it. **The question itself was never
answered** → [[current-state]].
- **Open:** whether the canned sentence should be **suppressed** for interactive sessions (where the user is
present and would rather see the real error) while kept for cron (where silence looks like success).
## Recurrence 2026-10-20 (interactive desktop session, rebuild-limit path)
Same visible outcome — a canned non-answer persisted as the assistant's reply — reached by **two routes on the
same day**: the 2026-10-18 cron path (repetition-loop guard) and, on 2026-10-20 18:37, the **rebuilt-message
restart limit**: the conversation loop rebuilt the message list three times, gave up, and ended the turn with
`response_len=0`, leaving the canned line as the only thing the user saw. Correlation with the same-day provider
change is **not established** — the session's own pin was `minimax/minimax-m3.1-flash:free`, while the turn-end
line names `model=qwen3.5:397b`; the two disagree and the audit cannot resolve which model the failing API call
actually went to. The **durable conclusion holds regardless: the canned substitute is emitted by the harness's
failure path generally**, not only by the repetition-loop guard. It is a delivery-integrity problem, not a
stream-parser one.
## Related
- [[systems/scheduled-tasks]]
- [[systems/current-profile]]
- [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
@@ -0,0 +1,92 @@
---
title: Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned (status `unknown`), No Alert
type: incident
status: active
created: 2026-10-21
updated: 2026-10-21
verified_on: 2026-10-21
review_after: 2026-10-28
confidence: high
tags: [gateway, cron, scheduler, lifecycle, restart, silent-failure]
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
---
# Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned, Nothing Alerted
## Headline
On **2026-10-20** the Hermes gateway process died **without running any exit path**. The scheduler came back,
found a cron execution whose owner had vanished mid-run, and closed it as **`unknown`** with the note:
> *Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects
> ran is unknown.*
**No incident row was created, no notification was sent, and the affected job's daily digest has no delivery
row.** This is the first *recorded* instance of the orphaned-execution failure mode, and the first unclean
gateway exit since 2026-09-21.
## Timeline (2026-10-20, AWST)
| Time | Event |
|---|---|
| 08:30 | slot for `08eb86f02d37` — **Gmail Action Digest**, `30 8 * * *`, `model: smart-route` |
| 08:31:03 | execution claimed (`status: running`) |
| 08:50:28 | **last heartbeat of the previous gateway life** (`pid 2496376`, started 2026-10-17 05:14:08 AWST, `rss 394 MB`, `mem_available 12.2 GB`, `swap_used 3.5 GB`) |
| 09:09:01 | ledger records that life as **exited UNCLEANLY** — *“no exit path ran — SIGKILL / OOM / VM death, or a process kill issued by the agent or one of its descendants”*; **`suspected_oom=False`** |
| 09:09:16 | log line `Starting Hermes Gateway...` (the running process reports start time 08:55:09 — the two disagree by ~14 min, so the restart instant is not pinned to the second) |
| ~09:10 | the orphaned execution row closes as **`unknown`** |
## Impact
- **The 2026-10-20 Gmail Action Digest has no delivery row in `deliveries.db` and its execution carries
`delivery_outcome: NULL`.** The ledger therefore offers **no evidence the digest was delivered** — and no
evidence it ran to completion either. Both questions are open by design of the `unknown` status.
- **The failure is invisible from the job table.** `08eb86f02d37` shows `last_status: error` with a **503 from
its 2026-10-19 slot** — i.e. the row that the operator sees is the *previous day's* failure, not this one.
- The incident ledger only ingests *execution errors it can classify*, and the orphan note exists **only in the
execution row**: the string `exited before a durable terminal state` does not appear anywhere in
`~/.hermes/logs/*.log*`. Log-only forensics would miss this entirely.
## How often does this happen?
- **20 unclean exits** are recorded in the retained gateway ledger, spanning **2026-08-01 → 2026-10-20** — but
they cluster hard: **5 in 24 h on 2026-08-14/15** (the worst cluster, during the 5 GB era), 3 in the first
week of August, 4 across 08-19 → 08-24. Since 2026-09-01 there have been only **four** (09-09 ×2, 09-21,
10-20).
- **The 2026-10-20 exit ended a ~27-day clean run.** The preceding unclean life (`pid 249364`) started
2026-09-21 22:11 AWST and last heartbeated 2026-09-23 13:07 AWST.
- **`suspected_oom=False` on every entry**, including this one — at 12.2 GB free and 394 MB RSS this was not
memory pressure. The ledger's own candidate list (SIGKILL / VM death / a `pkill` issued by the agent or a
descendant) points at an operator- or agent-initiated kill, not a crash.
- **Only one execution has ever been orphaned by it.** Across the same ledger window the other 19 exits either
happened between slots or were survived by their runs — this one landed 38 minutes into a live run.
## What would make this recoverable
1. **Orphan the run loudly.** An `unknown` execution that had a delivery target and produced no delivery row is a
*probable non-delivery* — that should raise an incident row like any other job failure.
2. **Record the orphan marker in `agent.log`** as well as the execution row, so a log-only audit can see it.
3. **Re-run on restart** where the job is idempotent (a digest is), instead of leaving the slot empty.
4. **Reconcile `last_status` with the orphaned execution** — the row currently shows the *previous* slot's error,
which is actively misleading.
None of the above is applied; the fix belongs with whoever owns the scheduler's restart path.
## Verification commands
```
# unclean gateway exits, newest first
grep -ah "lifecycle_ledger" ~/.hermes/logs/gateway.log*
# orphaned executions
sqlite3 ~/.hermes/.hermes/cron/executions.db \
"SELECT job_id,started_at,status,error FROM executions WHERE status='unknown' ORDER BY started_at DESC LIMIT 5"
# was the slot delivered?
sqlite3 ~/.hermes/.hermes/cron/deliveries.db \
"SELECT * FROM deliveries WHERE created_at LIKE '2026-10-20%'"
```
## Related
- [[systems/scheduled-tasks]]
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — the other 2026-10-20 non-delivery (a canned substitute served to a user).
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the other "the row is green, the work did not happen" case found in this audit.