🦉 wiki audit 2026-10-21: gateway unclean-exit orphaned run (new), Gitea config-backup push dead since 2026-08-08 (new), model default → nous, 503 back to monitoring, canned-substitute trigger corrected

This commit is contained in:
Tony0410
2026-10-21 00:11:44 +08:00
parent fbd4f297eb
commit b662254569
10 changed files with 386 additions and 25 deletions
+49 -4
View File
@@ -3,16 +3,61 @@ title: Current State
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-10-21
verified_on: 2026-10-20 verified_on: 2026-10-21
review_after: 2026-10-27 review_after: 2026-10-28
confidence: high confidence: high
tags: [system] tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
--- ---
# Current State # Current State
Updated: 2026-10-20 Updated: 2026-10-21
## Recent Changes (2026-10-21, nightly audit; live probes 2026-10-21 00:03-00:15 AWST)
- **A clean day: 2026-10-20 closed 168 `completed` / 0 `failed`** (plus 1 `unknown`, plus this audit) — the first
zero-failure day since **2026-10-12**, ending a seven-day run of failures. **The `omni-101` 503 fault did not
recur** (the last failure row carrying the resource-pressure string remains 2026-10-19), and **zero new
incident rows were opened** across the whole window; 4 closed, so **unclosed rows fall 13 → 9** (8 `alerted` /
1 `detected`). → [[incidents/2026-10-16-omniroute-503-resource-pressure]] (status back to **`monitoring`**)
- **A gateway unclean exit orphaned a cron run, and nothing reported it.** `08eb86f02d37` (Gmail Action Digest,
08:30 slot) was claimed 08:31:03 and closed as **`unknown`**: *“Scheduler restarted after this execution's owner
exited before a durable terminal state; whether side effects ran is unknown.”* The previous gateway life
(`pid 2496376`, started 2026-10-17 05:14) was recorded **UNCLEANLY** at 09:09:01 — last heartbeat 08:50:28, RSS
394 MB, 12.2 GB free, **`suspected_oom=False`**. `delivery_outcome` is NULL and **no delivery row exists for
the slot**, so the ledger offers no evidence the digest arrived; **no incident row was created**, and the job
table still shows the *previous* slot's 503. First recorded orphan of its kind; first unclean exit since
2026-09-21. → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]
- **⚠ A green cron row is not evidence the work happened — second instance, and this one is 2.5 months old.**
`e27da6c34bbb` (Hermes Config Backup to Gitea, daily 11:00) reads **`last_status: ok`**, `last_error: null`, and
its 2026-10-20 execution reads `completed` — while **nothing has reached Gitea since 2026-08-07 11:00:47**.
The repo is **75 commits ahead of `origin/main`**; `backup.log` ends
`could not read Username for 'https://gitea.kangaroo-eel.ts.net'` → `✗ Push to Gitea FAILED` →
`--- Backup complete: FAIL ---`. `origin` is HTTPS with no credential in the URL and no helper, so the
non-interactive push cannot authenticate: **no off-box `~/.hermes` backup since 2026-08-08**. Telegram failure
notices have been firing all along (last 2026-10-20 11:00, `message_id 8447`); the AgentMail step failed.
→ [[incidents/2026-08-08-gitea-config-backup-push-dead]], [[infrastructure/backups]]
- **The global model default moved off OmniRoute.** `model.provider: custom` → **`nous`**
(`https://inference-api.nousresearch.com/v1`), `model.default: smart-route` → **`meituan/longcat-2.5-preview:free`**
(not in place 2026-10-09 19:42; in place by 2026-10-20 11:00). **Pins are unchanged** — 20 of 28 jobs still pin
OmniRoute (`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2) — but the **8 jobs
with no pin now call Nous direct instead of the router**, so the fleet runs two upstream paths for the first
time. → [[systems/current-profile]], [[systems/scheduled-tasks]]
- **A user question was answered with a canned non-answer, and the request is still outstanding.** Desktop
session `20261020_183748_6d40ea` (2026-10-20 18:37, `model: minimax/minimax-m3.1-flash:free`) asked *how to use
Nous OAuth dashboard auth for the gateway* and *whether a gateway token could be configured*; the turn ended
`reason=rebuilt_restart_limit_exceeded … response_len=0` and the only assistant message is *“Your request was
not processed. Send it again if you still want me to carry it out.”* **Nothing was configured in response to
that request** — it needs re-asking. → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
- **`dashboard.basic_auth` now carries a `secret` and `session_ttl_seconds: 43200`** (present 2026-10-21 00:0x).
The plugin's *“no 'secret' configured; generating a random per-process signing key … sessions will not survive
a restart”* warning last appeared 2026-10-20 18:45:14, so dashboard sessions should now persist across
restarts.
- **Interactive sessions returned after weeks of cron-only windows:** `talaria-ios` (10:49 → 19:27, 25 messages,
12 tool calls, *“Get Australian tax return done”*, `provider: custom:command-code`) and the `desktop` session
above. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); fleet unchanged at **28 jobs / 25
enabled / 3 disabled**. → [[systems/scheduled-tasks]]
## Recent Changes (2026-10-20, nightly audit; live probes 2026-10-20 00:03-00:12 AWST) ## Recent Changes (2026-10-20, nightly audit; live probes 2026-10-20 00:03-00:12 AWST)
@@ -0,0 +1,90 @@
---
title: Hermes Config Backup to Gitea — Push Has Been Dead Since 2026-08-08 While the Job Reads `ok`
type: incident
status: active
created: 2026-10-21
updated: 2026-10-21
verified_on: 2026-10-21
review_after: 2026-10-28
confidence: high
tags: [backup, gitea, cron, git, green-vs-actual]
sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki]
---
# Hermes Config Backup to Gitea — Off-Box Config Backup Dead Since 2026-08-08
## Headline
The daily **Hermes Config Backup to Gitea** job (`e27da6c34bbb`, `0 11 * * *`, no-agent, `hermes-config-backup.sh`)
**commits locally every day and fails to push every day — and its cron row still reads `last_status: ok`.**
The last commit that actually reached Gitea is **2026-08-07 11:00:47**; every nightly commit since
**2026-08-08** is sitting in the local repo only.
## Evidence (live, 2026-10-21 AWST)
| Probe | Result |
|---|---|
| `git log origin/main -1` in `~/.hermes/config-backup` | **`2026-08-07 11:00:47 +0800 1e1eb208 🦝 nightly backup: 2026-08-07 11:00:29 AWST`** |
| first unpushed commit | `2026-08-08 11:00:43 c7d25358 🦝 nightly backup: 2026-08-08 11:00:59 AWST` |
| `git rev-list --count origin/main..main` | **75 commits ahead of the last known remote commit** |
| `git remote -v` | `origin` = `https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` — **HTTPS, no userinfo, no credential helper** |
| `backup.log` tail (2026-10-20 11:00:17) | `fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net': No such device or address` → `✗ Push to Gitea FAILED` |
| same run, second remote | `fatal: 'github' does not appear to be a git repository` → `✗ Push to GitHub FAILED` |
| same run, notification step | AgentMail: `send failed`; **Telegram: delivered** (`🚨 BACKUP FAILED — 2026-10-20 11:00:17 AWST`, `message_id 8447`) |
| same run, final line | `Your branch is ahead of 'origin/main' by 75 commits` / `--- Backup complete: FAIL ---` |
| `.backup_history` | last 30 entries **all `FAIL`** (the file keeps 30 — the true streak is ~75) |
| `jobs.json` entry `e27da6c34bbb` | `last_status: ok`, `last_error: null`, enabled |
| executions ledger, 2026-10-20 11:00:14 | `status: completed` |
**This is the second instance of the pattern in [[systems/scheduled-tasks]]'s notes: a green cron row is not
evidence the job's work happened.** (The first was `c0a9fa314542`, GitHub Trending Radar, green while its
delivery token was revoked.)
## Root cause
The remote is HTTPS with the credential expected to come from git's credential machinery, and the cron worker
has no credential helper, no `~/.git-credentials` entry for that host, and no userinfo in the URL — so a
non-interactive push cannot authenticate and dies with `could not read Username`. Nothing in the script treats
a failed push as failure of the job (it still commits, still sends the Telegram notify, and exits 0), so the
scheduler records `completed` / `ok`.
Contrast with the wiki repo, which pushes reliably because it carries the token **in the remote URL**
(`http://<token>@gitea:3000/Anthony/agent-estate-wiki.git`).
The secondary failures are cosmetic by comparison: the `github` remote named in the script does not exist in
this repo, and the AgentMail notify step also fails while the Telegram notify succeeds.
## Impact
- **No configuration state has left the box since 2026-08-07.** A host loss would take `~/.hermes/config.yaml`,
the cron job definitions, the skills tree and the dashboard server sources with it — only a 2026-07-22 full
zip and the wiki repo survive off-box (see [[infrastructure/backups]]).
- The **2026-10-20 20:06 `dashboard.basic_auth.secret` change and the 2026-10-20 `model.default` change are
among the 75 unpushed commits** — the config history the wiki cites for those decisions lives only locally.
- Daily failure notifications have been arriving in Telegram for ~75 days, so the signal exists; what is
missing is the *job-level* red.
## Proposed fix (not applied — needs Anthony's go-ahead)
1. Give `origin` a credential the cron worker can use, the same way the wiki repo does — either userinfo in
the URL or a `GIT_ASKPASS`/helper wired into the script's environment.
2. Push the 75-commit backlog **after** confirming the token is still valid (a 75-day-old URL token may be stale).
3. Delete or create the `github` remote so the script stops erroring on it.
4. Make the script **exit non-zero when the push fails**, so `last_status` stops lying.
5. Re-run and confirm: `git rev-list --count origin/main..main` → `0` and a fresh
`git log origin/main -1` timestamp.
## Verification command set (for the next audit)
```
git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s'
git -C ~/.hermes/config-backup rev-list --count origin/main..main
tail -20 ~/.hermes/config-backup/backup.log
tail -3 ~/.hermes/config-backup/.backup_history
```
## Related
- [[infrastructure/backups]]
- [[systems/scheduled-tasks]]
- [[systems/hermes-agent]]
@@ -1,11 +1,11 @@
--- ---
title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts
type: incident type: incident
status: active status: monitoring
created: 2026-10-18 created: 2026-10-18
updated: 2026-10-20 updated: 2026-10-21
verified_on: 2026-10-20 verified_on: 2026-10-21
review_after: 2026-10-27 review_after: 2026-10-28
confidence: high confidence: high
tags: [cron, router, provider, infrastructure] tags: [cron, router, provider, infrastructure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
@@ -13,10 +13,21 @@ sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-jso
# OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts (burst 1: 2026-10-16 → 10-17; burst 2: 2026-10-19) # OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts (burst 1: 2026-10-16 → 10-17; burst 2: 2026-10-19)
**Status: ACTIVE — this is a recurrence, not a closed incident.** The failure mode came back **twice in four **Status: `MONITORING` as of 2026-10-21 — not closed.** The failure mode came back **twice in four
days** (10-16 → 10-17 and 10-19) with an identical signature on an identical node, and the **exact same days** (10-16 → 10-17 and 10-19) with an identical signature on an identical node, and the **exact same
9-model-call blast radius** each time. The node was healthy before, between and after both bursts. 9-model-call blast radius** each time. The node was healthy before, between and after both bursts.
> **2026-10-21 update — burst 2 is over; the page returns to `monitoring` on the 18–24 h rule it set itself.**
> **2026-10-20 ran 168 `completed` / 0 `failed`** (plus one `unknown` orphan — see
> [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]), the **first zero-failure day since 2026-10-12**,
> and **no execution row carries the resource-pressure string** (the last such failure remains 2026-10-19).
> **4 incident rows closed on 10-20** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`,
> `ecc1fc_21a72aa11cc4`), taking unclosed rows **13 → 9** (8 `alerted` / 1 `detected`). **Nothing was changed on
> the node** — the recovery is self-clearing, exactly as in burst 1, and the underlying exposure is untouched:
> the faults are still invisible to `/api/health`, still single-node, and a fresh burst would still take out
> most of the fleet. Two bursts four days apart, both self-clearing, is the pattern to watch; this is a standing
> risk, not a resolved incident.
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**. **Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01). 10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven **Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven
@@ -188,6 +199,27 @@ a second provider pin on a subset of jobs, or router-side failover between `omni
cost of leaving it open: **two of the last four days took double-digit failures from this one node**, and every cost of leaving it open: **two of the last four days took double-digit failures from this one node**, and every
model-calling cron job still has exactly one route. model-calling cron job still has exactly one route.
## Update 2026-10-21 (audit of the 2026-10-20 window): the burst did **not** resume
The 2026-10-19 burst was followed by **2026-10-20's 168 `completed` / 0 `failed`** — a clean day and the
first zero-failure day since 2026-10-12, with **zero** rows carrying the resource-pressure string and **zero
new incident rows** opened for the whole window. The 503 fault's last recorded failure is therefore
**2026-10-19**, and per [[systems/scheduled-tasks]] the page status returns to **`monitoring`** (18–24 h of
clean running, not a fix).
**What did *not* change, and matters more than the clean day:**
- **No failover was added.** 20 of 28 jobs still pin `custom:omni-101` (`smart-route` ×11, `free-stack` ×6,
`deepseek-paid-smart` ×1) or `custom:omni` ×2. One quiet day on a single-route topology is luck, not
margin, and this page has already been wrong once about the burst being over (it was downgraded 44 h
before burst 2).
- **The 8 jobs with no model pin no longer ride the router at all.** The global provider default moved to
`nous` direct (see below), so the fleet is now split across **two** upstream paths for the first time —
worth watching as a separate risk, since a Nous-side fault now has its own blast radius.
- **Global default changed between 2026-10-09 and 2026-10-20:** `provider: custom` →
`nous` (`https://inference-api.nousresearch.com/v1`), `model.default: smart-route` →
`meituan/longcat-2.5-preview:free`. Pinned jobs are unaffected; unpinned jobs are. → [[systems/current-profile]]
## Related ## Related
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — the TCP-refusal storm, two days earlier, same node. - [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — the TCP-refusal storm, two days earlier, same node.
@@ -59,8 +59,34 @@ The delivered message, stored verbatim as the run's assistant output:
`custom:omni-101` alias family that produced the 2026-10-14 → 10-15 connection-refused storm and the `custom:omni-101` alias family that produced the 2026-10-14 → 10-15 connection-refused storm and the
2026-10-16 → 10-17 resource-pressure 503 burst. Single-node topology unchanged. 2026-10-16 → 10-17 resource-pressure 503 burst. Single-node topology unchanged.
- **Second recorded surface: an interactive session, 2026-10-20 18:37 AWST — and a different guard fired.**
Session `20261020_183748_6d40ea` (`source: desktop`, `model: minimax/minimax-m3.1-flash:free`, 2 messages,
0 tool calls, `end_reason: agent_close`) holds **one user question** — *“i want to use the nous oauth dashboard
aut for this gateway how do i do that. can i also set up a gateway token of …”* — and one assistant reply: the
**identical canned substitute**. The turn-end log names the trigger:
`Rebuilt-message restart limit (3) exceeded; ending turn instead of refunding the iteration budget
indefinitely.` → `reason=rebuilt_restart_limit_exceeded model=qwen3.5:397b api_calls=1/150 tool_turns=0
response_len=0` (single occurrence in the retained logs). **No incident row was created** — the ledger relays
*cron* failures only, so a user's unanswered question never enters it. **The question itself was never
answered** → [[current-state]].
- **Open:** whether the canned sentence should be **suppressed** for interactive sessions (where the user is
present and would rather see the real error) while kept for cron (where silence looks like success).
## Recurrence 2026-10-20 (interactive desktop session, rebuild-limit path)
Same visible outcome — a canned non-answer persisted as the assistant's reply — reached by **two routes on the
same day**: the 2026-10-18 cron path (repetition-loop guard) and, on 2026-10-20 18:37, the **rebuilt-message
restart limit**: the conversation loop rebuilt the message list three times, gave up, and ended the turn with
`response_len=0`, leaving the canned line as the only thing the user saw. Correlation with the same-day provider
change is **not established** — the session's own pin was `minimax/minimax-m3.1-flash:free`, while the turn-end
line names `model=qwen3.5:397b`; the two disagree and the audit cannot resolve which model the failing API call
actually went to. The **durable conclusion holds regardless: the canned substitute is emitted by the harness's
failure path generally**, not only by the repetition-loop guard. It is a delivery-integrity problem, not a
stream-parser one.
## Related ## Related
- [[systems/scheduled-tasks]] - [[systems/scheduled-tasks]]
- [[systems/current-profile]]
- [[incidents/2026-10-16-omniroute-503-resource-pressure]] - [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] - [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
@@ -0,0 +1,92 @@
---
title: Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned (status `unknown`), No Alert
type: incident
status: active
created: 2026-10-21
updated: 2026-10-21
verified_on: 2026-10-21
review_after: 2026-10-28
confidence: high
tags: [gateway, cron, scheduler, lifecycle, restart, silent-failure]
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
---
# Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned, Nothing Alerted
## Headline
On **2026-10-20** the Hermes gateway process died **without running any exit path**. The scheduler came back,
found a cron execution whose owner had vanished mid-run, and closed it as **`unknown`** with the note:
> *Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects
> ran is unknown.*
**No incident row was created, no notification was sent, and the affected job's daily digest has no delivery
row.** This is the first *recorded* instance of the orphaned-execution failure mode, and the first unclean
gateway exit since 2026-09-21.
## Timeline (2026-10-20, AWST)
| Time | Event |
|---|---|
| 08:30 | slot for `08eb86f02d37` — **Gmail Action Digest**, `30 8 * * *`, `model: smart-route` |
| 08:31:03 | execution claimed (`status: running`) |
| 08:50:28 | **last heartbeat of the previous gateway life** (`pid 2496376`, started 2026-10-17 05:14:08 AWST, `rss 394 MB`, `mem_available 12.2 GB`, `swap_used 3.5 GB`) |
| 09:09:01 | ledger records that life as **exited UNCLEANLY** — *“no exit path ran — SIGKILL / OOM / VM death, or a process kill issued by the agent or one of its descendants”*; **`suspected_oom=False`** |
| 09:09:16 | log line `Starting Hermes Gateway...` (the running process reports start time 08:55:09 — the two disagree by ~14 min, so the restart instant is not pinned to the second) |
| ~09:10 | the orphaned execution row closes as **`unknown`** |
## Impact
- **The 2026-10-20 Gmail Action Digest has no delivery row in `deliveries.db` and its execution carries
`delivery_outcome: NULL`.** The ledger therefore offers **no evidence the digest was delivered** — and no
evidence it ran to completion either. Both questions are open by design of the `unknown` status.
- **The failure is invisible from the job table.** `08eb86f02d37` shows `last_status: error` with a **503 from
its 2026-10-19 slot** — i.e. the row that the operator sees is the *previous day's* failure, not this one.
- The incident ledger only ingests *execution errors it can classify*, and the orphan note exists **only in the
execution row**: the string `exited before a durable terminal state` does not appear anywhere in
`~/.hermes/logs/*.log*`. Log-only forensics would miss this entirely.
## How often does this happen?
- **20 unclean exits** are recorded in the retained gateway ledger, spanning **2026-08-01 → 2026-10-20** — but
they cluster hard: **5 in 24 h on 2026-08-14/15** (the worst cluster, during the 5 GB era), 3 in the first
week of August, 4 across 08-19 → 08-24. Since 2026-09-01 there have been only **four** (09-09 ×2, 09-21,
10-20).
- **The 2026-10-20 exit ended a ~27-day clean run.** The preceding unclean life (`pid 249364`) started
2026-09-21 22:11 AWST and last heartbeated 2026-09-23 13:07 AWST.
- **`suspected_oom=False` on every entry**, including this one — at 12.2 GB free and 394 MB RSS this was not
memory pressure. The ledger's own candidate list (SIGKILL / VM death / a `pkill` issued by the agent or a
descendant) points at an operator- or agent-initiated kill, not a crash.
- **Only one execution has ever been orphaned by it.** Across the same ledger window the other 19 exits either
happened between slots or were survived by their runs — this one landed 38 minutes into a live run.
## What would make this recoverable
1. **Orphan the run loudly.** An `unknown` execution that had a delivery target and produced no delivery row is a
*probable non-delivery* — that should raise an incident row like any other job failure.
2. **Record the orphan marker in `agent.log`** as well as the execution row, so a log-only audit can see it.
3. **Re-run on restart** where the job is idempotent (a digest is), instead of leaving the slot empty.
4. **Reconcile `last_status` with the orphaned execution** — the row currently shows the *previous* slot's error,
which is actively misleading.
None of the above is applied; the fix belongs with whoever owns the scheduler's restart path.
## Verification commands
```
# unclean gateway exits, newest first
grep -ah "lifecycle_ledger" ~/.hermes/logs/gateway.log*
# orphaned executions
sqlite3 ~/.hermes/.hermes/cron/executions.db \
"SELECT job_id,started_at,status,error FROM executions WHERE status='unknown' ORDER BY started_at DESC LIMIT 5"
# was the slot delivered?
sqlite3 ~/.hermes/.hermes/cron/deliveries.db \
"SELECT * FROM deliveries WHERE created_at LIKE '2026-10-20%'"
```
## Related
- [[systems/scheduled-tasks]]
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — the other 2026-10-20 non-delivery (a canned substitute served to a user).
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the other "the row is green, the work did not happen" case found in this audit.
+6 -4
View File
@@ -1,6 +1,6 @@
# Anthony's Systems Wiki # Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure, > Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-10-20 > model routing and automation. Last updated: 2026-10-21
> >
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
@@ -34,12 +34,14 @@
### Integrations ### Integrations
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped** (peer offline 29d; host answers neither `:8065` nor ICMP). - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped** (peer offline 29d; host answers neither `:8065` nor ICMP).
- [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip.
- [[systems/scheduled-tasks]] — **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** - [[systems/scheduled-tasks]] — **As of 2026-10-21 00:0x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12.** The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
- [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. The **2026-10-17** slot failed (`HTTP 503`), but the **10-18 03:00** slot closed the hole: `freshness-20261018.txt` written, index regenerated and committed (`dd53e5f`) — freshness **47 stale / 41 fresh / 7 no-date**. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13). - [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. The **2026-10-17** slot failed (`HTTP 503`), but the **10-18 03:00** slot closed the hole: `freshness-20261018.txt` written, index regenerated and committed (`dd53e5f`) — freshness **47 stale / 41 fresh / 7 no-date**. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13).
## Incidents ## Incidents
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? - [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery.
- [[incidents/2026-10-16-omniroute-503-resource-pressure]] — **`omniroute-1` load-sheds model calls with `HTTP 503 … resource pressure` — two bursts: 2026-10-16 → 10-17, and again 2026-10-19.** Burst 1: **9 model calls across 7 jobs** over ~19 h, self-cleared (reported then as `10-17 = 162 / 5`; the DB reads **163 / 5** — corrected 2026-10-20). Burst 2, 44 h later with an identical signature and blast radius: **9 model calls across 8 jobs** on 2026-10-19 (06:01 + 07:01 → 12:01, whole day **159 / 13**). `/api/health` read `200 ok` throughout both. **Status back to `active`** — a self-clearing fault that returns twice in four days is a standing risk, not a closed incident. 4 incident rows opened and 2 auto-closed on 10-19 (a third, `ea79a2_747b27556fd0`, closed 10-20 00:04); **13 unclosed rows** overall (12 `alerted` / 1 `detected`); the ledger still tags every 503 row `failure_type: unknown`, so triage on error text. **Single-node topology still open — and every job that failed reads `model: smart-route`, `provider: custom:omni-101`.** - [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]]
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? **Second surface (2026-10-20 18:37):** an interactive `desktop` session received the same canned line after `Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … response_len=0`, so the substitute is emitted by the harness's failure path **generally**, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — **was never answered**.
- [[incidents/2026-10-16-omniroute-503-resource-pressure]] — **`omniroute-1` load-sheds model calls with `HTTP 503 … resource pressure` — two bursts: 2026-10-16 → 10-17, and again 2026-10-19.** Burst 1: **9 model calls across 7 jobs** over ~19 h, self-cleared (reported then as `10-17 = 162 / 5`; the DB reads **163 / 5** — corrected 2026-10-20). Burst 2, 44 h later with an identical signature and blast radius: **9 model calls across 8 jobs** on 2026-10-19 (06:01 + 07:01 → 12:01, whole day **159 / 13**). `/api/health` read `200 ok` throughout both. **Status: `active` → back to `monitoring` on 2026-10-21** — 2026-10-20 ran **168 / 0** with zero rows carrying the resource-pressure string, so the second burst is over; it stays a standing risk until the single-node topology changes. 4 rows closed on 10-20 (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`); **unclosed rows 13 → 9** (8 `alerted` / 1 `detected`); the ledger still tags every 503 row `failure_type: unknown`, so triage on error text. **Single-node topology still open — and every job that failed reads `model: smart-route`, `provider: custom:omni-101`.**
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved) - [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved)
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00) - [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00)
+17
View File
@@ -50,6 +50,23 @@ How Hermes's own configuration/state gets backed up, and what's missing.
> create it on first use). > create it on first use).
- `pre_update_backup: false` means an in-place `hermes update` will not auto-snapshot first — the manual step in [[runbooks/update-hermes-safely]] step 1 is currently the only safety net. - `pre_update_backup: false` means an in-place `hermes update` will not auto-snapshot first — the manual step in [[runbooks/update-hermes-safely]] step 1 is currently the only safety net.
## ⚠ Scheduled config backup — the Gitea mirror has been dead since 2026-08-08
`e27da6c34bbb` (**Hermes Config Backup to Gitea**, daily `0 11 * * *`, `hermes-config-backup.sh`) **commits
locally every night and fails to push every night — while its cron row still reads `last_status: ok` and its
execution row reads `completed`.** Last commit that reached Gitea: **2026-08-07 11:00:47**; the local repo is
**75 commits ahead of `origin/main`**. Root cause is a credential: `origin` is
`https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` — HTTPS, no userinfo, no credential
helper — so the non-interactive push dies on
`fatal: could not read Username … No such device or address` → `✗ Push to Gitea FAILED` →
`--- Backup complete: FAIL ---`. Full evidence and the proposed fix:
[[incidents/2026-08-08-gitea-config-backup-push-dead]].
**Consequence for this page: everything above describes a topology that has not carried `~/.hermes` off the box
since 2026-08-07.** The 2026-07-22 full zip and the wiki/authoring repos remain the only recent off-box copies.
Telegram failure notices *have* been firing daily (last: 2026-10-20 11:00, `message_id 8447`); the AgentMail
notification step failed.
## Relevant runbooks ## Relevant runbooks
- [[runbooks/backup-wiki]] - [[runbooks/backup-wiki]]
- [[runbooks/update-hermes-safely]] - [[runbooks/update-hermes-safely]]
+39
View File
@@ -951,3 +951,42 @@ No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe st
made from *other* agents' sessions; **a full vault mirror pass remains the open call.** made from *other* agents' sessions; **a full vault mirror pass remains the open call.**
- **Not re-run this audit:** the `public/` Quartz rebuild and `wiki-maintain.sh`. Authoring commits - **Not re-run this audit:** the `public/` Quartz rebuild and `wiki-maintain.sh`. Authoring commits
`95fd7d3` → `766882e` → `8da6ddf` → `0248698`, pushed to Gitea `Anthony/agent-estate-wiki` and confirmed at the remote. `95fd7d3` → `766882e` → `8da6ddf` → `0248698`, pushed to Gitea `Anthony/agent-estate-wiki` and confirmed at the remote.
### 2026-10-21 — nightly wiki audit (cron `cbfe434a3b5e`, window 2026-10-20)
- **A clean day — and a failure this ledger cannot see.** 2026-10-20 closed **168 `completed` / 0 `failed`**
(plus 1 `unknown`), the **first zero-failure day since 2026-10-12**, ending a seven-day run. The `omni-101` 503
fault did **not** recur (last failure row 2026-10-19), so
[[incidents/2026-10-16-omniroute-503-resource-pressure]] returns to **`monitoring`**. **Zero incident rows
opened, 4 closed → unclosed 13 → 9** (8 `alerted` / 1 `detected`).
- **New page: [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]].** The gateway died with no exit path at
**2026-10-20 09:09:01** (last heartbeat 08:50:28, `pid 2496376`, `suspected_oom=False`), orphaning the 08:30
**Gmail Action Digest** run (`08eb86f02d37`, claimed 08:31:03) as **`unknown`** — *"owner exited before a
durable terminal state"*. No delivery row for the slot, **no incident row created**, and the job table still
shows the *previous* slot's 503. First recorded orphan; first unclean exit since 2026-09-21 (20 unclean exits
since 2026-08-01, 5 of them clustered on 2026-08-14/15).
- **New page: [[incidents/2026-08-08-gitea-config-backup-push-dead]].** `e27da6c34bbb` reads `ok` and its
executions read `completed` nightly while **nothing has reached Gitea since 2026-08-07 11:00:47** — **75
unpushed commits**, `fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'`. Second instance
of the "green row, dead job" pattern after the GitHub Trending Radar token revocation.
- **Corrected the 2026-10-18 canned-substitute page.** Its "second surface" section attributed the
interactive-session substitution to an `HTTP 404` Model-not-found; the retained log evidence is
`Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … api_calls=1/150
tool_turns=0 response_len=0`. The page now carries the verified trigger and flags the unresolved model
disagreement (session pin `minimax/minimax-m3.1-flash:free` vs `model=qwen3.5:397b` in the turn-end line).
**The user's question in that session — Nous OAuth dashboard auth plus a gateway token — was never answered,
and is recorded as outstanding in [[current-state]].**
- **Pages updated:** [[current-state]] (new Recent Changes section), [[systems/scheduled-tasks]] (new count bullet
— non-clean `last_status` 12 → 9, config-backup row annotated, frontmatter), [[systems/current-profile]] (model
default moved to `nous` / `meituan/longcat-2.5-preview:free`, no `fallbacks` key, pins unchanged),
[[infrastructure/backups]] (the scheduled Gitea push has been dead since 2026-08-08),
[[incidents/2026-10-16-omniroute-503-resource-pressure]] (status back to `monitoring` + dated update),
[[incidents/2026-10-18-degenerate-stream-canned-substitute]] (trigger corrected), `index.md`, this log.
- **Fleet:** 28 jobs / 25 enabled / 3 disabled (unchanged). **20 of 28 pin OmniRoute**; the **8 unpinned jobs
moved upstream to `nous`** with the global default change — the fleet now runs two upstream paths. **2
interactive sessions in the window** (`talaria-ios` tax-return work; the `desktop` turn served the canned
non-answer) — the first non-cron sessions after weeks of cron-only windows.
- **Nothing removed or archived** — no page became obsolete, and no one-way sync was run.
- **No secrets written.** The gateway token pasted in the 2026-10-20 desktop session was **not** transcribed into
any page; both the incident and current-state entries describe the request without the credential. Host names,
IPs, job/incident IDs, ports and probe status codes only.
+24 -7
View File
@@ -3,11 +3,12 @@ title: Current Hermes Profile
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-02 updated: 2026-10-21
verified_on: 2026-10-02 verified_on: 2026-10-21
review_after: 2026-10-28
confidence: high confidence: high
tags: [system, hermes, profile] tags: [system, hermes, profile]
sources: [raw/configs/hermes-config-sanitized.txt, live-config] sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026-10-21]
--- ---
# Current Hermes Profile # Current Hermes Profile
@@ -17,10 +18,26 @@ sources: [raw/configs/hermes-config-sanitized.txt, live-config]
- **Location:** implicit — the top-level `~/.hermes/config.yaml`. There is **no** `~/.hermes/profiles/default/` directory (verified 2026-10-02); `~/.hermes/profiles/` holds only named sub-profiles. - **Location:** implicit — the top-level `~/.hermes/config.yaml`. There is **no** `~/.hermes/profiles/default/` directory (verified 2026-10-02); `~/.hermes/profiles/` holds only named sub-profiles.
- **Other profiles present:** `ops` only (`~/.hermes/profiles/ops/`) — the Tusk elephant persona, with its own SOUL.md, skills, plugins and memories (last modified 2026-09-30). Other profiles do **not** affect this session; see the profile-isolation note below. - **Other profiles present:** `ops` only (`~/.hermes/profiles/ops/`) — the Tusk elephant persona, with its own SOUL.md, skills, plugins and memories (last modified 2026-09-30). Other profiles do **not** affect this session; see the profile-isolation note below.
## Key Settings (verified against live config 2026-10-02) ## Key Settings (model routing re-verified against live config 2026-10-21; remaining items as recorded 2026-10-02)
- **Model default:** `smart-route` (was `freellmapi-auto` at the 2026-08-28 snapshot; Anthony rejects the free routing combos as a default)
- **Default provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute). The 2026-08-28 record said `custom:litellm` → `http://litellm:4000/v1` — that is now only the *last* entry in the fallback chain (`custom` / `or-inclusionai_ling-2_6-1t`). ### ⚠ Model default moved OFF OmniRoute (2026-10-20)
- **Fallback chain:** 6 providers, ending at `litellm:4000` - **Provider:** **`nous`** — `model.base_url: https://inference-api.nousresearch.com/v1`, `api_mode: chat_completions`.
- **Model default:** **`meituan/longcat-2.5-preview:free`**
- **When:** `config.yaml` last written **2026-10-20 22:01:38 AWST**. The move was **not** in place at the
2026-10-09 19:42 snapshot and **was** in place by 2026-10-20 11:00. Previous values (2026-10-02 record):
provider `custom` → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`, provider id `custom:omni-101`),
default `smart-route`.
- **The `model` block now carries only** `base_url`, `default`, `provider`, `api_mode`, `persist_switch_by_default`.
**The 6-provider fallback chain recorded at the 2026-10-02 snapshot is gone** — there is no `fallbacks` key under
`model`, so an unreachable `nous` is no longer covered by the old `custom` / `or-inclusionai_ling-2_6-1t` →
`custom:litellm` → `litellm:4000` chain.
- **⚠ Cron pins did NOT move with it.** **20 of the 28 scheduled jobs still pin OmniRoute explicitly**
(`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). **The 8 jobs with no pin are
the ones whose upstream changed** — they now call `nous` direct instead of the router. The fleet therefore
runs **two upstream paths for the first time**. → [[systems/scheduled-tasks]]
- **Routing posture:** no fallback chain under `model`; per-job pins are the only routing control that remains.
### Other settings (recorded 2026-10-02 — not re-verified 2026-10-21)
- **Auxiliary vision model:** `stepfun/step-3.7-flash:free` on `nous` - **Auxiliary vision model:** `stepfun/step-3.7-flash:free` on `nous`
- **Switch persistence:** `persist_switch_by_default: false` - **Switch persistence:** `persist_switch_by_default: false`
- **Timezone:** `Australia/Perth` - **Timezone:** `Australia/Perth`
+6 -5
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-10-21
verified_on: 2026-10-20 verified_on: 2026-10-21
review_after: 2026-10-27 review_after: 2026-10-28
confidence: high confidence: high
tags: [cron, scheduled, automation] tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db] sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks # Scheduled Tasks
## Count ## Count
- **⚠ Current as of 2026-10-20 00:09 AWST (`jobs.json.updated_at 2026-10-20T00:09:07+08:00`) — the `omni-101` 503 fault is BACK, and it is now the fleet's defining risk.** Window 2026-10-19: **159 `completed` / 13 `failed`**, against 10-18's 168 / 1 and 10-17's 163 / 5. **9 genuine model-call failures across 8 jobs** all carry the identical `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` on `custom:omni-101`, in two clusters — **06:01** (one job) and **07:01 → 12:01** (seven more runs across seven jobs). The other **4 rows are derivative Mnemosyne Watchdog exits** (`96eb95ce12f2` at 06:16, 06:31, 12:16, 12:31) that merely quote the coverage monitor's error. This is the **second burst in four days** on the same node with the same blast radius, while `/api/health` read `200 {"status":"ok"}` throughout — so [[incidents/2026-10-16-omniroute-503-resource-pressure]] is back to **`status: active`** (it had been downgraded to `monitoring` 44 h earlier). **Fleet unchanged at 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` 7 → 12** (10 `error` + 2 `delivery_failed`); **unclosed incident rows 13** (12 `alerted` / 1 `detected`) — 4 opened and 2 closed on 10-19. Every `failure_streak` still reads ≤ 2 and the 10-20 00:00 slots ran clean, so nothing is being suppressed yet. **15 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **⚠ Current as of 2026-10-21 00:0x AWST (`jobs.json.updated_at 2026-10-21T00:07:08+08:00`) — a clean day, one orphaned run, and one long-running silent failure.** Window 2026-10-20: **168 `completed` / 0 `failed`** (plus 1 `unknown` and this audit) — **the first zero-failure day since 2026-10-12**, ending a seven-day run of failures, and **the `omni-101` 503 fault did not recur** (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). **Zero incident rows opened and 4 closed** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`), so **unclosed rows fall 13 → 9** (8 `alerted` / 1 `detected`). ⚠ **The exception is a failure this ledger does not record:** `08eb86f02d37` (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed **`unknown`** when the gateway exited uncleanly — *“Scheduler restarted after this execution's owner exited before a durable terminal state”*; no delivery row exists for the slot and no incident row was raised → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); every `failure_streak` ≤ 2. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** ⚠ **Discovery — green row, dead job:** `e27da6c34bbb` (Config Backup, 11:00) reads `ok` while its Gitea push has failed **every night since 2026-08-08** (75 unpushed commits) → [[incidents/2026-08-08-gitea-config-backup-push-dead]]. **Model pins unchanged** — 20 of 28 jobs still pin OmniRoute — but the **8 unpinned jobs moved upstream to `nous` direct** with the global default change, so the fleet now has two upstream paths. **2 interactive sessions in the window** (`talaria-ios` tax work; a `desktop` turn that received the canned non-answer). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **Current as of 2026-10-20 00:09 AWST *(superseded by the 2026-10-21 entry above)* — the `omni-101` 503 fault is BACK, and it is now the fleet's defining risk.** Window 2026-10-19: **159 `completed` / 13 `failed`**, against 10-18's 168 / 1 and 10-17's 163 / 5. **9 genuine model-call failures across 8 jobs** all carry the identical `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` on `custom:omni-101`, in two clusters — **06:01** (one job) and **07:01 → 12:01** (seven more runs across seven jobs). The other **4 rows are derivative Mnemosyne Watchdog exits** (`96eb95ce12f2` at 06:16, 06:31, 12:16, 12:31) that merely quote the coverage monitor's error. This is the **second burst in four days** on the same node with the same blast radius, while `/api/health` read `200 {"status":"ok"}` throughout — so [[incidents/2026-10-16-omniroute-503-resource-pressure]] is back to **`status: active`** (it had been downgraded to `monitoring` 44 h earlier). **Fleet unchanged at 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` 7 → 12** (10 `error` + 2 `delivery_failed`); **unclosed incident rows 13** (12 `alerted` / 1 `detected`) — 4 opened and 2 closed on 10-19. Every `failure_streak` still reads ≤ 2 and the 10-20 00:00 slots ran clean, so nothing is being suppressed yet. **15 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **Current as of 2026-10-19 00:20 AWST *(superseded by the 2026-10-20 entry above)* (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]] - **Current as of 2026-10-19 00:20 AWST *(superseded by the 2026-10-20 entry above)* (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
- **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01; DB re-read 2026-10-20: **163 / 5** — one row was double-counted at the time) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01; DB re-read 2026-10-20: **163 / 5** — one row was double-counted at the time) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] - **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
@@ -66,7 +67,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 | | weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes. ⚠ **The 2026-10-16 23:59 slot failed** on the `omni-101` 503, so the 2026-10-15 → 2026-10-17 span went unaudited; incident `cbfe43_9b89947a046a` is **`detected`** and clears only on the next successful 23:59 run (the 2026-10-17 run, which produced this entry). **Not** a stale session lock — the execution row carries the explicit 503 error text | | Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes. ⚠ **The 2026-10-16 23:59 slot failed** on the `omni-101` 503, so the 2026-10-15 → 2026-10-17 span went unaudited; incident `cbfe43_9b89947a046a` is **`detected`** and clears only on the next successful 23:59 run (the 2026-10-17 run, which produced this entry). **Not** a stale session lock — the execution row carries the explicit 503 error text |
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ **The Quartz rebuild step is a no-op in practice** — no `quartz.config.ts` exists and `npx` is unavailable, so the static site in `public/` is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit `8c63ef2`) — a **second consecutive automatic success**, verified. The script itself was never broken. ⚠ **The 2026-10-17 03:00 slot failed** on the same `omni-101` 503 — the first miss since the **2026-10-15** storm slot (which died with `RuntimeError: Connection error.`; the 10-16 slot succeeded). `raw/stocktake/freshness-20261017.txt` was not written and `systems/skills-index.md` stays stamped **2026-10-16** (the 10-16 slot succeeded, so `freshness-20261016.txt` exists); the **2026-10-18 03:00** slot closes it — nothing was regenerated by hand, because a manual run would race the scheduled one. Incident `1a8e90_3c9c1d573cde` `detected`. Note the ledger tags this row `failure_type: unknown` even though the execution carries an explicit 503 message | | wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ **The Quartz rebuild step is a no-op in practice** — no `quartz.config.ts` exists and `npx` is unavailable, so the static site in `public/` is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit `8c63ef2`) — a **second consecutive automatic success**, verified. The script itself was never broken. ⚠ **The 2026-10-17 03:00 slot failed** on the same `omni-101` 503 — the first miss since the **2026-10-15** storm slot (which died with `RuntimeError: Connection error.`; the 10-16 slot succeeded). `raw/stocktake/freshness-20261017.txt` was not written and `systems/skills-index.md` stays stamped **2026-10-16** (the 10-16 slot succeeded, so `freshness-20261016.txt` exists); the **2026-10-18 03:00** slot closes it — nothing was regenerated by hand, because a manual run would race the scheduled one. Incident `1a8e90_3c9c1d573cde` `detected`. Note the ledger tags this row `failure_type: unknown` even though the execution carries an explicit 503 message |
| 🦝 Hermes Config Backup to Gitea (Daily) | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent | | 🦝 Hermes Config Backup to Gitea (Daily) ⚠ **push has failed every night since 2026-08-08 — 75 local commits unpushed, job still reads `ok`** → [[incidents/2026-08-08-gitea-config-backup-push-dead]] | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
## Removed Jobs ## Removed Jobs
| Name | Removed | Reason | | Name | Removed | Reason |