diff --git a/current-state.md b/current-state.md index f379a22..270002f 100644 --- a/current-state.md +++ b/current-state.md @@ -3,18 +3,45 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-25 -verified_on: 2026-10-25 -review_after: 2026-11-01 +updated: 2026-10-27 +verified_on: 2026-10-27 +review_after: 2026-11-03 confidence: high tags: [system] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe] --- # Current State -Updated: 2026-10-25 +Updated: 2026-10-27 -## Recent Changes (2026-10-25, nightly audit; live probes 2026-10-25 00:1x–00:5x AWST) +## Recent Changes (2026-10-27, nightly audit; live probes 2026-10-27 00:1x–00:5x AWST) + +- **A *planned* gateway restart still orphaned a live cron run — and this time it alerted loudly.** A planned + `SIGTERM` at **2026-10-26 14:33:35** produced a shutdown drain that reported **`timed_out=True` after + just 1.35 s** of its 30 s budget and interrupted `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) + **2 h 34 m** in. It closed **`failed`** on a **new** string — `lost its durable fire claim ownership` — + against the 2026-10-20 orphan's silent `unknown`, and raised **two** `cron_incidents` rows (14:33:37 + *unverified / reconcile admission*, 14:35:59 *fire claim*) that are still `alerted`; the failure **was + delivered** at 14:35:59. Two Mnemosyne Watchdog reds followed and **self-resolved at 15:16:12**. Window + 2026-10-26 settled **169 / 3** and **all three failures are this one fault** — so the fleet's failure + count is one interruption, not a systemic turn. Fleet unchanged at **28 jobs (25 enabled / 3 disabled)**; + **non-clean `last_status` 5 → 4**; **unclosed incident rows 2 → 3**; **pins unchanged**. → + new page [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] +- **One incident row closed, and the 2026-10-16 503 burst is fully shut.** `c0a9fa_764558b5ae83` + (GitHub Trending Radar) **auto-closed 09:13:00** when its 09:00 slot ran clean — the last open row of the + resource-pressure fault → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. No execution row + carries the 503 string on 10-25 or 10-26. The job stays **`monitoring`**: single-node exposure unchanged. +- **Two quiet degradations worth naming.** `fad5a86979d4`'s 03:08 slot **completed but delivered + `delivery_failed`** — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead + thread at 14:33:37); and the **radar's 07:00 slot closed `completed` at 09:10 while delivering + “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters, batch finished by + hand. That morning the cover-letter PDF pipeline moved to **Chromium over the canonical HTML** with model + **`deepseek-paid-smart`** and a **600 s** dashboard timeout → [[runbooks/job-radar]] +- **Also updated:** `incidents/2026-10-26-planned-restart-orphaned-cron-run.md` (new), + `incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md` (recurrence), `incidents/2026-10-16-omniroute-503-resource-pressure.md`, + `runbooks/job-radar.md`, `systems/scheduled-tasks.md`, `index.md`, `log.md`. **Nothing removed or archived.** + +## Recent Changes (2026-10-25, nightly audit; live probes 2026-10-25 00:1x–00:5x AWST) *(superseded by the 2026-10-27 block above)* - **A third consecutive clean cron day — and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at diff --git a/incidents/2026-10-16-omniroute-503-resource-pressure.md b/incidents/2026-10-16-omniroute-503-resource-pressure.md index b1c2a88..4bce263 100644 --- a/incidents/2026-10-16-omniroute-503-resource-pressure.md +++ b/incidents/2026-10-16-omniroute-503-resource-pressure.md @@ -3,9 +3,9 @@ title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bu type: incident status: monitoring created: 2026-10-18 -updated: 2026-10-22 -verified_on: 2026-10-22 -review_after: 2026-10-29 +updated: 2026-10-27 +verified_on: 2026-10-27 +review_after: 2026-11-03 confidence: high tags: [cron, router, provider, infrastructure] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe] @@ -28,6 +28,13 @@ days** (10-16 → 10-17 and 10-19) with an identical signature on an identical n > most of the fleet. Two bursts four days apart, both self-clearing, is the pattern to watch; this is a standing > risk, not a resolved incident. +> **2026-10-26 update — every incident row from this failure mode is now closed.** The last open row, +> `c0a9fa_764558b5ae83` (`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. +> Retry shortly.`, job `c0a9fa314542` GitHub Trending Radar, `alerted` since 2026-10-19 09:02), **auto-closed +> 2026-10-26 09:13:00** when that job's 09:00 slot ran clean. **No execution row carries the resource-pressure +> string on 2026-10-25 or 2026-10-26.** Status stays **`monitoring`** for the same reason as 10-21: the rows are +> shut, the single-node exposure is not. + **Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**. 10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01). **Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven diff --git a/incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md b/incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md index d916073..e61fda2 100644 --- a/incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md +++ b/incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md @@ -3,9 +3,9 @@ title: Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned (status type: incident status: active created: 2026-10-21 -updated: 2026-10-21 -verified_on: 2026-10-21 -review_after: 2026-10-28 +updated: 2026-10-27 +verified_on: 2026-10-27 +review_after: 2026-11-03 confidence: high tags: [gateway, cron, scheduler, lifecycle, restart, silent-failure] sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-sessions, live-probe] @@ -87,6 +87,7 @@ sqlite3 ~/.hermes/.hermes/cron/deliveries.db \ ## Related +- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **recurrence on 2026-10-26, and the loud-orphan gap here is now partly closed.** This one was a **planned `SIGTERM`** (not an unclean exit) and the drain still **timed out after 1.35 s**, interrupting `ea79a2107e36` (Mnemosyne Health Monitor) 2 h 34 m in. Unlike this page's case it closed **`failed`** on a **new** string (`lost its durable fire claim ownership`), **raised two `cron_incidents` rows**, and **delivered** the failure — but the work still did not happen. Any triage keyed on this page's `unknown` / "owner exited before a durable terminal state" wording will miss it. - [[systems/scheduled-tasks]] - [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — the other 2026-10-20 non-delivery (a canned substitute served to a user). - [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the other "the row is green, the work did not happen" case found in this audit. diff --git a/incidents/2026-10-26-planned-restart-orphaned-cron-run.md b/incidents/2026-10-26-planned-restart-orphaned-cron-run.md new file mode 100644 index 0000000..f143ceb --- /dev/null +++ b/incidents/2026-10-26-planned-restart-orphaned-cron-run.md @@ -0,0 +1,103 @@ +--- +title: Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in) +type: incident +status: active +created: 2026-10-27 +updated: 2026-10-27 +verified_on: 2026-10-27 +review_after: 2026-11-03 +confidence: high +tags: [gateway, cron, scheduler, lifecycle, restart, memory, silent-failure] +sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-probe] +--- + +# Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in) + +## Headline + +On **2026-10-26 at 14:33:35 AWST** the gateway received a **planned `SIGTERM`** (`under_systemd=yes`, +`parent_name=systemd`) — and the shutdown drain **still interrupted a running cron execution**, because +the drain reported `timed_out=True` after only **1.35 s**. The orphaned run was +`ea79a2107e36` **Mnemosyne Health Monitor** (12-hourly `0 */12 * * *`), which had been claimed at +**12:01:13** and was **2 h 34 m** into its work. + +This is the **second instance** of the orphaned-cron-run class documented for 2026-10-20 +([[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]), with three material differences: + +| | 2026-10-20 | 2026-10-26 | +|---|---|---| +| Stop type | **unclean exit** (no exit path, `suspected_oom=False`) | **planned `SIGTERM`** under systemd | +| Execution closed as | `unknown` ("owner exited before a durable terminal state") | **`failed`** with a **new** error string | +| Incident row raised | **no** | **yes — two** | +| Failure delivered | no | **yes**, 14:35:59 | + +So the loud-orphan recommendation from the 10-20 page is **now partly implemented** — the run no longer +disappears silently. What it did not do is *finish the work* or *avoid the interruption*. + +## Timeline (all times AWST, 2026-10-26) + +| Time | Event | Source | +|---|---|---| +| 12:01:13 | `ea79a2107e36` Mnemosyne Health Monitor execution claimed, `running` | `executions.db` | +| 00:01:46 → 00:10:45 | the same job's previous (00:01) slot — **completed in 9 m** | `executions.db` | +| 14:33:35 | `Received SIGTERM as a planned gateway stop`, `under_systemd=yes`, `parent_name=systemd`, `loadavg_1m=2.15` | `gateway.log` | +| 14:33:37 | `Thread 34795 not found` while sending the shutdown notice (same dead thread as the birthday/Uber jobs) | `gateway.log` | +| 14:33:37 | incident row `ea79a2_c3ab2d784ba3` opens — *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/724a69108b…json`"* (`alerted`) | `cron_incidents` | +| 14:33:38 | `Shutdown drain: 1 in-flight cron job(s) … drain took 1.35s, timed_out=True` → `interrupting remaining work` | `gateway.log` | +| 14:34:25 | `1 authority task(s) did not settle within 40s; left for recovery`; `Profile /home/hermes/.hermes still has live writers at shutdown; its admissions stay for recovery` | `gateway.log` | +| 14:34:27 | `ERROR gateway.session_authority: Admitted turn 28ad127ffb2a44c0bcd17283a93aa496 failed`; cron session `local-253a96…` **paused: runtime_draining** | `gateway.log` | +| 14:34:27 | that session finalises with `end_reason: cron_complete` (66 msgs, 39 tool calls, 127 API calls) | `state.db` | +| 14:34:29 | `Gateway stopped (total teardown 54.06s)`; **`Skipping .clean_shutdown marker — drain timed out with interrupted agents`** | `gateway.log` | +| 14:35:18 | next gateway life starts (`Starting Hermes Gateway...`) | `gateway.log` | +| 14:35:59 | execution closes **`failed`**: `RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership` | `executions.db` | +| 14:35:59 | incident row `ea79a2_a4661554a3c2` opens on that RuntimeError (`alerted`); execution `c5ac27e3…` **delivered** the failure | `cron_incidents`, `deliveries.db` | +| 14:46:12 / 15:01:15 | Mnemosyne Watchdog `96eb95ce12f2` fails twice (`Script exited with code 1`) reacting to the log `ERROR` — rows `96eb95_0c89bd373a0a`, `96eb95_e32174134546` | `cron_incidents` | +| 15:16:12 | both watchdog rows **`resolved`** (the error aged out of its 15-min log window); watchdog back to `ok` | `cron_incidents` | + +## What is new or unexplained + +1. **A planned stop is enough to orphan a run.** No crash was required. The drain had a **30 s** budget and + reported `timed_out=True` after **1.35 s** with `cron_now=0` — i.e. the drain did **not** wait out its + budget before taking the interrupt path. Logged verbatim; **not diagnosed**. +2. **New terminal error string:** `lost its durable fire claim ownership`. The 10-20 orphan closed as + `unknown` with a scheduler-side note; this one closed as `failed` from the worker's own perspective. + Any alerting or triage that pattern-matches on the 10-20 wording will miss this one. +3. **Two rows for one fault.** `ea79a2_a4661554a3c2` (fire claim) and `ea79a2_c3ab2d784ba3` (unverified / + reconcile admission) both remain **`alerted`, unclosed** as of 2026-10-27 00:1x, and the job carries + **`failure_streak: 2`** from a **single** interrupted fire. +4. **The admission was reconciled.** `~/.hermes/cron/admissions/` no longer holds `724a69108b…json` + (only the 00:00 / 00:01 files from the following night), so the "reconcile" step in the row's text has + been satisfied — the rows were simply never closed. +5. **The watchdog works.** This fault **did** alert, twice, and self-resolved — which is a genuine + improvement over 10-20 (one wakes up via the incident ledger, not via a missing digest). + +## Impact + +- **One 12-hourly memory consolidation cycle did not run.** `ea79a2107e36` is the Mnemosyne + health/consolidation job → [[systems/mnemosyne]]. The 00:01 slot completed normally; the 12:01 slot + produced no completed work. +- **Day tally:** 2026-10-26 settled **169 `completed` / 3 `failed`** — and **all three failures are this one + fault** (1 Health Monitor + 2 derivative Watchdog reds). 2026-10-25 was **172 / 2**. +- **Session cost sunk:** the interrupted run burned 66 messages / 39 tool calls / 127 API calls + (~597 k input, 254 k output tokens) before the interrupt. +- **Unclosed incident rows 2 → 3** (`0e753e_e7f8194a7d06` carried over from 2026-10-01, plus the two new + `ea79a2_*` rows). + +## Recommended follow-ups (not yet done) + +1. Decide the drain policy for **long-running cron jobs**: this job legitimately occupies 9 m → 2 h+. + Either exempt a cron execution from the drain deadline, or interrupt it with a durable + *"interrupted → re-fire this slot"* marker instead of a terminal `RuntimeError`. +2. **Close or ack the two `ea79a2_*` rows** — the admitted turn failed and the admission is gone, so they + are historical. +3. **Verify the drain-timeout arithmetic** (`timed_out=True` at 1.35 s against a 30 s budget) — if the + deadline is being bypassed whenever a writer is live, every future restart can interrupt work. +4. Confirm `failure_streak` should count the *unverified* row at all — a single interrupted fire reading + **streak 2** inflates the job's apparent health. + +## Related + +- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — first instance of this class +- [[systems/scheduled-tasks]] — fleet ledger and per-window tallies +- [[systems/mnemosyne]] — the interrupted job's purpose and settings +- [[systems/hermes-agent]] — gateway lifecycle and restart history diff --git a/index.md b/index.md index 7d4dbc5..db1c56d 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,6 @@ # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-26 (stocktake — 8 stale pages corrected, 0 stale remaining) +> model routing and automation. Last updated: 2026-10-27 (nightly audit — 1 new incident page, 4 pages updated) > > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. @@ -35,10 +35,11 @@ - [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.69.2` (HEAD `942ddb8`, 2026-10-24 15:35 — *"Pulse: list candidates, decide in code, journal them (#342)"*; 24 commits on since the `0.64.0` baseline, **0 behind `origin/main`**)**, upstream `kaishi00/hermes-conduit-notifier`. **Gateway on `:9119` restarted 2026-10-25 18:42:33 — so the upstream commits are now LIVE** (the 10-24 "not live" note is superseded). One registered device (iPhone app `2.0.0`). **Still carries the uncommitted in-tree `dashboard/plugin_api.py` patch**; `conduit-calls.json → history` is **empty**, so the memory fix still has no positive call evidence, but `conduit_show` ran once on 10-25 14:37 (card delivered). ⚠ **Known failure: three 10-24 requests to be phoned were refused by the agent, which never called the tool; no recurrence since** → [[incidents/2026-10-24-conduit-call-request-refused]]. - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26** (peer offline **36 d**, `tx 312 / rx 0`; host answers neither `:8065` nor ICMP). **268** connect timeouts on 10-25 vs 264 on 10-24; the reconnect **attempt counter reset to 54** at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost. - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. -- [[systems/scheduled-tasks]] — **As of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** +- [[systems/scheduled-tasks]] — **As of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** - [[systems/skills-index]] — Installed skills inventory (**306 skills / 70 categories** as of the **2026-10-23 03:07** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at that run: **2 stale / 94 fresh / 6 no-date of 102 pages** (`freshness-20261023.txt`). Note: `SKILL.md` files on disk still exceed the index (the generator de-dups one nested duplicate; see [[systems/scheduled-tasks]]). ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents +- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **A *planned* gateway restart orphaned a live cron run — the second instance of the orphan class, and the first that alerted.** On 2026-10-26 at **14:33:35 AWST** the gateway took a planned `SIGTERM` (`under_systemd=yes`); the shutdown drain reported **`timed_out=True` after just 1.35 s** of its 30 s budget and interrupted `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) **2 h 34 m** in. It closed **`failed`** on a **new** string — `lost its durable fire claim ownership` — raised **two** `cron_incidents` rows (14:33:37 *unverified / reconcile admission* + 14:35:59 *fire claim*), **delivered** the failure at 14:35:59, and left the job on **`failure_streak: 2`** from one interrupted fire. Two Mnemosyne Watchdog reds followed at 14:46 and 15:01 and self-resolved at **15:16:12**. Net = **169 / 3** for 10-26. Unresolved: why a **planned** stop bypasses the drain deadline, and whether `failure_streak` should count the unverified row. → also updates [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] (the class now recurs without a crash). - [[incidents/2026-10-25-gmail-digest-runaway-and-refire]] — **The Gmail Action Digest ran 1 h 55 m and fired six times on 2026-10-25.** The 08:30 slot started 08:31:14 and closed **10:26:37 `failed`** on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed)` — against **91 s** (10-23) and **104 s** (10-24). The same job then fired five more times (17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53); the 19:18:41 fire died on a **new signature** — `Cron admission missing: the owner never admitted this fire` (distinct from the 10-20 orphan) — and all six fires, failures included, carry `delivery_outcome: delivered`, so up to **six digests reached Anthony**. Neither failure raised a `cron_incidents` row; the job now reads `last_status: delivery_queued` with an **empty** `last_error`. Hypothesis (unconfirmed): the 08:31 fire **adopted an unrelated long-lived session** — its worker made **70 `terminal` calls on VM101 `/dev/sdb` backup work** with context growing 66 k → 148 k tokens, while **context compression was unavailable all day** (`invalid_grant`). → [[systems/scheduled-tasks]], [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] - [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] — **`Invalid refresh token` — Nous Portal runtime credentials are dead, so context summarisation failed.** Gateway-log counts: **1** line 2026-10-22, **3** on 10-24, **20** on 10-25 (last 14:53:04, `agent.context_compressor: Failed to generate context summary`), **0** since the 19:27:13 restart. ⚠ **Masked, not fixed:** at **2026-10-25 19:13** all **eight** `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) moved off `nous` to **`no-think/freeapi/auto` on `custom:omni-101`** (fallback `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, `free_only: true`; compression timeout 90 s, vision 180 s). No `hermes model` re-authentication has been run and **`nous` is still the first entry in `fallback_providers`**, so failover tries a dead credential. → [[systems/model-providers]], [[systems/current-profile]] - [[incidents/2026-10-24-conduit-call-request-refused]] — **The agent refused to phone Anthony, three times, on 2026-10-24 (12:53–12:56).** He asked to be called to test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`); all three replies asserted the agent has **no calling capability**, with **zero tool calls** in the session. `conduit_call_user` (toolset `conduit`) exists, is registered for every session, is reachable via `tool_search`, and the plugin ships the `calling-the-user` skill; his call settings (`enabled` / `when_asked` / `decides`) were permissive and the device was paired. **Root cause: capability asserted instead of discovered** — the first reply came from `kilo-auto/free`, but the switch to `smart-route` repeated it. Same audit also **corrects the 10-22 incident's closing evidence**: `conduit-calls.json → ends` records turn-ends that fire a watch, not calls, so “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne **cron** sessions — `history` (real calls) is **empty** and no call has been placed since the memory fix. → [[systems/conduit]], [[incidents/2026-10-22-conduit-memory-start-timeout]] diff --git a/log.md b/log.md index cb14e2e..f052627 100644 --- a/log.md +++ b/log.md @@ -1408,3 +1408,49 @@ The two top-level **stale** files are `current-state.md` and `reports/communities-wa-social-media-audit-2026.md` — both dated historical records rather than drift. The vault repo is **local-only** (no remotes); it was committed, not pushed. + +## [2026-10-27] audit | Nightly wiki audit — window 2026-10-26 00:00 → 2026-10-27 00:5x AWST: a planned restart orphaned a live cron run + +- **A *planned* gateway restart orphaned a cron run — the second instance of the 2026-10-20 orphan class, and + this time it was loud.** `gateway.log`: `Received SIGTERM as a planned gateway stop` at **14:33:35** + (`under_systemd=yes`, `parent_name=systemd`, `loadavg_1m=2.15`). The drain then logged + `Shutdown drain: 1 in-flight cron job(s) … drain took 1.35s, timed_out=True` → `interrupting remaining work`, + and at 14:34:29 `Skipping .clean_shutdown marker — drain timed out with interrupted agents`. The victim was + **`ea79a2107e36` (Mnemosyne Health Monitor)**, claimed **12:01:13** — **2 h 34 m** in. It closed **14:35:59 + `failed`** on a string this wiki had not seen: **`RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its + durable fire claim ownership`**. Unlike the 10-20 orphan it raised **two `cron_incidents` rows** (14:33:37 + `ea79a2_c3ab2d784ba3` *"Cron execution unverified; reconcile `admissions/724a69108b…json`"*; 14:35:59 + `ea79a2_a4661554a3c2`, the RuntimeError) and **delivered** the failure. → new + `incidents/2026-10-26-planned-restart-orphaned-cron-run.md` +- **The Mnemosyne Watchdog did its job.** `96eb95ce12f2` red-flagged twice (14:46:12, 15:01:15, + `Script exited with code 1`) reacting to the same log `ERROR`, then **self-resolved at 15:16:12** when the + line aged out of its 15-minute window. Both rows (`96eb95_0c89bd373a0a`, `96eb95_e32174134546`) are closed. +- **Window tally and fleet:** 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are + this one interruption** (Health Monitor + the two watchdog reds); 2026-10-25 settled **172 / 2**. Fleet + unchanged: **28 jobs, 25 enabled / 3 disabled**. **Non-clean `last_status` 5 → 4** (`c0a9fa314542` and + `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as `error`, streak 2). **Unclosed incident rows 2 → 3**. + **Pins unchanged.** One gateway start (14:35:18, teardown 54.06 s). **17 sessions in window** (15 `cron`, + 1 `ios`, 1 `telegram`). +- **One incident row closed — the 2026-10-16 503 burst is fully shut.** `c0a9fa_764558b5ae83` **auto-closed + 09:13:00** when the radar's 09:00 slot ran clean; no execution row carries the resource-pressure string on + 10-25 or 10-26. Status stays `monitoring` (the single-node exposure is unchanged). Updated + `incidents/2026-10-16-omniroute-503-resource-pressure.md`. +- **⚠ Two quiet degradations: a completed run that delivered nothing useful, and a broken delivery thread.** + (a) The radar's 07:00 slot closed **`completed` at 09:10**, but its delivered text read **“RUN INCOMPLETE — + tool-call budget exhausted”** — no Notion writes, no cover letters — so the day's backlog was executed by hand. + (b) `fad5a86979d4`'s 03:08 slot completed but ended `delivery_failed`: **`Thread 34795 not found`** — the same + dead thread the gateway's own 14:33:37 shutdown notice hit. → `runbooks/job-radar.md` +- **The radar's cover-letter pipeline changed the same morning.** The PDF is now printed from the **canonical + rendered HTML** via Chromium (`build_cover_letter_pdf_from_html.py`, created 10:37) instead of ReportLab, so a + download matches the on-screen letter; the generation default moved to **`deepseek-paid-smart`** (10:47) and + `job_dashboard_server.py`'s regenerate timeout went **120 s → 600 s** (10:48, server restarted). + → `runbooks/job-radar.md` +- **Files:** new `incidents/2026-10-26-planned-restart-orphaned-cron-run.md`; updated + `incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md`, + `incidents/2026-10-16-omniroute-503-resource-pressure.md`, `runbooks/job-radar.md`, + `systems/scheduled-tasks.md`, `current-state.md`, `index.md`, `log.md`. **Nothing removed or archived.** + No secrets written. +- **Unresolved:** why a *planned* stop bypasses the drain deadline (`timed_out=True` at 1.35 s of a 30 s budget); + whether `failure_streak` should count the *unverified* row (one interruption currently reads streak 2); and the + **`Thread 34795`** delivery target needs re-pointing, or the Monday 03:08 digest will keep falling back to the + unthreaded chat. diff --git a/runbooks/job-radar.md b/runbooks/job-radar.md index 4b2bee1..e6c3637 100644 --- a/runbooks/job-radar.md +++ b/runbooks/job-radar.md @@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery type: runbook status: active created: 2026-07-27 -updated: 2026-10-13 -review_after: 2026-10-25 -verified_on: 2026-10-13 +updated: 2026-10-27 +review_after: 2026-11-10 +verified_on: 2026-10-27 version_applies_to: Hermes Agent cron dca8482e4f76 confidence: high tags: [automation, cron, jobs, notion, dashboard, runbook] @@ -262,6 +262,36 @@ hermes cron run dca8482e4f76 # reproduce and capture the 403 Do **not** allow a silent fallback — the radar must surface a model failure rather than emit an empty "no jobs found" digest. +## Cover-letter PDF pipeline — printed from the dashboard HTML (changed 2026-10-26) + +The PDF is now produced from the **canonical rendered HTML**, so a download matches the letter on screen. +Previously ReportLab (`build_cover_letter_pdf.py`) laid the letter out independently of the template and +produced a visibly different document — Anthony rejected exactly that mismatch. + +- **Renderer:** `~/.hermes/scripts/build_cover_letter_pdf_from_html.py` (created 2026-10-26 10:37) drives + **Chromium `--print-to-pdf`** over the canonical HTML and writes the PDF into + `~/.hermes/radar-ref/drafts/`. **Do not go back to ReportLab.** +- **Model:** the cover-letter generation default moved from the `cover-letter-writer` combo to + **`deepseek-paid-smart`** on 2026-10-26 10:47 (`regenerate_cover_letter.py`, env override `CL_MODEL`). +- **Dashboard timeout:** `job_dashboard_server.py` must allow **`timeout=600`** on the regenerate + subprocess (set 2026-10-26 10:48, up from 120 s). One run is an LLM call + DOCX assembly + a Chromium + print; it can exceed two minutes. The dashboard server was restarted to pick the change up. +- All five artefacts (`.txt .json .html .docx .pdf`) must still exist and be nonempty, and the HTML must + contain no unresolved `{{...}}` tokens — the HTML is now the source of truth the PDF follows. + +### 2026-10-26 07:00 slot — `RUN INCOMPLETE`, and why the backlog was finished by hand + +The 07:00 run started on schedule and closed **`completed` at 09:10**, but the report it delivered read +**“RUN INCOMPLETE — tool-call budget exhausted”**: discovery and scoring consumed the run's tool budget, so +the **Notion writes and the entire cover-letter stage never executed**. The same-day backlog was then run +interactively (rows written for the outstanding strong matches, cover letters generated and verified). + +- **A `completed` execution row is not a completed radar run.** Read the delivered text for the + `RUN INCOMPLETE` marker before treating a slot as done. +- Budget the browser-heavy stages (one tab in flight, close before opening the next — see the + 2026-10-12 hazard below) and keep headroom for the Notion + cover-letter writes, which sit last in the + pipeline. + ## Browser-layer hazard inside the run (2026-10-12) This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index f469a74..f2cb2d4 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,9 +3,9 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-26 -verified_on: 2026-10-26 -review_after: 2026-10-31 +updated: 2026-10-27 +verified_on: 2026-10-27 +review_after: 2026-11-03 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db] @@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count -- **Current as of 2026-10-26 00:1x AWST (live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`). +- **Current as of 2026-10-27 00:1x AWST (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`). +- **Current as of 2026-10-26 00:1x AWST** *(superseded by the 2026-10-27 entry above)* **(live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`). - **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page. *(Superseded by the 2026-10-26 entry above.)* - **Current as of 2026-10-24 00:2x AWST *(superseded by the 2026-10-25 entry above)* (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]] - **Current as of 2026-10-23 00:1x AWST *(superseded by the 2026-10-24 entry above)* (`jobs.json.updated_at 2026-10-23T00:07:25+08:00`) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. **A DB recount by `started_at` and by `claimed_at` puts 2026-10-21 at 168 / 1**, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Five incidents closed in-window and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (`0e519874d280`, closed 08:01:40), `06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so **unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`: `1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, `c0a9fa_764558b5ae83`, `0e753e_e7f8194a7d06`). **Non-clean `last_status` falls 9 → 5:** `1b23c042e17d` (`error`, streak 2 — **next slot Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` (`delivery_failed` — Mon 10-26 03:00), `066bb5191280` (`delivery_failed` — 11-01). All four cleared rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its first slot of 10-22. **Pins:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (`provider`/`model` null → global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). ⚠ **Stocktake anomaly:** `1a8e90ea2271` ran **03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, finished `completed` with delivery `suppressed` — and **committed nothing**. It left its regenerated `systems/skills-index.md` uncommitted (mtime 10-22 03:00, with a `## .Archive (41)` section reading **347**) and its diagnosed fixes unapplied → [[incidents/2026-08-15-skills-index-drift]]. **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop` (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models `mimo-v2.5`, `or-inclusionai_ling-2_6-1t`, `minimax/minimax-m3.1-flash`), 2 `acp` (0 messages), 1 `oneshot` memory-capture probe. → [[systems/skills-index]], [[incidents/2026-08-15-skills-index-drift]]