diff --git a/current-state.md b/current-state.md index 2bd52ff..e372a47 100644 --- a/current-state.md +++ b/current-state.md @@ -3,8 +3,8 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-30 -verified_on: 2026-10-30 +updated: 2026-10-31 +verified_on: 2026-10-31 review_after: 2026-11-07 confidence: high tags: [system] @@ -43,7 +43,21 @@ Updated: 2026-10-30 **10 deliveries, all 10 `delivered`, 0 failures**. Non-clean `last_status` **5 → 3**; unclosed incident rows **4 → 1** (the two radar rows closed `resolved` at 10:16:12). Fleet unchanged at **29 jobs (26 enabled / 3 disabled)** at that audit. -- **Cron fleet changed shape overnight on 2026-10-30 — the Mnemosyne Health Monitor is paused and its failure +- **Cron: 2026-10-30 closed 138 / 0 — after a 5 h 30 m HOST OUTAGE — and the "auto-paused" health monitor was never + dead, it was running.** Fleet **29 jobs (26 enabled / 3 disabled)** + (`jobs.json.updated_at 2026-10-31T00:29:59+08:00`), re-verified 2026-10-31 00:3x. **Zero execution rows exist + between 10:58:20 and 16:37:21** — PVE `pve` rebooted (boot **2026-10-30 16:24:00**), CT460 came back **16:28:00**, + the old gateway life (`pid=3913841`) was logged **UNCLEANLY** at **16:35:50** (*"no exit path ran — SIGKILL / OOM / + VM death"* — a candidate list, not a diagnosis; root cause **not established**, no OOM evidence), and 8 catch-up + runs fired at **16:37:21–16:38:22** → [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]. Against 10-29's + **177 / 0**, that is **≈39 lost slots and no failures**. **`ea79a2107e36` (Mnemosyne Health Monitor) is ENABLED + and clean again** — `last_status: ok`, streak **0**, incident `ea79a2_d93c3acdc4f2` closed **`resolved` + 2026-10-30T10:02:48.662** when its 00:01:08 slot finished after **10 h 01 m 40 s**; its **12:01 slot was lost to + the outage** and is the still-`running` 16:37:21 catch-up. **Non-clean `last_status` 4 → 3**, **failing 1 → 0**, + **unclosed incident rows 2 → 1** (survivor: `0e753e_e7f8194a7d06`, due 11-01). **`deliveries.db` finished 10-30 + with 8 rows, all `delivered`** (latest 18:46:04) — the 03:3x "zero rows / notice never delivered" reading is + superseded. +- **⚠ Superseded by the 2026-10-31 entry above — read that first.** Cron fleet changed shape overnight on 2026-10-30 — the Mnemosyne Health Monitor is paused and its failure notice was never delivered.** `ea79a2107e36`'s 00:01:08 slot closed **`unknown`** at 01:04:22 on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*), opening incident `ea79a2_d93c3acdc4f2` (`detected`), and the diff --git a/incidents/2026-10-30-ct460-host-outage-scheduler-gap.md b/incidents/2026-10-30-ct460-host-outage-scheduler-gap.md new file mode 100644 index 0000000..26f7ba6 --- /dev/null +++ b/incidents/2026-10-30-ct460-host-outage-scheduler-gap.md @@ -0,0 +1,103 @@ +--- +title: CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30) +type: incident +status: resolved +created: 2026-10-31 +updated: 2026-10-31 +verified_on: 2026-10-31 +confidence: high +tags: [cron, gateway, host, outage, pve, ct460, incident] +sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-agent-log, live-deliveries-db, ssh-pve-uptime] +--- + +# CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30) + +**Status: RESOLVED** — the container and scheduler came back on 2026-10-30 ~16:28–16:37 and the day +closed clean (138 executions, 136 `completed`, **0 `failed`**). **Root cause of the host reboot is not +established.** No monitor alerted; the gap was found by this audit. + +## Summary + +The Hermes runtime went dark at **~10:58 AWST on 2026-10-30** and did not return until +**16:28** (container) / **16:37:21** (first execution). The PVE host `pve` itself rebooted — its boot +time is **2026-10-30 16:24:00** — so this was a **host-level outage, not a cron or gateway fault**. +The gateway's previous life died with no exit path, which the lifecycle ledger logs as +`SIGKILL / OOM / VM death`. Nothing noticed for **5 h 37 m**. + +## Timeline (all times AWST, 2026-10-30) + +| Time | Event | Evidence | +|---|---|---| +| 10:49:54 | Last execution before the gap starts | `executions.db` last row `< 16:37` | +| ~10:58:20 | Gateway life `pid=3913841` dies; no exit path ran | `gateway-exit-diag.log` `gateway.previous_unclean_exit` (prior life started 2026-10-28T17:21:58Z) | +| 10:58:20 → 16:37:21 | **Zero execution rows** (5 h 39 m) | `executions.db` count in window = **0** | +| 16:24:00 | PVE host `pve` (`100.96.100.82`) boots | `ssh root@100.96.100.82 'uptime -s; who -b'` → both read `2026-10-30 16:24` | +| 16:28:00 | CT460 container boots | local `uptime -s` → `2026-10-30 16:28:00` | +| 16:35:50 | Unclean exit noticed and logged | `logs/agent.log`, `logs/errors.log` — `Previous gateway life (pid=3913841, started_at=2026-10-28T17:21:58Z) exited UNCLEANLY (no exit path ran — SIGKILL / OOM / VM death)` | +| **16:37:21** | Scheduler resumes — **8-run catch-up burst** (16:37:21 → 16:38:22) | `executions.db` | +| 23:59:58 | Daily wiki audit (this run) starts; the 16:37 health-monitor catch-up is still `running` | `executions.db` | + +## The resume burst (16:37:21 → 16:38:22) + +Eight jobs fired within 61 seconds of the scheduler coming back. Seven closed `completed`; +the eighth is still running. + +| Job | Started | Closed | +|---|---|---| +| FreshRSS Midday News Briefing (`ecc1fc13655a`) | 16:37:21 | 17:45:31 — the 12:00 slot, delivered late | +| Mnemosyne Auto-Consolidate | 16:37:21 | 16:37:54 | +| Mnemosyne Embedding Coverage Monitor | 16:37:21 | 16:41:31 | +| Mnemosyne Watchdog | 16:37:21 | 16:38:33 | +| **Mnemosyne Health Monitor (`ea79a2107e36`)** | **16:37:21** | **RUNNING at audit** — the late 12:01 slot | +| Job Tracker Dashboard Refresh | 16:37:23 | 16:37:35 | +| 🦝 Hermes Config Backup to Gitea (Daily) | 16:38:06 | 16:42:02 | +| Daily Marketing Job Radar - Write and Letters | 16:38:22 | 16:41:39 | + +## Impact + +- **≈39 scheduled runs lost.** 2026-10-29 fired **177** executions (177 `completed`, 0 `failed`); 2026-10-30 + fired **138** (136 `completed`, 0 `failed`, 2 `running` — the 16:37 catch-up plus this audit). No row exists + inside the dark window, so the deficit is unfired slots, not failures. +- **No failures were caused.** Nothing was recorded `failed` on 2026-10-30 — the outage simply removed the + opportunities to run. +- **The two long Mnemosyne runs absorbed it.** The FreshRSS midday briefing regenerated on the late fire + (16:37:21 → 17:45:31), and the health monitor's 12:01 slot is the still-running 16:37:21 fire — normal for + this job, whose 00:01 slot took **10 h 01 m 40 s** the same day. +- **Deliveries were unaffected once back up:** `deliveries.db` holds **8 rows for 2026-10-30, all `delivered`** + (latest 18:46:04) — the opposite of the 03:3x snapshot, which read zero rows because the day had barely started. +- **2026-10-31 opened clean:** 5 executions (00:01:05 ×3, 00:15:59, 00:23:57), all `completed`. + +## Root cause — not established + +- The host rebooted. `pve` boot time is 16:24:00; **no OOM evidence was found** on the CT460 side (local + `dmesg` grep empty, `/var/log/wtmp` absent, no `suspected_oom` marker in the webui session store). +- The ledger's own wording (`SIGKILL / OOM / VM death`) is a **list of candidates for "no exit path ran"**, not a + finding. Treating it as a diagnosis would be wrong. +- Same-day, different host: [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] records **m93p** + reboots (~15:55) that took down NFS-backed containers. The two events are **separate hosts and separate + causes as far as the evidence goes** — they are not merged here, but the timing is close enough to warrant + checking the two hypervisors' last-boot records together before either is called a one-off. +- **To close it:** PVE syslog / `journalctl -b -1` on `100.96.100.82` around 2026-10-30 10:58, and the host's + own last-boot records (`last -x reboot`). Neither was read for this page — say so rather than guess. + +## Why nothing alerted + +The jobs positioned to notice a host-down event are **disabled**: + +| Job | ID | State | +|---|---|---| +| NFS mount monitor | `cbfe434a3b5e` | disabled | +| m93p LXC boot monitor | `1a8e90ea2271` | disabled | + +Everything else is *inside* the scheduler that went dark, so it could not have noticed. **A host-level outage of +the Hermes VM currently has no in-estate detector** — the fleet learned about it from the 23:59 audit, 13 h +after the fact. This is the incident's one open item. + +## Related + +- [[systems/scheduled-tasks]] — the fleet entry for this window +- [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — the same day's health-monitor story: its 00:01 slot + closed `unknown` at 01:04:22, incident `ea79a2_d93c3acdc4f2` **auto-closed `resolved` at 10:02:48** when that + run finally completed, and the job is live again (`enabled: true`, `last_status: ok`, streak 0) +- [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] — sister-host reboot, same day +- [[infrastructure/hosts]] — `pve` / CT460 addressing diff --git a/incidents/2026-10-30-mnemosyne-health-monitor-paused.md b/incidents/2026-10-30-mnemosyne-health-monitor-paused.md index de3f221..f1b9b4c 100644 --- a/incidents/2026-10-30-mnemosyne-health-monitor-paused.md +++ b/incidents/2026-10-30-mnemosyne-health-monitor-paused.md @@ -1,10 +1,10 @@ --- title: Mnemosyne Health Monitor auto-paused after gateway-connection failure type: incident -status: active +status: resolved created: 2026-10-30 -updated: 2026-10-30 -verified_on: 2026-10-30 +updated: 2026-10-31 +verified_on: 2026-10-31 confidence: high tags: [cron, mnemosyne, gateway, incident, delivery] sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents] @@ -12,7 +12,17 @@ sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-inci # Mnemosyne Health Monitor auto-paused after gateway-connection failure (2026-10-30) -**Status: ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram. +**Status: RESOLVED — 2026-10-30T10:02:48.662+08:00** (updated by the 2026-10-31 audit; everything below is the +2026-10-30 03:3x snapshot and its "job is disabled / notice never delivered" reading did not survive the day). +**What actually happened:** the 00:01:08 slot was **still running**, not dead — it closed **`completed` at +10:02:48** after **10 h 01 m 40 s** and **delivered**; incident `ea79a2_d93c3acdc4f2` **auto-closed `resolved` +at 2026-10-30T10:02:48.662+08:00**, and the job is live again (`enabled: true`, `last_status: ok`, +`failure_streak: 0`). `deliveries.db` finished 2026-10-30 with **8 rows, all `delivered`** (latest 18:46:04) — the +zero-row reading was an artefact of the day having barely begun. ⚠ Its **12:01 slot was then lost to the CT460 +host outage** and fired late in the 16:37:21 catch-up burst, still `running` at the 2026-10-31 audit — see +[[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]. + +**Historical snapshot (2026-10-30 03:3x): ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram. ## Summary diff --git a/index.md b/index.md index d10574e..015501c 100644 --- a/index.md +++ b/index.md @@ -3,15 +3,15 @@ title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 -updated: 2026-10-30 -verified_on: 2026-10-30 +updated: 2026-10-31 +verified_on: 2026-10-31 confidence: high tags: [index, infrastructure, systems] --- # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-30 (nightly audit — Conduit voice calls repaired and documented; the Job Radar's two-stage split completed its first clean end-to-end cycle, resolving both radar incidents; 2026-10-29 was cron's first 176/0 day since 10-24) +> model routing and automation. Last updated: 2026-10-31 (nightly audit — CT460's 5 h 30 m host outage is documented (PVE `pve` reboot 2026-10-30, ≈39 cron slots lost, 0 failures, 8 catch-up runs at 16:37:21), and the "auto-paused" Mnemosyne Health Monitor turned out to have been *running*; it closed `completed` at 10:02:48, its incident auto-closed, and the job is enabled again; 2026-10-30 closed 138 / 0) > > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. @@ -46,11 +46,12 @@ tags: [index, infrastructure, systems] - [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.69.2` (HEAD `942ddb8`, 2026-10-24 15:35 — *"Pulse: list candidates, decide in code, journal them (#342)"*; 24 commits on since the `0.64.0` baseline, **0 behind `origin/main`**)**, upstream `kaishi00/hermes-conduit-notifier`. **Gateway on `:9119` restarted 2026-10-25 18:42:33 — so the upstream commits are now LIVE** (the 10-24 "not live" note is superseded). One registered device (iPhone app **2.2.0**). ✅ **The first two real calls in the product's history are now on record — 2026-10-29 08:38:04 and 08:49:36** — after three faults in series were repaired (truncated speech-server URL → `…/v1/realtime`; 12 h dashboard session TTL → 30 d; dead `groq` STT → `local`, dead `openai` TTS → `ollama_local`) → [[incidents/2026-10-29-conduit-voice-silence]]. **So the memory fix is no longer evidence-free.** Still carries the uncommitted in-tree `dashboard/plugin_api.py` patch. The 10-24 refusal fault has not recurred → [[incidents/2026-10-24-conduit-call-request-refused]]. - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26** (peer offline **36 d**, `tx 312 / rx 0`; host answers neither `:8065` nor ICMP). **268** connect timeouts on 10-25 vs 264 on 10-24; the reconnect **attempt counter reset to 54** at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost. - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. -- [[systems/scheduled-tasks]] — **As of 2026-10-30 03:3x: 29 jobs (25 enabled, 4 disabled); non-clean `last_status` 3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries all `delivered`. ⚠ `ea79a2107e36` (Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closed `unknown` on a gateway-connection failure — and its failure notice was NEVER DELIVERED (`deliveries.db` has zero rows for 2026-10-30), so the incident row is the only trace → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. Every other previously-red job has cleared to `ok`/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 5 → 3 (all three are stale re-runs of old faults and every `failure_streak` read 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all `delivered`. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with `missing_cover_letter_count 0` (Notion 455 → 475). Both stage-1 incident rows closed `resolved` at 10:16:12; the lone survivor is the Facebook monthly's `0e753e_e7f8194a7d06`.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** +- [[systems/scheduled-tasks]] — **As of 2026-10-31 00:3x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 4 → 3; failing 1 → 0; unclosed incident rows 2 → 1; 2026-10-30 closed 138 / 0 after a 5 h 30 m CT460 host outage (PVE reboot, ≈39 slots lost, 8 catch-up runs at 16:37:21), and `ea79a2107e36` is ENABLED again (`last_status: ok`, streak 0) after its 00:01:08 slot closed `completed` at 10:02:48; deliveries 8/8 `delivered`.** **Prior window — as of 2026-10-30 03:3x:** 29 jobs (25 enabled, 4 disabled); non-clean `last_status` 3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries all `delivered`. ⚠ `ea79a2107e36` (Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closed `unknown` on a gateway-connection failure — and its failure notice was NEVER DELIVERED (`deliveries.db` has zero rows for 2026-10-30), so the incident row is the only trace → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. Every other previously-red job has cleared to `ok`/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 5 → 3 (all three are stale re-runs of old faults and every `failure_streak` read 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all `delivered`. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with `missing_cover_letter_count 0` (Notion 455 → 475). Both stage-1 incident rows closed `resolved` at 10:16:12; the lone survivor is the Facebook monthly's `0e753e_e7f8194a7d06`.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** - [[systems/skills-index]] — Installed skills inventory (**302 skills** (69 categories) as of the **2026-10-30** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at the 2026-10-30 run: **0 stale / 109 fresh / 3 no-date of 112 pages** (`freshness-20261030.txt`). Note: `SKILL.md` files on disk still exceed the index by one (303 vs 302) — the generator de-dups one nested duplicate, so **302 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents -- [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — **The Mnemosyne Health Monitor cron job was auto-paused overnight and its failure notice never reached Telegram — the only trace is the incident row.** `ea79a2107e36`'s 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, opening incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, `detected`) — and the scheduler **paused the job at that moment**. Fleet is now **29 jobs (25 enabled / 4 disabled)**; live `last_status: error`, `failure_streak: 1`. ⚠ **Two traps in one fault:** the failure notice itself failed to send (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed` — `deliveries.db` has **zero rows for 2026-10-30**), and the run closed `unknown` rather than `failed`. The job was **not re-enabled** and the gateway was **not restarted** by the stocktake — root cause is **unverified**. → [[systems/scheduled-tasks]], [[systems/mnemosyne]] +- [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]] — **RESOLVED (root cause not established).** CT460 went dark for **5 h 30 m on 2026-10-30**: the PVE host `pve` rebooted (boot **16:24:00**), CT460 returned **16:28:00**, the previous gateway life (`pid=3913841`) was logged as exited **UNCLEANLY** at **16:35:50** (*"no exit path ran — SIGKILL / OOM / VM death"*), and the scheduler resumed at **16:37:21** with **8 catch-up runs**. **Zero execution rows exist between 10:58:20 and 16:37:21**; the day still closed **138 / 0** against 10-29's 177 / 0 (≈39 slots lost, no failures). **No monitor alerted** — the two jobs that could have (`cbfe434a3b5e`, `1a8e90ea2271`) are disabled. +- [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — **RESOLVED 2026-10-30T10:02:48** (its 00:01:08 slot was still *running*, not dead: it closed `completed` after 10 h 01 m 40 s and delivered, auto-closing `ea79a2_d93c3acdc4f2`; `ea79a2107e36` is enabled again with `last_status: ok` / streak 0). **The Mnemosyne Health Monitor cron job was auto-paused overnight and its failure notice never reached Telegram — the only trace is the incident row.** `ea79a2107e36`'s 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, opening incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, `detected`) — and the scheduler **paused the job at that moment**. Fleet is now **29 jobs (25 enabled / 4 disabled)**; live `last_status: error`, `failure_streak: 1`. ⚠ **Two traps in one fault:** the failure notice itself failed to send (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed` — `deliveries.db` has **zero rows for 2026-10-30**), and the run closed `unknown` rather than `failed`. The job was **not re-enabled** and the gateway was **not restarted** by the stocktake — root cause is **unverified**. → [[systems/scheduled-tasks]], [[systems/mnemosyne]] - [[incidents/2026-10-29-conduit-voice-silence]] — **Every Conduit voice call connected and then died silently; three faults sat in series, and the first two real calls in the product's history were placed on 2026-10-29 after all three were fixed.** (A) `voice.speech_server.url` was the truncated `wss://…:8443/v1/` — the plugin dials it **verbatim** and the bare path answers **403**, reported as *"the speech server refused the key."* (B) `dashboard.basic_auth.session_ttl_seconds` was the **12 h default**, so an expired phone session gets **WS 4401** with no Hermes turn and no error on the device (the phone re-authenticated at 08:00:59, minutes *after* a silent call). (C) Both host providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq` → **403/Cloudflare 1010** (blocked, not mis-keyed). Fixed 08:02–08:4x: URL → `…/v1/realtime`, TTL → **2 592 000 s (30 d)**, `stt.provider: local` (faster-whisper `base`), `tts.provider: ollama_local`; dashboard restarted 08:03:26. Verified by driving the whole call path for real, then by calls at **08:38:04** and **08:49:36**. ⚠ Two TTL keys exist — `dashboard.basic_auth.session_ttl_seconds` (raised) vs `dashboard.session_ttl_seconds` (**still 43 200**). `status: resolved`. - [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] — **The radar's two-stage split ran for real on 2026-10-28 and landed nothing: stage 1 stopped at the agent iteration cap before writing its handoff, so both stage-2 slots correctly no-oped (silent). Two consecutive stage-1 failures, two different ceilings (tool-call budget 10-27, iteration cap 10-28); no Notion writes for a second day.** **RESOLVED 2026-10-29** — the next stage-1 run (`dca8482e4f76`, 07:00:59 → 10:16:12, 3 h 15 m 13 s) closed `completed` *and* wrote `handoff/2026-10-29.json`, which stage 2 consumed at 10:40:59 and delivered as **20 pages + 2 cover letters** (`missing_cover_letter_count 0`). `status: resolved`, review 2026-11-13. - [[incidents/2026-10-27-job-radar-budget-exhausted]] — **The Daily Marketing Job Radar exhausted its tool-call budget for the second day running — and this time the run is recorded as `failed`, not `completed`.** On 2026-10-27 the 07:00 slot ran **07:01:08 → 11:03:54** (4 h 2 m 46 s) and its delivered report opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"*: discovery/verify/dedup/classify finished, **Notion writes, cover letters and the final audit never ran** (Notion DB held 451 entries, 0 added for the day). Incident row `dca848_e93a538ade6b` opened 11:03:53 (`alerted` 11:03:59, `failure_type: auth` — wrong for a budget fault) and was still open at the 2026-10-28 audit. **Response the same day:** the radar was split into stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) and new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030), created 17:34:15, two `completed` test fires that evening → [[runbooks/job-radar]]. **RESOLVED 2026-10-29:** both of this incident's ledger rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12** on the first successful post-split run — `dca8482e4f76` restored delivery to `local`, wrote the handoff inside budget, and no cover letter went missing (`missing_cover_letter_count 0`). `status: resolved`, review 2026-11-13. diff --git a/infrastructure/hosts.md b/infrastructure/hosts.md index 5b29268..72843e5 100644 --- a/infrastructure/hosts.md +++ b/infrastructure/hosts.md @@ -3,8 +3,8 @@ title: Known Hosts and Paths type: infrastructure status: partial created: 2026-07-22 -updated: 2026-10-30 -verified_on: 2026-10-30 +updated: 2026-10-31 +verified_on: 2026-10-31 confidence: medium tags: [infrastructure, hosts, containers, paths] sources: [] @@ -35,7 +35,7 @@ sources: [] | `/home/hermes/.local/bin/hermes` | Hermes binary | | | `/home/hermes/.hermes/config.yaml` | Config | `_config_version: 51` | | `/home/hermes/.hermes/skills/` | Skills dir | 302 skills across 69 categories (`.archive/` excluded) | -| `/home/hermes/.hermes/cron/` | Cron jobs | 29 jobs — 25 enabled / 4 disabled (the 4th disabled job, `ea79a2107e36` Mnemosyne Health Monitor, was auto-paused 2026-10-30 01:04 after a gateway-connection failure) | +| `/home/hermes/.hermes/cron/` | Cron jobs | **29 jobs — 26 enabled / 3 disabled** (re-verified 2026-10-31 00:3x). ⚠ Superseded: the 2026-10-30 03:3x reading was 25 / 4 — the 4th disabled job, `ea79a2107e36` Mnemosyne Health Monitor, was auto-paused 2026-10-30 01:04 after a gateway-connection failure, but **that pause lasted only until 10:02:48**, when its long-running slot closed `completed`, incident `ea79a2_d93c3acdc4f2` auto-closed `resolved`, and the job returned to `enabled: true` / `last_status: ok` / streak 0. **CT460 itself was then dark for 5 h 30 m on 2026-10-30** (PVE `pve` rebooted; host boot 16:24:00, CT460 16:28:00, scheduler resumed 16:37:21 — ≈39 cron slots lost, 0 failures) → [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]] | | `/home/hermes/.hermes/mnemosyne/` | Mnemosyne DB | backups/, data/, logs/, models/ | | `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | cua-driver | v0.9.0 | | `~/.camofox/` | Camofox profiles | managed persistence | diff --git a/log.md b/log.md index 2c3bc7d..2d17700 100644 --- a/log.md +++ b/log.md @@ -1646,3 +1646,38 @@ not pushed. - **Not changed:** `systems/skills-index.md` count (302 is correct), CT460's Serve route table (matched live exactly), and the steel-removal records across the wiki (confirmed still true). - **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a key or token value. + +## [2026-10-31] audit | Nightly wiki audit — window 2026-10-30 00:00 → 2026-10-31 00:3x AWST + +**Verdict: one large undocumented event found and written up, and one *documented* incident found to have been +recorded wrong.** + +- **NEW PAGE — [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]].** CT460 was dark for **5 h 30 m** on + 2026-10-30 and **no monitor alerted**. Evidence: **zero `executions.db` rows between 10:58:20 and 16:37:21**; + PVE host `pve` boot time **2026-10-30 16:24:00**; CT460 boot **16:28:00**; the previous gateway life + (`pid=3913841`, started 2026-10-28T17:21:58Z) logged **UNCLEANLY at 16:35:50** (*"no exit path ran — + SIGKILL / OOM / VM death"*); scheduler resumed **16:37:21** with an **8-run catch-up burst** through 16:38:22. + The day still closed **138 executions / 0 failed** against 10-29's **177 / 0** — **≈39 lost slots, no failures**. + Root cause **not established**: no OOM evidence locally or on the PVE host, so the ledger string is read as a + candidate list, not a diagnosis. **Still open:** the two jobs that could have detected it (`cbfe434a3b5e` NFS + mount monitor, `1a8e90ea2271` m93p LXC boot monitor) are disabled. +- **CORRECTED — [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] was right about the record and wrong + about the run.** `ea79a2107e36`'s 2026-10-30 00:01:08 slot was **still running** at the 03:3x reading; it closed + **`completed` at 10:02:48** after **10 h 01 m 40 s** and **delivered**, auto-closing incident + `ea79a2_d93c3acdc4f2` **`resolved` at 2026-10-30T10:02:48.662+08:00**. Live job state is now `enabled: true`, + `last_status: ok`, `failure_streak: 0`. The page's *"zero `deliveries.db` rows / the notice was never + delivered"* claim was an artefact of reading the day at 03:3x: 2026-10-30 finished with **8 delivery rows, all + `delivered`** (latest 18:46:04). Page frontmatter moved `active` → **`resolved`**, with the old snapshot kept + below as explicitly historical. ⚠ Its **12:01 slot was lost to the host outage** and is the still-`running` + 16:37:21 catch-up. **Reconciliation lesson: a slot mid-flight is not a failure — check `finished_at`, not + `last_status`, before writing an incident.** +- **Reconciled — `systems/scheduled-tasks.md`.** Fleet **29 jobs (26 enabled / 3 disabled)** — the "4th disabled + job" reading is superseded (frontmatter date, fleet block, Total jobs, non-clean `last_status` **4 → 3**, + failing **1 → 0**, the health-monitor table row, the disabled table row, and a new **"Resolved 2026-10-30"** + subsection). Non-clean now **3 = 1 `error` + 2 `delivery_failed`**; unclosed incident rows **2 → 1** (survivor: + the Facebook monthly's `0e753e_e7f8194a7d06`, due 11-01). +- **Reconciled** — `current-state.md` (new leading cron entry + superseded marker), `infrastructure/hosts.md` + (cron row), `index.md` (frontmatter, banner, scheduled-tasks entry, both 10-30 incident entries), + `incidents/2026-10-30-mnemosyne-health-monitor-paused.md`. +- **Checked, unchanged:** 12 sessions active in-window (longest `local-40472e0b…`, 1,351 messages), no orphaned + `running` rows, no incidents opened or closed after 10-30, no obsolete pages to archive, no secrets written. diff --git a/systems/mnemosyne.md b/systems/mnemosyne.md index d0c3b6a..f5f321b 100644 --- a/systems/mnemosyne.md +++ b/systems/mnemosyne.md @@ -3,8 +3,8 @@ title: Mnemosyne type: system status: active created: 2026-07-22 -updated: 2026-10-24 -verified_on: 2026-10-24 +updated: 2026-10-31 +verified_on: 2026-10-31 review_after: 2026-11-07 confidence: high tags: [memory, mnemosyne, backend] @@ -94,7 +94,14 @@ Primary agent memory store for Hermes. ## Known Management Jobs - Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent - Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent -- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception +- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception. + ⚠ **2026-10-30: the monitor was twice flagged and neither flag survived contact with `finished_at`.** Its + 00:01:08 slot was auto-paused at 01:04:22 on an *"execution unverified / gateway connection failed"* admission + ([[incidents/2026-10-30-mnemosyne-health-monitor-paused]]) — but the slot was **still running** and closed + **`completed` at 10:02:48** after **10 h 01 m 40 s**, delivering normally and auto-closing incident + `ea79a2_d93c3acdc4f2`; the job is **enabled**, `last_status: ok`, streak 0. Its **12:01 slot was then lost to + the CT460 host outage** (5 h 30 m dark, [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]) and fired in + the 16:37:21 catch-up. **Check `finished_at` before believing a long-running slot failed.** - Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95% - Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent - ⚠ **Blast-radius note (2026-10-16).** Both monitors above (`ea79a2107e36` Health every 12 h, diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index ee188b5..d50ae2b 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,8 +3,8 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-30 -verified_on: 2026-10-30 +updated: 2026-10-31 +verified_on: 2026-10-31 review_after: 2026-11-06 confidence: high tags: [cron, scheduled, automation] @@ -14,7 +14,24 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count -- **Current as of 2026-10-30 03:3x AWST (live `jobs.json` + `executions.db`) — the 10-29 clean day held, but the fleet changed shape overnight: the Mnemosyne Health Monitor was auto-paused and its failure notice could not be delivered.** Fleet now **29 jobs (25 enabled / 4 disabled)** — `ea79a2107e36` (Mnemosyne Health Monitor) was **auto-paused at 2026-10-30 01:04:22** after its 00:01:08 slot closed **`unknown`** on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, which opened incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`). Non-clean `last_status` is now **4** — the three long-standing survivors (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`) **plus `ea79a2107e36` (`error`, `failure_streak: 1`)** — so the standing "every `failure_streak` reads 0" claim is now false. Window **2026-10-29 settled 177 `completed` / 0 `failed`** (the page previously said 176 — the 23:59 Wiki Audit row `cbfe434a3b5e` had still been `running` at audit time and has since settled `completed`); **2026-10-28 stands at 169 / 2** (also +1, same cause). **Unclosed incident rows 1 → 2:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving the Facebook monthly's `0e753e_e7f8194a7d06` (`detected`, cannot clear before its 2026-11-01 slot) **plus the new `ea79a2_d93c3acdc4f2`**. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report); **zero delivery rows were created on 2026-10-30**, because the Mnemosyne Health Monitor's own failure notice **failed to send** (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed`) — a failure notice that was itself never delivered. The radar's two-stage split held its ground: `dca8482e4f76` (stage 1) and `5fc292ff0f3b` (stage 2) both read `ok` with streak 0, and every previously-red row (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) has since **cleared to `ok`** on its own scheduled slot — the page's older per-row notes still claimed `error` for several of them. **Pins: 18 of the 25 enabled jobs pin `custom:omni-101`** (11 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`); the other 7 enabled rows are `no_agent` jobs with no pin. +- **Current as of 2026-10-31 00:3x AWST (live `jobs.json` + `executions.db`) — the fleet is whole, but it is whole + because a 5 h 30 m HOST OUTAGE ended, not because anything was repaired.** Fleet **29 jobs — 26 enabled / + 3 disabled** (`jobs.json.updated_at 2026-10-31T00:29:59+08:00`). **Window 2026-10-30: 138 executions — 136 + `completed`, 0 `failed`, 2 `running`** (the 16:37 catch-up + this audit), against **177 / 177 / 0** on 10-29 → + **≈39 slots lost**. **Zero execution rows exist between 10:58:20 and 16:37:21**: PVE host `pve` rebooted (**boot + 2026-10-30 16:24:00**, `ssh … uptime -s`), CT460 came back **16:28:00**, the previous gateway life + (`pid=3913841`, started 2026-10-28T17:21:58Z) was logged as exited **UNCLEANLY — *"no exit path ran — SIGKILL / + OOM / VM death"*** at **16:35:50**, and the scheduler resumed with an **8-run burst at 16:37:21–16:38:22**. + Root cause **not established** (no OOM evidence; that ledger string is a candidate list, not a diagnosis) → + [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]. **`ea79a2107e36` (Mnemosyne Health Monitor) is + ENABLED AGAIN and clean**: its 00:01:08 slot did not die at 01:04:22 — it **closed `completed` at 10:02:48** + after **10 h 01 m 40 s** and **delivered**, auto-closing incident `ea79a2_d93c3acdc4f2` **`resolved` at + 2026-10-30T10:02:48.662+08:00**; live `last_status: ok`, `failure_streak: 0`. Its **12:01 slot was lost to the + outage** and is the still-`running` 16:37:21 catch-up. **Non-clean `last_status` 4 → 3**, **failing (run failed + outright) 1 → 0**, **unclosed incident rows 2 → 1** (survivor: `0e753e_e7f8194a7d06`, due 11-01). + **`deliveries.db` finished 2026-10-30 with 8 rows, all `delivered`** (latest 18:46:04). 2026-10-31 opened clean + (5 runs, all `completed`). +- **⚠ Superseded by the 2026-10-31 entry above — Cron as of 2026-10-30 03:3x AWST (live `jobs.json` + `executions.db`) — the 10-29 clean day held, but the fleet changed shape overnight: the Mnemosyne Health Monitor was auto-paused and its failure notice could not be delivered.** Fleet now **29 jobs (25 enabled / 4 disabled)** — `ea79a2107e36` (Mnemosyne Health Monitor) was **auto-paused at 2026-10-30 01:04:22** after its 00:01:08 slot closed **`unknown`** on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, which opened incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`). Non-clean `last_status` is now **4** — the three long-standing survivors (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`) **plus `ea79a2107e36` (`error`, `failure_streak: 1`)** — so the standing "every `failure_streak` reads 0" claim is now false. Window **2026-10-29 settled 177 `completed` / 0 `failed`** (the page previously said 176 — the 23:59 Wiki Audit row `cbfe434a3b5e` had still been `running` at audit time and has since settled `completed`); **2026-10-28 stands at 169 / 2** (also +1, same cause). **Unclosed incident rows 1 → 2:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving the Facebook monthly's `0e753e_e7f8194a7d06` (`detected`, cannot clear before its 2026-11-01 slot) **plus the new `ea79a2_d93c3acdc4f2`**. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report); **zero delivery rows were created on 2026-10-30**, because the Mnemosyne Health Monitor's own failure notice **failed to send** (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed`) — a failure notice that was itself never delivered. The radar's two-stage split held its ground: `dca8482e4f76` (stage 1) and `5fc292ff0f3b` (stage 2) both read `ok` with streak 0, and every previously-red row (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) has since **cleared to `ok`** on its own scheduled slot — the page's older per-row notes still claimed `error` for several of them. **Pins: 18 of the 25 enabled jobs pin `custom:omni-101`** (11 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`); the other 7 enabled rows are `no_agent` jobs with no pin. - **Prior window — as of 2026-10-30 00:0x AWST *(superseded by the 2026-10-30 03:3x entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split worked end to end for the first time, and the fleet had its first clean day in a week.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3**, and every `failure_streak` read **0** — the three survivors are all stale re-runs of long-standing rows (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`). Window **2026-10-29 settled 176 `completed` / 0 `failed`** (+ this audit `running`) — the first zero-failure day since 2026-10-24; 2026-10-28 stands at **168 / 2**. **Unclosed incident rows 4 → 1:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving only the Facebook monthly's `0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report). `dca8482e4f76` (stage 1) ran **07:00:59 → 10:16:12** (**3 h 15 m 13 s**), closed **`completed`**, and wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok` — opening **no** new incident row (contrast the two consecutive stage-1 failures on 10-27 and 10-28). `5fc292ff0f3b` (stage 2) fired **10:13:46**, consumed the handoff **10:40:59**, closed **10:44:48 `completed`** and delivered the same second: **20 pages / 20 `page_ids` / 2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went **455 → 475**. Both later slots no-oped as designed (**11:30 → 10 m 46 s**, **14:30 → 1 m 51 s**). Stage 1 correctly no longer delivers. See [[incidents/2026-10-27-job-radar-budget-exhausted]] and [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], now both **resolved**. **Pins unchanged** — 19 of the 26 enabled jobs pin `custom:omni-101` (12 `smart-route`, 6 `free-stack`, 1 explicit). - **Prior window — as of 2026-10-29 00:3x AWST *(superseded by the 2026-10-30 entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real, but stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.** - **Prior window — as of 2026-10-28 00:2x AWST** (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. *(superseded by the 2026-10-29 entry above)* @@ -29,9 +46,19 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] - **Current as of 2026-10-19 00:20 AWST *(superseded by the 2026-10-20 entry above)* (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]] - **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01; DB re-read 2026-10-20: **163 / 5** — one row was double-counted at the time) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] -- **Total jobs:** **29 — 25 enabled, 4 disabled** (verified from live `~/.hermes/cron/jobs.json` at 2026-10-30 03:3x AWST). The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. The **4th disabled job is `ea79a2107e36` (Mnemosyne Health Monitor)**, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; the other three are the long-standing `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher) and `0a7f933d06e2` (AgentMail Rhino Inbox Poll). -- **Non-clean `last_status`:** **4 as of 2026-10-30 03:3x AWST** — 2 `error` + 2 `delivery_failed`. Live `jobs.json` (authoritative for the per-job counters): `ea79a2107e36` (Mnemosyne Health Monitor — `error`, `failure_streak: 1`, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; incident `ea79a2_d93c3acdc4f2` open `detected`), `0e753e9720fd` (Facebook Birthday Monthly Scrape — `error`, streak 0, next slot 11-01 09:00), `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — `delivery_failed`, streak 0, most recent failure **2026-10-26 03:08:15**) and `066bb5191280` (Uber Eats Monthly Spend Report — `delivery_failed`, streak 0, next slot 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → 12 (10-20 00:09) → 3 (10-30 00:0x) → **4 now**, the increase being the Mnemosyne Health Monitor's overnight gateway fault. -- **Failing (last run failed outright):** **1** — `ea79a2107e36` (Mnemosyne Health Monitor), `failure_streak: 1`. Its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, and the job was **auto-paused** at that moment — it is now the 4th disabled job. ⚠ **Its failure notice was never delivered**: `hermes cron list` reports *"The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"*, and `deliveries.db` holds **zero rows for 2026-10-30**. A failure notice that fails to send is the same green-row/dead-job trap as [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the incident row `ea79a2_d93c3acdc4f2` is the only trace. The other previously-red jobs (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) have all **cleared to `ok` with streak 0** on their own scheduled slots. +- **Total jobs:** **29 — 26 enabled, 3 disabled** (re-verified from live `~/.hermes/cron/jobs.json` at 2026-10-31 + 00:3x AWST). ⚠ Superseded reading: the 2026-10-30 03:3x audit counted **25 enabled / 4 disabled**; the 4th + disabled job was `ea79a2107e36` (Mnemosyne Health Monitor), and it is **enabled again** as of 2026-10-30 + 10:02:48 (`last_status: ok`, streak 0) — so **every "4th disabled job" statement below is superseded**. The job + count itself is unchanged at 29. The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. The **4th disabled job is `ea79a2107e36` (Mnemosyne Health Monitor)**, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; the other three are the long-standing `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher) and `0a7f933d06e2` (AgentMail Rhino Inbox Poll). +- **Non-clean `last_status`:** **3 as of 2026-10-31 00:3x AWST** — **1 `error` + 2 `delivery_failed`**. ⚠ Superseded: + the 2026-10-30 03:3x reading was **4** (2 `error` + 2 `delivery_failed`); its first entry, `ea79a2107e36` + (Mnemosyne Health Monitor, `error`, `failure_streak: 1`), has **cleared to `ok` / 0** — its 00:01:08 slot was + still running, not failed, and closed `completed` at 10:02:48. The job list that follows is the 03:3x snapshot. Live `jobs.json` (authoritative for the per-job counters): `ea79a2107e36` (Mnemosyne Health Monitor — `error`, `failure_streak: 1`, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; incident `ea79a2_d93c3acdc4f2` open `detected`), `0e753e9720fd` (Facebook Birthday Monthly Scrape — `error`, streak 0, next slot 11-01 09:00), `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — `delivery_failed`, streak 0, most recent failure **2026-10-26 03:08:15**) and `066bb5191280` (Uber Eats Monthly Spend Report — `delivery_failed`, streak 0, next slot 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → 12 (10-20 00:09) → 3 (10-30 00:0x) → **4 now**, the increase being the Mnemosyne Health Monitor's overnight gateway fault. +- **Failing (last run failed outright):** **0 as of 2026-10-31 00:3x AWST**. ⚠ Superseded: the 2026-10-30 03:3x + reading was **1** — `ea79a2107e36` (Mnemosyne Health Monitor), `failure_streak: 1` — and that verdict was the + reconciliation path, **not the run**: the slot closed `completed` at 10:02:48 after 10 h 01 m 40 s and the + counters reset. **Only delivery-only failures remain (2).** Its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, and the job was **auto-paused** at that moment — it is now the 4th disabled job. ⚠ **Its failure notice was never delivered**: `hermes cron list` reports *"The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"*, and `deliveries.db` holds **zero rows for 2026-10-30**. A failure notice that fails to send is the same green-row/dead-job trap as [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the incident row `ea79a2_d93c3acdc4f2` is the only trace. The other previously-red jobs (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) have all **cleared to `ok` with streak 0** on their own scheduled slots. - Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04). - **Date-bucketing note (added 2026-10-25):** `executions.db` stores `started_at` with a `+08:00` offset, and SQLite's `date()` normalises to UTC — so `date(started_at)` moves every row that ran 00:00–07:59 AWST onto the **previous** day. Daily tallies on this page are bucketed with `substr(started_at,1,10)`, which preserves AWST calendar days. A recount via `date()` reads low by roughly a day's early-morning runs; this is the same class of drift that put 10-21 at 168 instead of 171. - **Failing (delivery only):** **2** — `066bb5191280` (Uber Eats Monthly Spend Report: the report itself generated; the send failed against the DM thread — bare `telegram` target, 2026-10-01) and `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — the same dead-thread cause, most recently **2026-10-26 03:08:15**). Both carry `failure_streak: 0` because the *run* succeeded; only the delivery failed. @@ -67,7 +94,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] | Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent | | Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent | | Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95%. ⚠ **Both 503-era slots failed** (10-16 12:00, 10-17 00:00 — two of the four watchdog reds quote this job's error); cleared by its own **10-17 06:01** run. ⚠ **Both 2026-10-19 slots failed the same way** (06:01:05 and 12:01:16), taking its streak to **2** and re-triggering the watchdog; its **18:01 slot succeeded**, clearing the streak and auto-closing `b63ed5_bbe44c89aac1` at 18:01:56. Live `last_status: ok`. With `0 */6 * * *` this job is the fleet's best 503 sentinel — four probes a day | -| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. ⚠ Same two 503 slots in burst 1 (10-16 12:00, 10-17 00:00), then **again at 2026-10-19 12:01** — incident `ea79a2_747b27556fd0` was first opened 10-16 12:03, refreshed 10-19 12:02:55, and **auto-closed 2026-10-20 00:04:24** when the 00:01 slot succeeded. Live `last_status: ok`, streak 0 ⚠ **Live status re-verified 2026-10-30 03:3x — the job is now DISABLED and failing:** its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, opening incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`), and the scheduler **auto-paused the job at that moment**. Live `last_status: error`, `failure_streak: 1`, `enabled: false`. The "Live `last_status: ok`, streak 0" note above is **superseded**. ⚠ **The failure notice was never delivered** — `hermes cron list` reports *"live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"* and `deliveries.db` has **zero rows for 2026-10-30**. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. | +| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. **Live status re-verified 2026-10-31 00:3x: ENABLED, `last_status: ok`, `failure_streak: 0`; incident `ea79a2_d93c3acdc4f2` closed `resolved` 2026-10-30T10:02:48** (⚠ the 2026-10-30 03:3x DISABLED/`unknown` note later in this row is superseded by that; its 12:01 slot was lost to the CT460 host outage and fired late at 16:37:21). ⚠ Same two 503 slots in burst 1 (10-16 12:00, 10-17 00:00), then **again at 2026-10-19 12:01** — incident `ea79a2_747b27556fd0` was first opened 10-16 12:03, refreshed 10-19 12:02:55, and **auto-closed 2026-10-20 00:04:24** when the 00:01 slot succeeded. Live `last_status: ok`, streak 0 ⚠ **Live status re-verified 2026-10-30 03:3x — the job is now DISABLED and failing:** its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, opening incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`), and the scheduler **auto-paused the job at that moment**. Live `last_status: error`, `failure_streak: 1`, `enabled: false`. The "Live `last_status: ok`, streak 0" note above is **superseded**. ⚠ **The failure notice was never delivered** — `hermes cron list` reports *"live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"* and `deliveries.db` has **zero rows for 2026-10-30**. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. | | Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30). ⚠ **Its reds are derivative — it double-counts other jobs' failures.** During the 503 burst it exited 1 on **4 slots** (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own error from the log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…`. One sentinel watching another job, not four independent faults. **The pattern repeated exactly in burst 2** — four more reds on 2026-10-19 (06:16, 06:31, 12:16, 12:31), all quoting the same coverage-monitor error, all clearing at **12:46:11** (`96eb95_565ec1deb1c7`, a 10-16 row refreshed through 10-19 12:31). **8 of the 26 failure rows across the two bursts were this one sentinel** | | Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified `delivered`, status `ok` | | Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | ⚠ Two node faults, no code fault: 2026-10-15 08:06 `RuntimeError: Connection error.` (`0e5198_be2291809e56`) and **2026-10-19 08:01 `HTTP 503 … resource pressure`** (new incident `0e5198_ec2fac98a91d`, `alerted`). Both are open; `failure_streak: 2`. Next slot Thu 2026-10-22 08:00 | ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "`failure_streak: 2`" note above is **superseded**; the job cleared on its Mon/Thu 08:00 slots after 2026-10-19. | @@ -89,16 +116,33 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] | Mosquito Tracker | 2026-10-16 (after the 15:00 run) | `1b31187397a5` — absent from `jobs.json` at the 2026-10-18 audit; last execution 2026-10-16 15:00 was `completed` / `delivered` | | Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up | -## Disabled Jobs (not removed) +## Disabled Jobs (not removed) — **3 as of 2026-10-31** | Name | ID | Disabled since | Reason | |---|---|---|---| | Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. | | Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. | | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. | -| Mnemosyne Health Monitor | `ea79a2107e36` | **2026-10-30 01:04:22** | **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. | +| ~~Mnemosyne Health Monitor~~ (**re-enabled 2026-10-30**) | `ea79a2107e36` | **2026-10-30 01:04:22 → re-enabled same day** | ⚠ **SUPERSEDED as of 2026-10-31 — the job is ENABLED** (`enabled: true`, `last_status: ok`, `failure_streak: 0`) and incident `ea79a2_d93c3acdc4f2` closed **`resolved` 2026-10-30T10:02:48.662+08:00**. The following describes the 03:3x snapshot only. **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. | ## Known Issues +### Resolved 2026-10-30 + +- **The Mnemosyne Health Monitor was never really down — and it is enabled again (`ea79a2107e36`).** Its 00:01:08 + slot was still **running** at the 03:3x reading; the `unknown`/`error` verdict came from the admission + reconciliation, not the run. It **closed `completed` at 10:02:48** after **10 h 01 m 40 s** and **delivered**, + auto-closing incident `ea79a2_d93c3acdc4f2` **`resolved` at 2026-10-30T10:02:48.662+08:00**. Live: + `enabled: true`, `last_status: ok`, `failure_streak: 0`. **Unclosed incident rows 2 → 1**, **non-clean + `last_status` 4 → 3**, **failing 1 → 0**. ⚠ The pause was real to record, but the "job is disabled / notice + never delivered" state described in [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] lasted only until + 10:02:48 — `deliveries.db` finished 2026-10-30 with **8 rows, all `delivered`** (latest 18:46:04), so the + zero-row/never-delivered reading was a 03:3x artefact of the day having just begun. +- **CT460's 5 h 30 m host outage is closed only because the reboot ended it — nothing was repaired.** PVE `pve` + booted **2026-10-30 16:24:00**, CT460 **16:28:00**, scheduler resumed **16:37:21** with 8 catch-up runs + → [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]. **≈39 slots lost, 0 failures.** **No monitor + alerted** — the two jobs that could have (`cbfe434a3b5e` NFS mount monitor, `1a8e90ea2271` m93p LXC boot monitor) + are disabled, and this remains the open item. + ### Resolved 2026-10-29 - **The Job Radar's two-stage split finished end to end for the first time (`dca8482e4f76` + `5fc292ff0f3b`):** stage 1 ran **07:00:59 → 10:16:12** (3 h 15 m 13 s), closed **`completed`** and wrote `handoff/2026-10-29.json` (**170 discovered / 33 duplicates avoided / 37 verified**) — the first post-split run to write a handoff, after 10-27 died on the **tool-call budget** and 10-28 on the **iteration cap**. Stage 2 consumed it at **10:40:59**, closed `completed` at **10:44:48** and **delivered** to topic 1030 in the same second: **20 pages, 2 cover letters, `missing_cover_letter_count 0`**, Notion **455 → 475**. Both stage-1 ledger rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12**, and the run opened no new row → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], [[runbooks/job-radar]]. ⚠ The discovery stage is still **slow** (3 h 15 m) and both ceilings remain reachable — a `completed` stage-1 row with **no handoff file** is the same fault in new clothing.