audit(2026-10-31): document CT460 5h30m host outage; correct the 'paused' health monitor (it was running, closed completed 10:02:48); reconcile fleet counts to 26/3

This commit is contained in:
hermes
2026-10-31 00:39:11 +08:00
parent db6af66878
commit 74a5b2d117
8 changed files with 241 additions and 27 deletions
+17 -3
View File
@@ -3,8 +3,8 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-10-30
verified_on: 2026-10-30
updated: 2026-10-31
verified_on: 2026-10-31
review_after: 2026-11-07
confidence: high
tags: [system]
@@ -43,7 +43,21 @@ Updated: 2026-10-30
**10 deliveries, all 10 `delivered`, 0 failures**. Non-clean `last_status` **5 → 3**; unclosed incident rows
**4 → 1** (the two radar rows closed `resolved` at 10:16:12). Fleet unchanged at **29 jobs (26 enabled / 3
disabled)** at that audit.
- **Cron fleet changed shape overnight on 2026-10-30 — the Mnemosyne Health Monitor is paused and its failure
- **Cron: 2026-10-30 closed 138 / 0 — after a 5 h 30 m HOST OUTAGE — and the "auto-paused" health monitor was never
dead, it was running.** Fleet **29 jobs (26 enabled / 3 disabled)**
(`jobs.json.updated_at 2026-10-31T00:29:59+08:00`), re-verified 2026-10-31 00:3x. **Zero execution rows exist
between 10:58:20 and 16:37:21** — PVE `pve` rebooted (boot **2026-10-30 16:24:00**), CT460 came back **16:28:00**,
the old gateway life (`pid=3913841`) was logged **UNCLEANLY** at **16:35:50** (*"no exit path ran — SIGKILL / OOM /
VM death"* — a candidate list, not a diagnosis; root cause **not established**, no OOM evidence), and 8 catch-up
runs fired at **16:37:21–16:38:22** → [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]. Against 10-29's
**177 / 0**, that is **≈39 lost slots and no failures**. **`ea79a2107e36` (Mnemosyne Health Monitor) is ENABLED
and clean again** — `last_status: ok`, streak **0**, incident `ea79a2_d93c3acdc4f2` closed **`resolved`
2026-10-30T10:02:48.662** when its 00:01:08 slot finished after **10 h 01 m 40 s**; its **12:01 slot was lost to
the outage** and is the still-`running` 16:37:21 catch-up. **Non-clean `last_status` 4 → 3**, **failing 1 → 0**,
**unclosed incident rows 2 → 1** (survivor: `0e753e_e7f8194a7d06`, due 11-01). **`deliveries.db` finished 10-30
with 8 rows, all `delivered`** (latest 18:46:04) — the 03:3x "zero rows / notice never delivered" reading is
superseded.
- **⚠ Superseded by the 2026-10-31 entry above — read that first.** Cron fleet changed shape overnight on 2026-10-30 — the Mnemosyne Health Monitor is paused and its failure
notice was never delivered.** `ea79a2107e36`'s 00:01:08 slot closed **`unknown`** at 01:04:22 on a
gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`:
Gateway connection failed; no local fallback"*), opening incident `ea79a2_d93c3acdc4f2` (`detected`), and the
@@ -0,0 +1,103 @@
---
title: CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30)
type: incident
status: resolved
created: 2026-10-31
updated: 2026-10-31
verified_on: 2026-10-31
confidence: high
tags: [cron, gateway, host, outage, pve, ct460, incident]
sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-agent-log, live-deliveries-db, ssh-pve-uptime]
---
# CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30)
**Status: RESOLVED** — the container and scheduler came back on 2026-10-30 ~16:28–16:37 and the day
closed clean (138 executions, 136 `completed`, **0 `failed`**). **Root cause of the host reboot is not
established.** No monitor alerted; the gap was found by this audit.
## Summary
The Hermes runtime went dark at **~10:58 AWST on 2026-10-30** and did not return until
**16:28** (container) / **16:37:21** (first execution). The PVE host `pve` itself rebooted — its boot
time is **2026-10-30 16:24:00** — so this was a **host-level outage, not a cron or gateway fault**.
The gateway's previous life died with no exit path, which the lifecycle ledger logs as
`SIGKILL / OOM / VM death`. Nothing noticed for **5 h 37 m**.
## Timeline (all times AWST, 2026-10-30)
| Time | Event | Evidence |
|---|---|---|
| 10:49:54 | Last execution before the gap starts | `executions.db` last row `< 16:37` |
| ~10:58:20 | Gateway life `pid=3913841` dies; no exit path ran | `gateway-exit-diag.log` `gateway.previous_unclean_exit` (prior life started 2026-10-28T17:21:58Z) |
| 10:58:20 → 16:37:21 | **Zero execution rows** (5 h 39 m) | `executions.db` count in window = **0** |
| 16:24:00 | PVE host `pve` (`100.96.100.82`) boots | `ssh root@100.96.100.82 'uptime -s; who -b'` → both read `2026-10-30 16:24` |
| 16:28:00 | CT460 container boots | local `uptime -s` → `2026-10-30 16:28:00` |
| 16:35:50 | Unclean exit noticed and logged | `logs/agent.log`, `logs/errors.log` — `Previous gateway life (pid=3913841, started_at=2026-10-28T17:21:58Z) exited UNCLEANLY (no exit path ran — SIGKILL / OOM / VM death)` |
| **16:37:21** | Scheduler resumes — **8-run catch-up burst** (16:37:21 → 16:38:22) | `executions.db` |
| 23:59:58 | Daily wiki audit (this run) starts; the 16:37 health-monitor catch-up is still `running` | `executions.db` |
## The resume burst (16:37:21 → 16:38:22)
Eight jobs fired within 61 seconds of the scheduler coming back. Seven closed `completed`;
the eighth is still running.
| Job | Started | Closed |
|---|---|---|
| FreshRSS Midday News Briefing (`ecc1fc13655a`) | 16:37:21 | 17:45:31 — the 12:00 slot, delivered late |
| Mnemosyne Auto-Consolidate | 16:37:21 | 16:37:54 |
| Mnemosyne Embedding Coverage Monitor | 16:37:21 | 16:41:31 |
| Mnemosyne Watchdog | 16:37:21 | 16:38:33 |
| **Mnemosyne Health Monitor (`ea79a2107e36`)** | **16:37:21** | **RUNNING at audit** — the late 12:01 slot |
| Job Tracker Dashboard Refresh | 16:37:23 | 16:37:35 |
| 🦝 Hermes Config Backup to Gitea (Daily) | 16:38:06 | 16:42:02 |
| Daily Marketing Job Radar - Write and Letters | 16:38:22 | 16:41:39 |
## Impact
- **≈39 scheduled runs lost.** 2026-10-29 fired **177** executions (177 `completed`, 0 `failed`); 2026-10-30
fired **138** (136 `completed`, 0 `failed`, 2 `running` — the 16:37 catch-up plus this audit). No row exists
inside the dark window, so the deficit is unfired slots, not failures.
- **No failures were caused.** Nothing was recorded `failed` on 2026-10-30 — the outage simply removed the
opportunities to run.
- **The two long Mnemosyne runs absorbed it.** The FreshRSS midday briefing regenerated on the late fire
(16:37:21 → 17:45:31), and the health monitor's 12:01 slot is the still-running 16:37:21 fire — normal for
this job, whose 00:01 slot took **10 h 01 m 40 s** the same day.
- **Deliveries were unaffected once back up:** `deliveries.db` holds **8 rows for 2026-10-30, all `delivered`**
(latest 18:46:04) — the opposite of the 03:3x snapshot, which read zero rows because the day had barely started.
- **2026-10-31 opened clean:** 5 executions (00:01:05 ×3, 00:15:59, 00:23:57), all `completed`.
## Root cause — not established
- The host rebooted. `pve` boot time is 16:24:00; **no OOM evidence was found** on the CT460 side (local
`dmesg` grep empty, `/var/log/wtmp` absent, no `suspected_oom` marker in the webui session store).
- The ledger's own wording (`SIGKILL / OOM / VM death`) is a **list of candidates for "no exit path ran"**, not a
finding. Treating it as a diagnosis would be wrong.
- Same-day, different host: [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] records **m93p**
reboots (~15:55) that took down NFS-backed containers. The two events are **separate hosts and separate
causes as far as the evidence goes** — they are not merged here, but the timing is close enough to warrant
checking the two hypervisors' last-boot records together before either is called a one-off.
- **To close it:** PVE syslog / `journalctl -b -1` on `100.96.100.82` around 2026-10-30 10:58, and the host's
own last-boot records (`last -x reboot`). Neither was read for this page — say so rather than guess.
## Why nothing alerted
The jobs positioned to notice a host-down event are **disabled**:
| Job | ID | State |
|---|---|---|
| NFS mount monitor | `cbfe434a3b5e` | disabled |
| m93p LXC boot monitor | `1a8e90ea2271` | disabled |
Everything else is *inside* the scheduler that went dark, so it could not have noticed. **A host-level outage of
the Hermes VM currently has no in-estate detector** — the fleet learned about it from the 23:59 audit, 13 h
after the fact. This is the incident's one open item.
## Related
- [[systems/scheduled-tasks]] — the fleet entry for this window
- [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — the same day's health-monitor story: its 00:01 slot
closed `unknown` at 01:04:22, incident `ea79a2_d93c3acdc4f2` **auto-closed `resolved` at 10:02:48** when that
run finally completed, and the job is live again (`enabled: true`, `last_status: ok`, streak 0)
- [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] — sister-host reboot, same day
- [[infrastructure/hosts]] — `pve` / CT460 addressing
@@ -1,10 +1,10 @@
---
title: Mnemosyne Health Monitor auto-paused after gateway-connection failure
type: incident
status: active
status: resolved
created: 2026-10-30
updated: 2026-10-30
verified_on: 2026-10-30
updated: 2026-10-31
verified_on: 2026-10-31
confidence: high
tags: [cron, mnemosyne, gateway, incident, delivery]
sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents]
@@ -12,7 +12,17 @@ sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-inci
# Mnemosyne Health Monitor auto-paused after gateway-connection failure (2026-10-30)
**Status: ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
**Status: RESOLVED — 2026-10-30T10:02:48.662+08:00** (updated by the 2026-10-31 audit; everything below is the
2026-10-30 03:3x snapshot and its "job is disabled / notice never delivered" reading did not survive the day).
**What actually happened:** the 00:01:08 slot was **still running**, not dead — it closed **`completed` at
10:02:48** after **10 h 01 m 40 s** and **delivered**; incident `ea79a2_d93c3acdc4f2` **auto-closed `resolved`
at 2026-10-30T10:02:48.662+08:00**, and the job is live again (`enabled: true`, `last_status: ok`,
`failure_streak: 0`). `deliveries.db` finished 2026-10-30 with **8 rows, all `delivered`** (latest 18:46:04) — the
zero-row reading was an artefact of the day having barely begun. ⚠ Its **12:01 slot was then lost to the CT460
host outage** and fired late in the 16:37:21 catch-up burst, still `running` at the 2026-10-31 audit — see
[[incidents/2026-10-30-ct460-host-outage-scheduler-gap]].
**Historical snapshot (2026-10-30 03:3x): ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
## Summary
+6 -5
View File
File diff suppressed because one or more lines are too long
+3 -3
View File
@@ -3,8 +3,8 @@ title: Known Hosts and Paths
type: infrastructure
status: partial
created: 2026-07-22
updated: 2026-10-30
verified_on: 2026-10-30
updated: 2026-10-31
verified_on: 2026-10-31
confidence: medium
tags: [infrastructure, hosts, containers, paths]
sources: []
@@ -35,7 +35,7 @@ sources: []
| `/home/hermes/.local/bin/hermes` | Hermes binary | |
| `/home/hermes/.hermes/config.yaml` | Config | `_config_version: 51` |
| `/home/hermes/.hermes/skills/` | Skills dir | 302 skills across 69 categories (`.archive/` excluded) |
| `/home/hermes/.hermes/cron/` | Cron jobs | 29 jobs — 25 enabled / 4 disabled (the 4th disabled job, `ea79a2107e36` Mnemosyne Health Monitor, was auto-paused 2026-10-30 01:04 after a gateway-connection failure) |
| `/home/hermes/.hermes/cron/` | Cron jobs | **29 jobs — 26 enabled / 3 disabled** (re-verified 2026-10-31 00:3x). ⚠ Superseded: the 2026-10-30 03:3x reading was 25 / 4 — the 4th disabled job, `ea79a2107e36` Mnemosyne Health Monitor, was auto-paused 2026-10-30 01:04 after a gateway-connection failure, but **that pause lasted only until 10:02:48**, when its long-running slot closed `completed`, incident `ea79a2_d93c3acdc4f2` auto-closed `resolved`, and the job returned to `enabled: true` / `last_status: ok` / streak 0. **CT460 itself was then dark for 5 h 30 m on 2026-10-30** (PVE `pve` rebooted; host boot 16:24:00, CT460 16:28:00, scheduler resumed 16:37:21 — ≈39 cron slots lost, 0 failures) → [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]] |
| `/home/hermes/.hermes/mnemosyne/` | Mnemosyne DB | backups/, data/, logs/, models/ |
| `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | cua-driver | v0.9.0 |
| `~/.camofox/` | Camofox profiles | managed persistence |
+35
View File
@@ -1646,3 +1646,38 @@ not pushed.
- **Not changed:** `systems/skills-index.md` count (302 is correct), CT460's Serve route table (matched live exactly), and
the steel-removal records across the wiki (confirmed still true).
- **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a key or token value.
## [2026-10-31] audit | Nightly wiki audit — window 2026-10-30 00:00 → 2026-10-31 00:3x AWST
**Verdict: one large undocumented event found and written up, and one *documented* incident found to have been
recorded wrong.**
- **NEW PAGE — [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]].** CT460 was dark for **5 h 30 m** on
2026-10-30 and **no monitor alerted**. Evidence: **zero `executions.db` rows between 10:58:20 and 16:37:21**;
PVE host `pve` boot time **2026-10-30 16:24:00**; CT460 boot **16:28:00**; the previous gateway life
(`pid=3913841`, started 2026-10-28T17:21:58Z) logged **UNCLEANLY at 16:35:50** (*"no exit path ran —
SIGKILL / OOM / VM death"*); scheduler resumed **16:37:21** with an **8-run catch-up burst** through 16:38:22.
The day still closed **138 executions / 0 failed** against 10-29's **177 / 0** — **≈39 lost slots, no failures**.
Root cause **not established**: no OOM evidence locally or on the PVE host, so the ledger string is read as a
candidate list, not a diagnosis. **Still open:** the two jobs that could have detected it (`cbfe434a3b5e` NFS
mount monitor, `1a8e90ea2271` m93p LXC boot monitor) are disabled.
- **CORRECTED — [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] was right about the record and wrong
about the run.** `ea79a2107e36`'s 2026-10-30 00:01:08 slot was **still running** at the 03:3x reading; it closed
**`completed` at 10:02:48** after **10 h 01 m 40 s** and **delivered**, auto-closing incident
`ea79a2_d93c3acdc4f2` **`resolved` at 2026-10-30T10:02:48.662+08:00**. Live job state is now `enabled: true`,
`last_status: ok`, `failure_streak: 0`. The page's *"zero `deliveries.db` rows / the notice was never
delivered"* claim was an artefact of reading the day at 03:3x: 2026-10-30 finished with **8 delivery rows, all
`delivered`** (latest 18:46:04). Page frontmatter moved `active` → **`resolved`**, with the old snapshot kept
below as explicitly historical. ⚠ Its **12:01 slot was lost to the host outage** and is the still-`running`
16:37:21 catch-up. **Reconciliation lesson: a slot mid-flight is not a failure — check `finished_at`, not
`last_status`, before writing an incident.**
- **Reconciled — `systems/scheduled-tasks.md`.** Fleet **29 jobs (26 enabled / 3 disabled)** — the "4th disabled
job" reading is superseded (frontmatter date, fleet block, Total jobs, non-clean `last_status` **4 → 3**,
failing **1 → 0**, the health-monitor table row, the disabled table row, and a new **"Resolved 2026-10-30"**
subsection). Non-clean now **3 = 1 `error` + 2 `delivery_failed`**; unclosed incident rows **2 → 1** (survivor:
the Facebook monthly's `0e753e_e7f8194a7d06`, due 11-01).
- **Reconciled** — `current-state.md` (new leading cron entry + superseded marker), `infrastructure/hosts.md`
(cron row), `index.md` (frontmatter, banner, scheduled-tasks entry, both 10-30 incident entries),
`incidents/2026-10-30-mnemosyne-health-monitor-paused.md`.
- **Checked, unchanged:** 12 sessions active in-window (longest `local-40472e0b…`, 1,351 messages), no orphaned
`running` rows, no incidents opened or closed after 10-30, no obsolete pages to archive, no secrets written.
+10 -3
View File
@@ -3,8 +3,8 @@ title: Mnemosyne
type: system
status: active
created: 2026-07-22
updated: 2026-10-24
verified_on: 2026-10-24
updated: 2026-10-31
verified_on: 2026-10-31
review_after: 2026-11-07
confidence: high
tags: [memory, mnemosyne, backend]
@@ -94,7 +94,14 @@ Primary agent memory store for Hermes.
## Known Management Jobs
- Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception.
⚠ **2026-10-30: the monitor was twice flagged and neither flag survived contact with `finished_at`.** Its
00:01:08 slot was auto-paused at 01:04:22 on an *"execution unverified / gateway connection failed"* admission
([[incidents/2026-10-30-mnemosyne-health-monitor-paused]]) — but the slot was **still running** and closed
**`completed` at 10:02:48** after **10 h 01 m 40 s**, delivering normally and auto-closing incident
`ea79a2_d93c3acdc4f2`; the job is **enabled**, `last_status: ok`, streak 0. Its **12:01 slot was then lost to
the CT460 host outage** (5 h 30 m dark, [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]) and fired in
the 16:37:21 catch-up. **Check `finished_at` before believing a long-running slot failed.**
- Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
- ⚠ **Blast-radius note (2026-10-16).** Both monitors above (`ea79a2107e36` Health every 12 h,
+53 -9
View File
@@ -3,8 +3,8 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-30
verified_on: 2026-10-30
updated: 2026-10-31
verified_on: 2026-10-31
review_after: 2026-11-06
confidence: high
tags: [cron, scheduled, automation]
@@ -14,7 +14,24 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks
## Count
- **Current as of 2026-10-30 03:3x AWST (live `jobs.json` + `executions.db`) — the 10-29 clean day held, but the fleet changed shape overnight: the Mnemosyne Health Monitor was auto-paused and its failure notice could not be delivered.** Fleet now **29 jobs (25 enabled / 4 disabled)** — `ea79a2107e36` (Mnemosyne Health Monitor) was **auto-paused at 2026-10-30 01:04:22** after its 00:01:08 slot closed **`unknown`** on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, which opened incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`). Non-clean `last_status` is now **4** — the three long-standing survivors (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`) **plus `ea79a2107e36` (`error`, `failure_streak: 1`)** — so the standing "every `failure_streak` reads 0" claim is now false. Window **2026-10-29 settled 177 `completed` / 0 `failed`** (the page previously said 176 — the 23:59 Wiki Audit row `cbfe434a3b5e` had still been `running` at audit time and has since settled `completed`); **2026-10-28 stands at 169 / 2** (also +1, same cause). **Unclosed incident rows 1 → 2:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving the Facebook monthly's `0e753e_e7f8194a7d06` (`detected`, cannot clear before its 2026-11-01 slot) **plus the new `ea79a2_d93c3acdc4f2`**. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report); **zero delivery rows were created on 2026-10-30**, because the Mnemosyne Health Monitor's own failure notice **failed to send** (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed`) — a failure notice that was itself never delivered. The radar's two-stage split held its ground: `dca8482e4f76` (stage 1) and `5fc292ff0f3b` (stage 2) both read `ok` with streak 0, and every previously-red row (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) has since **cleared to `ok`** on its own scheduled slot — the page's older per-row notes still claimed `error` for several of them. **Pins: 18 of the 25 enabled jobs pin `custom:omni-101`** (11 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`); the other 7 enabled rows are `no_agent` jobs with no pin.
- **Current as of 2026-10-31 00:3x AWST (live `jobs.json` + `executions.db`) — the fleet is whole, but it is whole
because a 5 h 30 m HOST OUTAGE ended, not because anything was repaired.** Fleet **29 jobs — 26 enabled /
3 disabled** (`jobs.json.updated_at 2026-10-31T00:29:59+08:00`). **Window 2026-10-30: 138 executions — 136
`completed`, 0 `failed`, 2 `running`** (the 16:37 catch-up + this audit), against **177 / 177 / 0** on 10-29 →
**≈39 slots lost**. **Zero execution rows exist between 10:58:20 and 16:37:21**: PVE host `pve` rebooted (**boot
2026-10-30 16:24:00**, `ssh … uptime -s`), CT460 came back **16:28:00**, the previous gateway life
(`pid=3913841`, started 2026-10-28T17:21:58Z) was logged as exited **UNCLEANLY — *"no exit path ran — SIGKILL /
OOM / VM death"*** at **16:35:50**, and the scheduler resumed with an **8-run burst at 16:37:21–16:38:22**.
Root cause **not established** (no OOM evidence; that ledger string is a candidate list, not a diagnosis) →
[[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]. **`ea79a2107e36` (Mnemosyne Health Monitor) is
ENABLED AGAIN and clean**: its 00:01:08 slot did not die at 01:04:22 — it **closed `completed` at 10:02:48**
after **10 h 01 m 40 s** and **delivered**, auto-closing incident `ea79a2_d93c3acdc4f2` **`resolved` at
2026-10-30T10:02:48.662+08:00**; live `last_status: ok`, `failure_streak: 0`. Its **12:01 slot was lost to the
outage** and is the still-`running` 16:37:21 catch-up. **Non-clean `last_status` 4 → 3**, **failing (run failed
outright) 1 → 0**, **unclosed incident rows 2 → 1** (survivor: `0e753e_e7f8194a7d06`, due 11-01).
**`deliveries.db` finished 2026-10-30 with 8 rows, all `delivered`** (latest 18:46:04). 2026-10-31 opened clean
(5 runs, all `completed`).
- **⚠ Superseded by the 2026-10-31 entry above — Cron as of 2026-10-30 03:3x AWST (live `jobs.json` + `executions.db`) — the 10-29 clean day held, but the fleet changed shape overnight: the Mnemosyne Health Monitor was auto-paused and its failure notice could not be delivered.** Fleet now **29 jobs (25 enabled / 4 disabled)** — `ea79a2107e36` (Mnemosyne Health Monitor) was **auto-paused at 2026-10-30 01:04:22** after its 00:01:08 slot closed **`unknown`** on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, which opened incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`). Non-clean `last_status` is now **4** — the three long-standing survivors (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`) **plus `ea79a2107e36` (`error`, `failure_streak: 1`)** — so the standing "every `failure_streak` reads 0" claim is now false. Window **2026-10-29 settled 177 `completed` / 0 `failed`** (the page previously said 176 — the 23:59 Wiki Audit row `cbfe434a3b5e` had still been `running` at audit time and has since settled `completed`); **2026-10-28 stands at 169 / 2** (also +1, same cause). **Unclosed incident rows 1 → 2:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving the Facebook monthly's `0e753e_e7f8194a7d06` (`detected`, cannot clear before its 2026-11-01 slot) **plus the new `ea79a2_d93c3acdc4f2`**. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report); **zero delivery rows were created on 2026-10-30**, because the Mnemosyne Health Monitor's own failure notice **failed to send** (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed`) — a failure notice that was itself never delivered. The radar's two-stage split held its ground: `dca8482e4f76` (stage 1) and `5fc292ff0f3b` (stage 2) both read `ok` with streak 0, and every previously-red row (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) has since **cleared to `ok`** on its own scheduled slot — the page's older per-row notes still claimed `error` for several of them. **Pins: 18 of the 25 enabled jobs pin `custom:omni-101`** (11 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`); the other 7 enabled rows are `no_agent` jobs with no pin.
- **Prior window — as of 2026-10-30 00:0x AWST *(superseded by the 2026-10-30 03:3x entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split worked end to end for the first time, and the fleet had its first clean day in a week.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3**, and every `failure_streak` read **0** — the three survivors are all stale re-runs of long-standing rows (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`). Window **2026-10-29 settled 176 `completed` / 0 `failed`** (+ this audit `running`) — the first zero-failure day since 2026-10-24; 2026-10-28 stands at **168 / 2**. **Unclosed incident rows 4 → 1:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving only the Facebook monthly's `0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report). `dca8482e4f76` (stage 1) ran **07:00:59 → 10:16:12** (**3 h 15 m 13 s**), closed **`completed`**, and wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok` — opening **no** new incident row (contrast the two consecutive stage-1 failures on 10-27 and 10-28). `5fc292ff0f3b` (stage 2) fired **10:13:46**, consumed the handoff **10:40:59**, closed **10:44:48 `completed`** and delivered the same second: **20 pages / 20 `page_ids` / 2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went **455 → 475**. Both later slots no-oped as designed (**11:30 → 10 m 46 s**, **14:30 → 1 m 51 s**). Stage 1 correctly no longer delivers. See [[incidents/2026-10-27-job-radar-budget-exhausted]] and [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], now both **resolved**. **Pins unchanged** — 19 of the 26 enabled jobs pin `custom:omni-101` (12 `smart-route`, 6 `free-stack`, 1 explicit).
- **Prior window — as of 2026-10-29 00:3x AWST *(superseded by the 2026-10-30 entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real, but stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.**
- **Prior window — as of 2026-10-28 00:2x AWST** (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. *(superseded by the 2026-10-29 entry above)*
@@ -29,9 +46,19 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
- **Current as of 2026-10-19 00:20 AWST *(superseded by the 2026-10-20 entry above)* (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
- **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01; DB re-read 2026-10-20: **163 / 5** — one row was double-counted at the time) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
- **Total jobs:** **29 — 25 enabled, 4 disabled** (verified from live `~/.hermes/cron/jobs.json` at 2026-10-30 03:3x AWST). The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. The **4th disabled job is `ea79a2107e36` (Mnemosyne Health Monitor)**, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; the other three are the long-standing `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher) and `0a7f933d06e2` (AgentMail Rhino Inbox Poll).
- **Non-clean `last_status`:** **4 as of 2026-10-30 03:3x AWST** — 2 `error` + 2 `delivery_failed`. Live `jobs.json` (authoritative for the per-job counters): `ea79a2107e36` (Mnemosyne Health Monitor — `error`, `failure_streak: 1`, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; incident `ea79a2_d93c3acdc4f2` open `detected`), `0e753e9720fd` (Facebook Birthday Monthly Scrape — `error`, streak 0, next slot 11-01 09:00), `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — `delivery_failed`, streak 0, most recent failure **2026-10-26 03:08:15**) and `066bb5191280` (Uber Eats Monthly Spend Report — `delivery_failed`, streak 0, next slot 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → 12 (10-20 00:09) → 3 (10-30 00:0x) → **4 now**, the increase being the Mnemosyne Health Monitor's overnight gateway fault.
- **Failing (last run failed outright):** **1** — `ea79a2107e36` (Mnemosyne Health Monitor), `failure_streak: 1`. Its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, and the job was **auto-paused** at that moment — it is now the 4th disabled job. ⚠ **Its failure notice was never delivered**: `hermes cron list` reports *"The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"*, and `deliveries.db` holds **zero rows for 2026-10-30**. A failure notice that fails to send is the same green-row/dead-job trap as [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the incident row `ea79a2_d93c3acdc4f2` is the only trace. The other previously-red jobs (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) have all **cleared to `ok` with streak 0** on their own scheduled slots.
- **Total jobs:** **29 — 26 enabled, 3 disabled** (re-verified from live `~/.hermes/cron/jobs.json` at 2026-10-31
00:3x AWST). ⚠ Superseded reading: the 2026-10-30 03:3x audit counted **25 enabled / 4 disabled**; the 4th
disabled job was `ea79a2107e36` (Mnemosyne Health Monitor), and it is **enabled again** as of 2026-10-30
10:02:48 (`last_status: ok`, streak 0) — so **every "4th disabled job" statement below is superseded**. The job
count itself is unchanged at 29. The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. The **4th disabled job is `ea79a2107e36` (Mnemosyne Health Monitor)**, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; the other three are the long-standing `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher) and `0a7f933d06e2` (AgentMail Rhino Inbox Poll).
- **Non-clean `last_status`:** **3 as of 2026-10-31 00:3x AWST** — **1 `error` + 2 `delivery_failed`**. ⚠ Superseded:
the 2026-10-30 03:3x reading was **4** (2 `error` + 2 `delivery_failed`); its first entry, `ea79a2107e36`
(Mnemosyne Health Monitor, `error`, `failure_streak: 1`), has **cleared to `ok` / 0** — its 00:01:08 slot was
still running, not failed, and closed `completed` at 10:02:48. The job list that follows is the 03:3x snapshot. Live `jobs.json` (authoritative for the per-job counters): `ea79a2107e36` (Mnemosyne Health Monitor — `error`, `failure_streak: 1`, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; incident `ea79a2_d93c3acdc4f2` open `detected`), `0e753e9720fd` (Facebook Birthday Monthly Scrape — `error`, streak 0, next slot 11-01 09:00), `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — `delivery_failed`, streak 0, most recent failure **2026-10-26 03:08:15**) and `066bb5191280` (Uber Eats Monthly Spend Report — `delivery_failed`, streak 0, next slot 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → 12 (10-20 00:09) → 3 (10-30 00:0x) → **4 now**, the increase being the Mnemosyne Health Monitor's overnight gateway fault.
- **Failing (last run failed outright):** **0 as of 2026-10-31 00:3x AWST**. ⚠ Superseded: the 2026-10-30 03:3x
reading was **1** — `ea79a2107e36` (Mnemosyne Health Monitor), `failure_streak: 1` — and that verdict was the
reconciliation path, **not the run**: the slot closed `completed` at 10:02:48 after 10 h 01 m 40 s and the
counters reset. **Only delivery-only failures remain (2).** Its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, and the job was **auto-paused** at that moment — it is now the 4th disabled job. ⚠ **Its failure notice was never delivered**: `hermes cron list` reports *"The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"*, and `deliveries.db` holds **zero rows for 2026-10-30**. A failure notice that fails to send is the same green-row/dead-job trap as [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the incident row `ea79a2_d93c3acdc4f2` is the only trace. The other previously-red jobs (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) have all **cleared to `ok` with streak 0** on their own scheduled slots.
- Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04).
- **Date-bucketing note (added 2026-10-25):** `executions.db` stores `started_at` with a `+08:00` offset, and SQLite's `date()` normalises to UTC — so `date(started_at)` moves every row that ran 00:00–07:59 AWST onto the **previous** day. Daily tallies on this page are bucketed with `substr(started_at,1,10)`, which preserves AWST calendar days. A recount via `date()` reads low by roughly a day's early-morning runs; this is the same class of drift that put 10-21 at 168 instead of 171.
- **Failing (delivery only):** **2** — `066bb5191280` (Uber Eats Monthly Spend Report: the report itself generated; the send failed against the DM thread — bare `telegram` target, 2026-10-01) and `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — the same dead-thread cause, most recently **2026-10-26 03:08:15**). Both carry `failure_streak: 0` because the *run* succeeded; only the delivery failed.
@@ -67,7 +94,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
| Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent |
| Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent |
| Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95%. ⚠ **Both 503-era slots failed** (10-16 12:00, 10-17 00:00 — two of the four watchdog reds quote this job's error); cleared by its own **10-17 06:01** run. ⚠ **Both 2026-10-19 slots failed the same way** (06:01:05 and 12:01:16), taking its streak to **2** and re-triggering the watchdog; its **18:01 slot succeeded**, clearing the streak and auto-closing `b63ed5_bbe44c89aac1` at 18:01:56. Live `last_status: ok`. With `0 */6 * * *` this job is the fleet's best 503 sentinel — four probes a day |
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. ⚠ Same two 503 slots in burst 1 (10-16 12:00, 10-17 00:00), then **again at 2026-10-19 12:01** — incident `ea79a2_747b27556fd0` was first opened 10-16 12:03, refreshed 10-19 12:02:55, and **auto-closed 2026-10-20 00:04:24** when the 00:01 slot succeeded. Live `last_status: ok`, streak 0 ⚠ **Live status re-verified 2026-10-30 03:3x — the job is now DISABLED and failing:** its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, opening incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`), and the scheduler **auto-paused the job at that moment**. Live `last_status: error`, `failure_streak: 1`, `enabled: false`. The "Live `last_status: ok`, streak 0" note above is **superseded**. ⚠ **The failure notice was never delivered** — `hermes cron list` reports *"live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"* and `deliveries.db` has **zero rows for 2026-10-30**. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. **Live status re-verified 2026-10-31 00:3x: ENABLED, `last_status: ok`, `failure_streak: 0`; incident `ea79a2_d93c3acdc4f2` closed `resolved` 2026-10-30T10:02:48** (⚠ the 2026-10-30 03:3x DISABLED/`unknown` note later in this row is superseded by that; its 12:01 slot was lost to the CT460 host outage and fired late at 16:37:21). ⚠ Same two 503 slots in burst 1 (10-16 12:00, 10-17 00:00), then **again at 2026-10-19 12:01** — incident `ea79a2_747b27556fd0` was first opened 10-16 12:03, refreshed 10-19 12:02:55, and **auto-closed 2026-10-20 00:04:24** when the 00:01 slot succeeded. Live `last_status: ok`, streak 0 ⚠ **Live status re-verified 2026-10-30 03:3x — the job is now DISABLED and failing:** its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, opening incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`), and the scheduler **auto-paused the job at that moment**. Live `last_status: error`, `failure_streak: 1`, `enabled: false`. The "Live `last_status: ok`, streak 0" note above is **superseded**. ⚠ **The failure notice was never delivered** — `hermes cron list` reports *"live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"* and `deliveries.db` has **zero rows for 2026-10-30**. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30). ⚠ **Its reds are derivative — it double-counts other jobs' failures.** During the 503 burst it exited 1 on **4 slots** (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own error from the log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…`. One sentinel watching another job, not four independent faults. **The pattern repeated exactly in burst 2** — four more reds on 2026-10-19 (06:16, 06:31, 12:16, 12:31), all quoting the same coverage-monitor error, all clearing at **12:46:11** (`96eb95_565ec1deb1c7`, a 10-16 row refreshed through 10-19 12:31). **8 of the 26 failure rows across the two bursts were this one sentinel** |
| Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified `delivered`, status `ok` |
| Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | ⚠ Two node faults, no code fault: 2026-10-15 08:06 `RuntimeError: Connection error.` (`0e5198_be2291809e56`) and **2026-10-19 08:01 `HTTP 503 … resource pressure`** (new incident `0e5198_ec2fac98a91d`, `alerted`). Both are open; `failure_streak: 2`. Next slot Thu 2026-10-22 08:00 | ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "`failure_streak: 2`" note above is **superseded**; the job cleared on its Mon/Thu 08:00 slots after 2026-10-19. |
@@ -89,16 +116,33 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
| Mosquito Tracker | 2026-10-16 (after the 15:00 run) | `1b31187397a5` — absent from `jobs.json` at the 2026-10-18 audit; last execution 2026-10-16 15:00 was `completed` / `delivered` |
| Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up |
## Disabled Jobs (not removed)
## Disabled Jobs (not removed) — **3 as of 2026-10-31**
| Name | ID | Disabled since | Reason |
|---|---|---|---|
| Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. |
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
| Mnemosyne Health Monitor | `ea79a2107e36` | **2026-10-30 01:04:22** | **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
| ~~Mnemosyne Health Monitor~~ (**re-enabled 2026-10-30**) | `ea79a2107e36` | **2026-10-30 01:04:22 → re-enabled same day** | ⚠ **SUPERSEDED as of 2026-10-31 — the job is ENABLED** (`enabled: true`, `last_status: ok`, `failure_streak: 0`) and incident `ea79a2_d93c3acdc4f2` closed **`resolved` 2026-10-30T10:02:48.662+08:00**. The following describes the 03:3x snapshot only. **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
## Known Issues
### Resolved 2026-10-30
- **The Mnemosyne Health Monitor was never really down — and it is enabled again (`ea79a2107e36`).** Its 00:01:08
slot was still **running** at the 03:3x reading; the `unknown`/`error` verdict came from the admission
reconciliation, not the run. It **closed `completed` at 10:02:48** after **10 h 01 m 40 s** and **delivered**,
auto-closing incident `ea79a2_d93c3acdc4f2` **`resolved` at 2026-10-30T10:02:48.662+08:00**. Live:
`enabled: true`, `last_status: ok`, `failure_streak: 0`. **Unclosed incident rows 2 → 1**, **non-clean
`last_status` 4 → 3**, **failing 1 → 0**. ⚠ The pause was real to record, but the "job is disabled / notice
never delivered" state described in [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] lasted only until
10:02:48 — `deliveries.db` finished 2026-10-30 with **8 rows, all `delivered`** (latest 18:46:04), so the
zero-row/never-delivered reading was a 03:3x artefact of the day having just begun.
- **CT460's 5 h 30 m host outage is closed only because the reboot ended it — nothing was repaired.** PVE `pve`
booted **2026-10-30 16:24:00**, CT460 **16:28:00**, scheduler resumed **16:37:21** with 8 catch-up runs
→ [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]. **≈39 slots lost, 0 failures.** **No monitor
alerted** — the two jobs that could have (`cbfe434a3b5e` NFS mount monitor, `1a8e90ea2271` m93p LXC boot monitor)
are disabled, and this remains the open item.
### Resolved 2026-10-29
- **The Job Radar's two-stage split finished end to end for the first time (`dca8482e4f76` + `5fc292ff0f3b`):** stage 1 ran **07:00:59 → 10:16:12** (3 h 15 m 13 s), closed **`completed`** and wrote `handoff/2026-10-29.json` (**170 discovered / 33 duplicates avoided / 37 verified**) — the first post-split run to write a handoff, after 10-27 died on the **tool-call budget** and 10-28 on the **iteration cap**. Stage 2 consumed it at **10:40:59**, closed `completed` at **10:44:48** and **delivered** to topic 1030 in the same second: **20 pages, 2 cover letters, `missing_cover_letter_count 0`**, Notion **455 → 475**. Both stage-1 ledger rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12**, and the run opened no new row → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], [[runbooks/job-radar]]. ⚠ The discovery stage is still **slow** (3 h 15 m) and both ceilings remain reachable — a `completed` stage-1 row with **no handoff file** is the same fault in new clothing.