diff --git a/current-state.md b/current-state.md index 6529e4d..328528a 100644 --- a/current-state.md +++ b/current-state.md @@ -3,16 +3,29 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-11-17 -verified_on: 2026-11-17 -review_after: 2026-11-18 +updated: 2026-11-20 +verified_on: 2026-11-20 +review_after: 2026-11-21 confidence: high tags: [system] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db] --- # Current State -Updated: 2026-11-17 +Updated: 2026-11-20 + +## Recent Changes (2026-11-17 → 2026-11-20, three-window audit; live probes 2026-11-20 00:0x – 00:5x AWST) + +**The estate's memory freeze ended by itself, its two alarm paths are still red, and the audits that would have reported either had silently died twice.** + +- **Capture stall: over, unassisted.** `working_memory` **1130 → 1131**, `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST** — a **105 h 52 m** break that ended with no config change, restart or repair. ⚠ **And the "capture dead" reading was wrong:** `state.db` holds only **three** non-cron sessions since 11-14 00:00 — two `desktop` on **11-14 08:35 / 08:52** (both *before* the 09:33 boundary) and the **11-18 19:18** `desktop` whose turn *is* the captured row — so **every interactive session in the window was captured**. `mnemosyne_freshness_gate.py` reads the age of the newest non-`cron`/`subagent` message, so **silence and failure read identically**; that ambiguity is the real defect. The monitor is still failing on it (`failure_streak: 2`; slots 11-18 00:01 and 12:01, 11-19 12:01, 11-20 00:01 — 11-19 00:01 alone passed). → [[incidents/2026-11-14-mnemosyne-capture-stall]], [[systems/mnemosyne]] +- **The nightly audit missed two nights.** `cbfe434a3b5e` failed on **11-18** and **11-19** on a chained-provider credit/auth fault; **both deliveries `suppressed`**, and its incident row is still `alerted` with **`alerted_at: None`** → [[incidents/2026-11-18-wiki-audit-credit-outage]]. **This run covers 11-17, 11-18 and 11-19.** +- **The watchdog is in a threshold storm.** `96eb95ce12f2` failed **115 consecutive** 15-minute slots from **2026-11-18 20:01:06** to **2026-11-20 00:31:16** on **59 unresolved conflicts vs a threshold of 50** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] +- **Cron ledger.** 11-17 **173 rows (171 `completed` / 2 `failed`)**; 11-18 **172 (153/19)**; 11-19 **176 (78/98)** — the worst day in the retained window; 11-20 so far **8 (3 / 4 / 1 `running`)**. Deliveries since 11-18 **24 — 23 `delivered`** (9 `for_failure: 1`) / **1 `failed`** (content, Telegram timeout `2026-11-19 08:34:13`). Fleet **29 (26 enabled / 3 disabled)**, a twelfth consecutive window; non-clean `last_status` **1 → 4**; unclosed `cron_incidents` **5 → 4**. The **03:00 stocktake ran `completed` on all three days** (11 m 35 s / 23 m 07 s / 24 m 38 s). +- **Mnemosyne consolidation still zero-item:** newest `consolidation_log` row `2026-11-19T16:20:14+00:00`, `items_consolidated: 0`, **199 of the last 200 cycles zero-item** (was 120 on 11-17). Zero-item-streak threshold still **proposed, not applied**. +- **Pages touched:** [[incidents/2026-11-18-wiki-audit-credit-outage]] (new), [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] (new), [[incidents/2026-11-14-mnemosyne-capture-stall]] (new re-read + corrected title), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[index]], `log.md` — frontmatter dates → **2026-11-20**. + +**Previous window:** ## Recent Changes (2026-11-16, nightly audit; live probes 2026-11-16 23:59 – 2026-11-17 00:27 AWST) diff --git a/incidents/2026-11-14-mnemosyne-capture-stall.md b/incidents/2026-11-14-mnemosyne-capture-stall.md index f990dc5..524d795 100644 --- a/incidents/2026-11-14-mnemosyne-capture-stall.md +++ b/incidents/2026-11-14-mnemosyne-capture-stall.md @@ -1,17 +1,21 @@ --- -title: Mnemosyne capture stalled for 62+ hours (2026-11-14 09:33 AWST onward) — five unclosed alerts, freeze total on re-read +title: Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts type: incident status: active created: 2026-11-15 -updated: 2026-11-17 -verified_on: 2026-11-17 -review_after: 2026-11-18 +updated: 2026-11-20 +verified_on: 2026-11-20 +review_after: 2026-11-21 confidence: high tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate] sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script] --- -# Mnemosyne capture stalled for 62+ hours (from 2026-11-14 09:33 AWST) +# Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts + +> ⚠ **2026-11-20 re-read — the freeze ended by itself at 2026-11-18 19:25 AWST (105 h 52 m total), and every +> interactive session inside the window was captured.** The "capture is dead" reading below is a *quiet estate*, +> not a broken hook. Read the newest re-read section before the earlier readings. **Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**, and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it @@ -144,6 +148,58 @@ total.** `review_after` 2026-11-18 — this page is re-read at the next audit, and the monitor keeps alerting. +## Re-read 2026-11-20 00:5x AWST — the freeze ended by itself after **105 h 52 m**, and the "capture is dead" reading was wrong + +**The capture freeze is over. The alarm is not.** Live reads of `mnemosyne.db` at 2026-11-20 00:0x–00:5x AWST: + +| counter | 2026-11-17 00:27 | 2026-11-20 00:5x | +| --- | --- | --- | +| `working_memory` | 1130 | **1131** | +| `episodic_memory` | 1249 | **1250** | +| newest write | `2026-11-14 09:32:59` AWST | **`2026-11-18 19:25:03` AWST** (stored UTC `11:25:03`) | + +The break coincides with nothing: no config edit, no restart and no repair is recorded against the capture path +in the window — the only nearby state change is the 11-18 19:18 desktop session itself. The single new +`working_memory` row carries `session_id: hermes_20261118_191808_5c9e08` and its content begins +`[USER] You are helping me migrate contex…`, which identifies the captured turn (the `desktop` session +**`20261118_191808_5c9e08`**, *"Helping migrate context AI assistant"*, 9 messages, started 19:18 AWST). +**Stall duration: 105 h 52 m** (2026-11-14 09:33 → 2026-11-18 19:25 AWST). + +⚠ **The correction that matters — the window contained almost no conversation to capture.** `state.db` lists +**three** non-cron sessions since 2026-11-14 00:00 AWST: two `desktop` sessions on **11-14** (**08:35**, 40 +messages; **08:52**, 41 messages — both *before* the 09:33 boundary), then **nothing** until the **11-18 19:18** +session, whose turn is the row that ends the freeze. Every interactive session in the window was captured, so +the earlier figures on this page (`0 writes in 38 h 46 m`, `62 h 54 m`) are real but measure **a quiet estate**, +not a broken hook: `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message +whose session source is not `cron`/`subagent`, so **silence and failure read identically**. That ambiguity — not +a broken capture path — is the durable defect on this page. + +**The monitor is still failing on that same metric, after the break.** `ea79a2107e36` slots since 11-18: +`11-18 00:01:28` failed · `11-18 12:01:13` failed · **`11-19 00:01:34` `completed`** (the only clean slot since +11-17) · `11-19 12:01:17` failed · `11-20 00:01:22` failed — every failure `Script exited with code 1 / 🚨 +Mnemosyne Health Monitor — exception(s): STALE MEMORY PIPELINE: capture …`, i.e. **the same signature firing on +a healthy pipeline** (`failure_streak: 2`; incidents `ea79a2_f2be2db3bfce` at 2026-11-19 12:01:58 and +`ea79a2_aae4d914f3c9` at 2026-11-20 00:03:36, both `alerted`, both `failure_type: auth` — the scheduler's +classifier, not a credential fault). The `11-19 00:01` pass and the `12:01` failure bracket the threshold: the +metric is *age of the newest captured conversational turn*, which only grows while the estate is silent. + +**The other half of the original finding is unchanged, and larger.** The consolidator still consolidates zero +items: newest `consolidation_log` row **`2026-11-19T16:20:14.587267+00:00`** (= 2026-11-20 00:20 AWST, so it is +running on its ~31-minute cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle +— timestamp updated for watchdog"`. **199 of the last 200 cycles are zero-item** (was **120** at the 11-17 read +— consistent with ~31 min cadence over the three days). The zero-item-streak threshold remains **proposed, not +applied**. + +**Unclosed incidents: 5 → 4** (one of the rows present at the 11-17 read has since closed). Current four, none +closed automatically: `cbfe43_fa12cca1ad4b` (*auth*, 2026-11-18 00:01:42, `alerted_at: None` — the wiki audit's +silent skip, see [[incidents/2026-11-18-wiki-audit-credit-outage]]), `96eb95_98ade422d195` (*script*, +2026-11-18 20:01:15 → last_seen 2026-11-20 00:31:16 — see +[[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]), `ea79a2_f2be2db3bfce` (2026-11-19 +12:01:58) and `ea79a2_aae4d914f3c9` (2026-11-20 00:03:36). + +**Standing open question, restated:** not *"why did capture stop"* but **"why does the gate treat conversational +silence as a fault?"** Until that is fixed, this page will keep re-opening on quiet days. + ## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST) - Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e. diff --git a/incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md b/incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md new file mode 100644 index 0000000..c2868a1 --- /dev/null +++ b/incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md @@ -0,0 +1,111 @@ +--- +title: Mnemosyne Watchdog red for 115 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST) +type: incident +status: active +created: 2026-11-20 +updated: 2026-11-20 +verified_on: 2026-11-20 +review_after: 2026-11-21 +confidence: high +tags: [mnemosyne, memory, conflicts, watchdog, cron, alert-fatigue, incident] +sources: [live-cron-executions-db, live-cron-incidents, live-mnemosyne-db, live-deliveries-db, live-jobs-json] +--- + +# Mnemosyne Watchdog red for 115 consecutive slots (from 2026-11-18 20:01 AWST) + +**Open.** `96eb95ce12f2` (*Mnemosyne Watchdog*, 15-minute cadence) has failed **every slot** since +**2026-11-18 20:01:06 AWST** — **115 consecutive failures** through **2026-11-20 00:31:16 AWST**, the last +recorded before this page was written (it was 114 when the audit began; the streak gained a slot mid-audit). +`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`**. + +This is the estate's **only** red cron row with a large streak, and it is a *single durable condition* +re-reported every 15 minutes, not a series of separate faults. + +## The failure is a fixed, self-describing threshold trip + +``` +Script exited with code 1 +stdout: +🚨 Many unresolved conflicts: 59 (threshold: 50) +``` + +`cron_incidents` row **`96eb95_98ade422d195`** — `state: alerted`, `failure_type: script`, +`first_seen_at 2026-11-18T20:01:15.906837+08:00`, `last_seen_at 2026-11-20T00:31:16.326357+08:00`, +`alerted_at 2026-11-19T20:46:23.428143+08:00`, `closed_at: None`, output file +`/home/hermes/.hermes/cron/output/96eb95ce12f2/2026-11-20_00-16-15.md`. + +Unlike the health monitor's rows (whose signatures embed a changing float and so fragment into a new row per +slot → [[incidents/2026-11-14-mnemosyne-capture-stall]]), **this one fixes on the constant count `59` and +therefore stays a single row.** The cost of that stability is the mirror image: it never closes, and it +re-alerts forever. + +## Alerting works — and that is the problem + +`deliveries.db`, window 2026-11-18 → 2026-11-20 00:0x — the watchdog's own failure notices: + +| delivered | execution | +|---|---| +| 2026-11-18 20:01:16 | `f9865723b4f64d9fbfca7711ab053056` | +| 2026-11-19 02:16:16 | `7aa35681fd914a1e8db64b8dd0e09c84` | +| 2026-11-19 08:31:17 | `1ccc7e280d234287973edf5e0b35a5b7` | +| 2026-11-19 12:01:59 | `396940d9514b419eb024b83b3c10198d` | +| 2026-11-19 14:31:35 | `0766953f92dc4fc4b0d0404dca51683f` | +| 2026-11-19 20:46:19 | `db010a8294bf4cd78457a265a5fb167c` | +| 2026-11-20 00:03:37 | `64c6b5202463487c906f860bb746138b` | + +All `status: delivered`, all `for_failure: 1`. So detection, the failure notice and the push path are **all +working**. The output is an alert roughly every 3–6 h for a threshold breach that has not moved in two days — +which is exactly how a red signal stops meaning anything: while this row is red, a *new* Mnemosyne fault +arriving on the same job is indistinguishable from the standing one. + +## What the 59 conflicts actually are + +Live read of `~/.hermes/mnemosyne/data/mnemosyne.db` (2026-11-20 00:3x AWST): **223 conflicts total, 59 +unresolved.** Unresolved by creation day: + +| created | unresolved | +|---|---| +| 2026-10-31 | 11 | +| 2026-11-02 | 2 | +| 2026-11-03 | 12 | +| 2026-11-06 | 1 | +| 2026-11-07 | 2 | +| 2026-11-08 | 1 | +| 2026-11-12 | 13 | +| 2026-11-13 | 3 | +| **2026-11-18** | **14** | +| **total** | **59** | + +Two things stand out: + +1. **Most of the backlog is old.** 42 of the 59 predate 2026-11-14; the oldest batch is from **2026-10-31**, + which is the same batch recorded on [[systems/mnemosyne]] as *"all created in one batch at 2026-10-31 + 19:26:39 and sharing a single `fact_a_id`, the extraction-noise signature"*. The documented remediation for + this class is to bulk-resolve them as `false_positive` (see the `mnemosyne-operations` skill) — **not + applied here**. +2. **The threshold has been raised before.** The 2026-10-18 and 2026-11-01 reads on [[systems/mnemosyne]] both + record *"1 unresolved conflict (benign, threshold 10)"*. It now reads **50**. So the tripwire was already + loosened 5×, and the backlog has since grown past even that. **UNVERIFIED:** when the threshold was changed + from 10 to 50, and by whom; basis: two dated page reads vs the live alert text — the config line itself was + not opened in this window. + +## The 2026-11-18 batch coincides with the migration session + +The 14 conflicts created on **2026-11-18** are the newest batch, and 2026-11-18 is also the day of the only +interactive session in this window — `20261118_191808_5c9e08` (desktop, 9 messages, title *"Helping migrate +context AI assistant"*), whose turns were captured into `working_memory` at **19:25:03 AWST** and summarised +episodically at **19:55:20 AWST**. Consistent with the standing assessment that most conflicts in this bank are +LLM-extraction artefacts rather than real contradictions, but **UNVERIFIED:** that the 14 rows are extraction +noise from that session's turns specifically; basis: same-day coincidence plus the page's prior noise analysis — +the fact pairs were not inspected. + +## Not applied / open + +- **Nothing was resolved, reclassified or deleted.** No conflict row was touched; no threshold was changed. +- The watchdog's cadence, signature construction and alert path were left as-is. +- Open question: should a *standing* threshold breach alert on state change rather than on every slot? + Same family as the zero-item-consolidation streak problem on + [[incidents/2026-11-14-mnemosyne-capture-stall]] — both are conditions that re-report continuously because + the gate tests a level, not a change. + +→ [[systems/mnemosyne]], [[systems/scheduled-tasks]] diff --git a/incidents/2026-11-18-wiki-audit-credit-outage.md b/incidents/2026-11-18-wiki-audit-credit-outage.md new file mode 100644 index 0000000..5ebab87 --- /dev/null +++ b/incidents/2026-11-18-wiki-audit-credit-outage.md @@ -0,0 +1,89 @@ +--- +title: Wiki Audit failed two consecutive nights on a chained-provider credit/auth fault (2026-11-18, 2026-11-19) — 11-17 and 11-18 unaudited +type: incident +status: active +created: 2026-11-20 +updated: 2026-11-20 +verified_on: 2026-11-20 +review_after: 2026-11-21 +confidence: high +tags: [cron, wiki-audit, incident, model-routing, credentials, credits, router, silent-failure] +sources: [live-cron-executions-db, live-cron-incidents, live-jobs-json, live-deliveries-db, live-wiki-log] +--- + +# Wiki Audit failed two consecutive nights (2026-11-18, 2026-11-19) — 11-17 and 11-18 were never audited + +**Open.** `cbfe434a3b5e` (**Wiki Audit — Nightly**, schedule `0 0 * * *`) failed on **two consecutive slots**, +so the audit that should have reviewed **2026-11-17** and the audit that should have reviewed **2026-11-18** +never ran. The **2026-11-20 00:00** slot is the first to fire since; at the time of writing its execution row +reads `running` (started `2026-11-20T00:00:08`). + +This run therefore audits **three windows** — the standard last 24 h plus the two that were skipped. + +## Evidence (live probes 2026-11-20 00:1x–00:4x AWST) + +| slot | started | finished / last seen | duration | status | +|---|---|---|---|---| +| 2026-11-18 00:00 | `2026-11-18T00:00:08` | `2026-11-18T00:01:43` (incident `first_seen_at` `00:01:42.342149+08:00`) | ~1 m 35 s | `failed` | +| 2026-11-19 00:00 | `2026-11-19T00:00:16` | `2026-11-19T00:03:02` (incident `last_seen_at` `00:03:01.420319+08:00`) | ~2 m 46 s | `failed` | +| 2026-11-20 00:00 | `2026-11-20T00:00:08` | — | — | `running` | + +Both failures carry the **same 314-character `RuntimeError`** — a concatenation of every target the router tried: + +``` +RuntimeError: command-code/deepseek/deepseek-v4-flash: model — [400]: You have insufficient credits to +make this request. Please purchase more credits to continue using the service. (HTTP 400); +openai-compatible-chat-conn:52258193/deepseek-v4-flash-0731: auth — [401]: invalid auth token (HTTP 401); +openai-compati… +``` + +The tail is cut off at the 314-char cap, mid-way through a further `openai-compatible-…` target, so the +**final** target's verdict is not recoverable from this field. The shape matters more than the wording: this is +not one bad pin — the **whole fallback chain** was refused, first on **credit exhaustion** (`command-code`), +then on an **invalid auth token** (the second target), then at least one more target whose error is truncated away. + +- `cron/jobs.json`: `last_status: error`, `failure_streak: 2` — clean 0 at the 2026-11-17 00:27 run. +- `cron_incidents`: row **`cbfe43_fa12cca1ad4b`** — `state: detected`, `failure_type: auth`, + `first_seen_at 2026-11-18T00:01:42.342149+08:00`, `last_seen_at 2026-11-19T00:03:01.420319+08:00`, + **`alerted_at: None`**, `acked_at: None`, `closed_at: None`, output file + `/home/hermes/.hermes/cron/output/cbfe434a3b5e/2026-11-19_00-03-01.md`. +- `deliveries.db`, window since 2026-11-18: **24 rows — 23 `delivered` (9 of them `for_failure: 1`) and one + `failed`** (`2026-11-19 08:34:13`, a *content* delivery: `live adapter send failed: Timed out; live adapter + delivery to telegram`, consistent with *Gmail Action Digest* `08eb86f02d37` reading `delivery_failed`). + **No row is attributable to either failed audit slot** — both carry `delivery_outcome: suppressed` on the + execution row (as does the clean `2026-11-17` slot), i.e. this job's alerts are not wired to leave the box. + So the outage produced **no push to Anthony** and the red cron row was the only signal. + +## Why this matters beyond one job + +The wiki audit is this estate's own feedback loop: its output is the `log.md` entry and the page edits that keep +`current-state.md`, `systems/mnemosyne.md` and `systems/scheduled-tasks.md` honest. Two silent skips mean the wiki +stood still for two nights while the estate did not — including the window in which the **Mnemosyne watchdog went +red for 114 consecutive slots** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]. The gap is +visible in `index.md`: its newest feed entry was still **2026-11-16** on the morning of 11-20. + +## Precedent — and a repeating pattern + +Same class as [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] (**HTTP 429**, *"all targets were +skipped by pre-dispatch filters"*, 6 targets evaluated / **0 attempted**) and +[[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. Two observations worth carrying forward: + +1. **The 11-10 failure self-cleared on the next slot** — so a single red night for this job is not necessarily a + standing fault. This one did **not** self-clear: two slots, identical error, ~1.5 and ~2.7 minutes each. +2. **`alerted_at: NULL` on both rows.** The 11-10 row was also left unalerted. Two independent failures of this + job have now produced incident rows that never alerted, which points at the job's failure-*delivery* path + rather than at the alert throttle. + +## Not applied / open + +- **Nothing was changed**: the job's pin, provider and schedule are untouched, no credential was rotated, no + cron entry was edited. +- No re-run was forced; the 2026-11-20 00:00 slot was left to fire on schedule (it did, and its result is the + entry appended to `log.md` the same night). +- **UNVERIFIED:** whether the credit exhaustion is on the `command-code` account specifically or the router's + shared pool — the concatenated error does not name which target a *successful* dispatch would have used; + basis: the truncated 314-char error string. +- **UNVERIFIED:** whether this job's failures are configured `suppressed`, or whether its failure path is simply + not wired to delivery; basis: absence of delivery rows, not a read of the job's delivery config. + +→ [[systems/scheduled-tasks]], [[systems/omniroute]] diff --git a/index.md b/index.md index ffddd4e..04f33ae 100644 --- a/index.md +++ b/index.md @@ -3,8 +3,8 @@ title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 -updated: 2026-11-17 -verified_on: 2026-11-17 +updated: 2026-11-20 +verified_on: 2026-11-20 confidence: high tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout] sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake] @@ -12,7 +12,12 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. last updated: 2026-11-18 +> model routing and automation. last updated: 2026-11-20 +> +> **2026-11-20 (three-window audit; live probes 2026-11-20 00:0x – 00:5x AWST) — the memory freeze broke by itself, the alarm did not, and the audits that would have said so had died twice.** +> **The capture stall is over — silently.** `working_memory` **1130 → 1131** / `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST**: **105 h 52 m**, ended with **no intervention**. And the "capture is dead" reading was wrong — `state.db` holds only **three** non-cron sessions in that window (two `desktop` on 11-14, both *before* the 09:33 boundary, then the 11-18 **19:18** session whose turn is the captured row), so **every interactive session was captured**; the alert measures **conversational silence** (age of the newest non-`cron`/`subagent` message). Corrected on [[incidents/2026-11-14-mnemosyne-capture-stall]] and [[systems/mnemosyne]]. +> **Two of the three jobs that guard the estate are themselves red — and nobody was told.** The **Wiki Audit failed on 11-18 and 11-19** (chained-provider credit/auth), so **neither night was ever audited** (this run covers three) → [[incidents/2026-11-18-wiki-audit-credit-outage]]; both slots `delivery_outcome: suppressed` and its incident row still reads **`alerted_at: None`**. The **Mnemosyne Watchdog has failed 115 consecutive 15-minute slots** since **11-18 20:01:06** — one durable cause, **59 unresolved conflicts against a threshold of 50** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]. Ledger **11-19: 176 rows — 78 `completed` / 98 `failed`** (worst day in the retained window); 11-17 173 (171/2), 11-18 172 (153/19). Deliveries since 11-18 **24 — 23 `delivered` / 1 `failed`** (content, Telegram timeout `2026-11-19 08:34:13`). +> Fleet flat at **29 jobs (26 enabled / 3 disabled)** — a twelfth consecutive window — but non-clean `last_status` **1 → 4**; unclosed `cron_incidents` **5 → 4**. The 03:00 stocktake kept working throughout (11 m 35 s / 23 m 07 s / 24 m 38 s), and the consolidator is still consolidating **0 items — 199 of the last 200 cycles**. > > **2026-11-16 (nightly audit; live probes 2026-11-16 23:59 – 2026-11-17 00:27 AWST) — memory dead at 62 h 54 m, and a correction to the 11-15 record.** > The capture stall is unchanged in every respect but the counters: **1130** / **1249** rows, newest write `2026-11-14 09:32:59` AWST, **five** unclosed `alerted` incidents, monitor `failure_streak` 5, consolidator zero-item streak **120** cycles. → [[incidents/2026-11-14-mnemosyne-capture-stall]] @@ -66,7 +71,7 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi - [[systems/guanaco]] — Guanaco LLM proxy on CT205 (Ollama stack) - [[systems/model-providers]] — LiteLLM + OmniRoute providers, routing setup - [[systems/memory-backend]] — Mnemosyne vector memory, settings -- [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence +- [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence. **As of 2026-11-20: capture broke its 105 h 52 m freeze at 2026-11-18 19:25 AWST with no intervention, and every interactive session in the window was captured — the "stale pipeline" alert measures conversational silence, not a dead hook** (the gate reads the age of the newest non-`cron`/`subagent` message); monitor still failing on that metric (`failure_streak: 2`), consolidator **199 of the last 200 cycles zero-item**, unclosed `cron_incidents` **4**; the watchdog's own fault is **59 unresolved conflicts vs its threshold of 50**. - [[systems/phonon-tts]] — **Phonon TTS stack on CT200** (`:8881` OpenAI-compatible API, `:8882` WASM demo, `:8883` WebSocket; added 2026-11-14) - [[systems/omniroute]] — OmniRoute catalog, combos, and host path - [[systems/openrouter]] — OpenRouter current state and history @@ -84,11 +89,13 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26** (peer offline **36 d**, `tx 312 / rx 0`; host answers neither `:8065` nor ICMP). **268** connect timeouts on 10-25 vs 264 on 10-24; the reconnect **attempt counter reset to 54** at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost. - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. - [[systems/hermes-hq-push]] — **Hermes HQ iPhone app plugin** (standalone, **v0.9.42**, `kind: standalone`, author *Hermes HQ*; tree mtime **2026-11-08 02:19**). `hermes plugins list` → **enabled** (state `user`); `config.yaml` → `plugins.enabled` (line **13226** as of 2026-11-13; the number drifts with each catalogue write). ⚠ **2026-11-12: the upgrade past 0.9.42 is blocked by the pre-install security scan** (dangerous verdict, one critical in the plugin's own tile skill) — see [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]]. Host side of the Hermes HQ app: Apple push (finished turns, lock-screen approvals, "waiting for a password/login/code" alerts, **live Lock Screen cards** sealed to the phone), phone **device access** (contacts, location, reminders, alarms, photos, Health, motion, music), the desktop-app **Hermes HQ theme**, the app's **Terminal page**, fast **Sessions list** on a busy host, cross-posting answers back to Telegram/Discord, a **restart guard**, and **Grok Live** voice-token minting (`XAI_API_KEY` or subscription sign-in; bearer never reaches the phone). Tools `device_contacts_search` / `device_request_result`; 7 hooks. Its manifest states it **replaces `dispatch-push`** — and the replacement is clean: **no `dispatch-push` reference survives in `config.yaml`** and no such directory remains. **Different stack** from the Cadu/Vory bridge on [[systems/push-notifications]]. ⚠ Its `pre_tool_call` guard also **blocks any tool call whose arguments contain the literal text of a launchd one-off-restart command** — describe it, never quote it. -- [[systems/scheduled-tasks]] — **As of 2026-11-14 (window 11-14): 167 rows / 166 `completed` / 0 `failed` / 1 `running` (this audit); deliveries 4 / 4 `delivered`; `cron_incidents` 0 → 1 unclosed (138 total) — the Mnemosyne Health Monitor wrote the estate's first red row since 11-13 (`last_status: error`, streak 1); fleet 29 (26 / 3).** Earlier — **As of 2026-11-12 (window 11-12): 176 rows / 175 `completed` / **0 `failed`** / 1 `running` (this audit); deliveries 9 / 9 `delivered`; unclosed incident rows **0 of 137** — the 429 row closed `resolved` at 2026-11-12T00:25:36; fleet 29 (26 / 3).** Earlier — **As of 2026-11-09 (window 11-09): 174 rows / 173 `completed` / **0 `failed`** — the first `failed`-free day since 11-05; deliveries 8/8 `delivered`; unclosed incident rows 3 → 0; the stocktake closed `completed` after 9 h 10 m with **zero edits landed**; Job Radar Stage 1 overran 9 h 09 m and landed nothing (new incident).** Earlier: **As of 2026-10-31 00:3x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 4 → 3; failing 1 → 0; unclosed incident rows 2 → 1; 2026-10-30 closed 138 / 0 after a 5 h 30 m CT460 host outage (PVE reboot, ≈39 slots lost, 8 catch-up runs at 16:37:21), and `ea79a2107e36` is ENABLED again (`last_status: ok`, streak 0) after its 00:01:08 slot closed `completed` at 10:02:48; deliveries 8/8 `delivered`.** **Prior window — as of 2026-10-30 03:3x:** 29 jobs (25 enabled, 4 disabled); non-clean `last_status` 3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries all `delivered`. ⚠ `ea79a2107e36` (Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closed `unknown` on a gateway-connection failure — and its failure notice was NEVER DELIVERED (`deliveries.db` has zero rows for 2026-10-30), so the incident row is the only trace → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. Every other previously-red job has cleared to `ok`/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 5 → 3 (all three are stale re-runs of old faults and every `failure_streak` read 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all `delivered`. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with `missing_cover_letter_count 0` (Notion 455 → 475). Both stage-1 incident rows closed `resolved` at 10:16:12; the lone survivor is the Facebook monthly's `0e753e_e7f8194a7d06`.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** +- [[systems/scheduled-tasks]] — **As of 2026-11-20 (three-window audit 11-17 / 11-18 / 11-19, because both prior audits died): fleet 29 (26 / 3), a twelfth consecutive window at the same count; non-clean `last_status` 1 → 4 (Wiki Audit `error` 2, Watchdog `error` 115, Health Monitor `error` 2, Gmail Action Digest `delivery_failed`); ledger 11-17 173 (171/2), 11-18 172 (153/19), 11-19 176 (78/98); deliveries since 11-18 24 (23 `delivered` / 1 `failed` — a content Telegram timeout 11-19 08:34:13); `cron_incidents` unclosed 5 → 4.** Earlier — **As of 2026-11-14 (window 11-14): 167 rows / 166 `completed` / 0 `failed` / 1 `running` (this audit); deliveries 4 / 4 `delivered`; `cron_incidents` 0 → 1 unclosed (138 total) — the Mnemosyne Health Monitor wrote the estate's first red row since 11-13 (`last_status: error`, streak 1); fleet 29 (26 / 3).** Earlier — **As of 2026-11-12 (window 11-12): 176 rows / 175 `completed` / **0 `failed`** / 1 `running` (this audit); deliveries 9 / 9 `delivered`; unclosed incident rows **0 of 137** — the 429 row closed `resolved` at 2026-11-12T00:25:36; fleet 29 (26 / 3).** Earlier — **As of 2026-11-09 (window 11-09): 174 rows / 173 `completed` / **0 `failed`** — the first `failed`-free day since 11-05; deliveries 8/8 `delivered`; unclosed incident rows 3 → 0; the stocktake closed `completed` after 9 h 10 m with **zero edits landed**; Job Radar Stage 1 overran 9 h 09 m and landed nothing (new incident).** Earlier: **As of 2026-10-31 00:3x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 4 → 3; failing 1 → 0; unclosed incident rows 2 → 1; 2026-10-30 closed 138 / 0 after a 5 h 30 m CT460 host outage (PVE reboot, ≈39 slots lost, 8 catch-up runs at 16:37:21), and `ea79a2107e36` is ENABLED again (`last_status: ok`, streak 0) after its 00:01:08 slot closed `completed` at 10:02:48; deliveries 8/8 `delivered`.** **Prior window — as of 2026-10-30 03:3x:** 29 jobs (25 enabled, 4 disabled); non-clean `last_status` 3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries all `delivered`. ⚠ `ea79a2107e36` (Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closed `unknown` on a gateway-connection failure — and its failure notice was NEVER DELIVERED (`deliveries.db` has zero rows for 2026-10-30), so the incident row is the only trace → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. Every other previously-red job has cleared to `ok`/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 5 → 3 (all three are stale re-runs of old faults and every `failure_streak` read 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all `delivered`. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with `missing_cover_letter_count 0` (Notion 455 → 475). Both stage-1 incident rows closed `resolved` at 10:16:12; the lone survivor is the Facebook monthly's `0e753e_e7f8194a7d06`.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** - [[systems/skills-index]] — Installed skills inventory (**300 skills** (**67 categories**) as of the **2026-11-13** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents -- [[incidents/2026-11-14-mnemosyne-capture-stall]] - ⚠ **OPEN — Mnemosyne has captured nothing since 2026-11-14 09:32:59 AWST (~15 h at this read), and the health monitor took 14 h 31 m to say so.** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); the `pre_llm_call` capture hook logs `Mnemosyne capture ok` on every desktop turn while writing **0** rows, so the hook's own success is not evidence of capture. The monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault; this job's twelve earlier rows read `agent` / `timeout` / `unknown` — state **`alerted`**, unclosed), and its alert text prints the raw **UTC** instant, so it reads as a 23 h gap when the true AWST gap was 14 h 31 m. ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 AWST run saw nothing** (`capture delay 29.4 min`, exit 0, `suppressed`) — the `-589.5 min` ℹ️ line belongs to the **00:03:35** run itself (monitor log lines 957 / 1030 / 1046). Detection is **activity-aware** (`mnemosyne_freshness_gate.py` fires on `delay < −120 min`, i.e. on **magnitude**) and the first message to postdate the stall arrived at **19:22 AWST**, after that slot — so the ~12 h of quiet was the `0 */12` **cadence**, not a threshold bug. **What went unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, visible as `(0 items)` in every run but tested only for age (36 h). Consequences: ~15 h of desktop / Telegram / cron context uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established** → [[systems/mnemosyne]], [[systems/phonon-tts]]. +- [[incidents/2026-11-18-wiki-audit-credit-outage]] - **OPEN — the nightly audit failed on two consecutive nights, so 2026-11-17 and 2026-11-18 were never audited, and no failure notice was ever sent.** `cbfe434a3b5e` (Wiki Audit) ran **2026-11-18T00:00:08 → 00:01:43** and **2026-11-19T00:00:16 → 00:03:02**, both closing `failed` on a chained-provider **credit/auth** fault (`failure_streak: 2`, last clean run 11-17). Both slots carry **`delivery_outcome: suppressed`**, so the estate's own auditor was the one thing nobody heard about; incident row `cbfe43_fa12cca1ad4b` (`failure_type: auth`, opened 2026-11-18 00:01:42) is still `alerted` with **`alerted_at: None`** — the same silent-skip shape as [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]. The 2026-11-20 00:00 slot (this run) covers all three windows. +- [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] - **OPEN — 115 consecutive watchdog failures from one cause: the conflict queue.** `96eb95ce12f2` (Mnemosyne Watchdog) has failed **every** 15-minute slot since **2026-11-18 20:01:06**, reaching **`failure_streak: 115`** at **2026-11-20 00:31:16** (`last_status: error`), because **59 unresolved conflicts sit above its threshold of 50** — a single durable trip, not 115 faults. Each slot delivers a *failure notice* (9 of the 24 deliveries since 11-18 are `for_failure: 1`), so the noisy channel is also the only channel; every slot re-opens incident `96eb95_98ade422d195` and nothing escalates or closes. +- [[incidents/2026-11-14-mnemosyne-capture-stall]] - ⚠ **OPEN (corrected 2026-11-20) — the freeze ended by itself at 2026-11-18 19:25 AWST after 105 h 52 m, and the "captured nothing" reading was a quiet estate, not a broken hook.** Earlier headline: **Mnemosyne has captured nothing since 2026-11-14 09:32:59 AWST (~15 h at this read), and the health monitor took 14 h 31 m to say so.** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); the `pre_llm_call` capture hook logs `Mnemosyne capture ok` on every desktop turn while writing **0** rows, so the hook's own success is not evidence of capture. The monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault; this job's twelve earlier rows read `agent` / `timeout` / `unknown` — state **`alerted`**, unclosed), and its alert text prints the raw **UTC** instant, so it reads as a 23 h gap when the true AWST gap was 14 h 31 m. ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 AWST run saw nothing** (`capture delay 29.4 min`, exit 0, `suppressed`) — the `-589.5 min` ℹ️ line belongs to the **00:03:35** run itself (monitor log lines 957 / 1030 / 1046). Detection is **activity-aware** (`mnemosyne_freshness_gate.py` fires on `delay < −120 min`, i.e. on **magnitude**) and the first message to postdate the stall arrived at **19:22 AWST**, after that slot — so the ~12 h of quiet was the `0 */12` **cadence**, not a threshold bug. **What went unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, visible as `(0 items)` in every run but tested only for age (36 h). Consequences: ~15 h of desktop / Telegram / cron context uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established** → [[systems/mnemosyne]], [[systems/phonon-tts]]. - [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]] - **OPEN — the `hermes-hq-push` upgrade past v0.9.42 was refused by the pre-install plugin security scan.** `hermes plugins install mrcharlesiv/hermes-hq-plugins/hermes-hq-push --enable --force` returned **"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."** The sole critical is the plugin's **own** `skills/dashboard-tile/SKILL.md:41` shell one-liner (`echo '' | python3 …` = *obfuscation*); the other 38 findings are low/medium and alone would have installed as `caution`. **`--force` lifts `caution` only**; a `dangerous` verdict is an unconditional block (`tools/plugin_guard.py:406-418`), and **no per-verdict override exists** — the only key is `plugins.scan_on_install` (default `true`, currently unset), which disables scanning for *every* future install; the `trust_level`/allowlist machinery lives in `tools/skills_guard.py` (the skills path) and `plugin_guard.py` hardcodes `trust_level="community"`. **Installed state unchanged:** v0.9.42, `enabled`, still delivering. **Fix = PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc` → `main`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file, created 2026-11-12T10:00:01Z) — pushed from a fork because this host's `gh` account has **read-only** access upstream; local clone `/home/hermes/workspace/hq-plugins-fix` (`90c9d28`). - [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **RESOLVED 2026-11-12T00:25:36** (self-cleared on the next slot's clean completion; *cron_incidents* now 0 unclosed of 137) **— the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and is **still open with `alerted_at: NULL` — the estate's only open incident row**. Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]). - [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - ⚠ **REOPENED 2026-11-14 — the hook was the fault, and the 30 → 120 s fix did not hold at current load.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** ⚠ **Recurred 2026-11-14 00:05:40** — **1 timeout + 10 skips**, overrunning the **120 s** budget at `load average 12.08` on 2 vCPU (**~2×** the 5.89 at diagnosis), with `state.db` grown to **2.04 GB**. Status **`resolved` → `active`**; the lever is now urgent. → [[systems/conduit]], [[systems/hermes-agent]] diff --git a/log.md b/log.md index 025f2d9..8cc5162 100644 --- a/log.md +++ b/log.md @@ -1864,3 +1864,14 @@ recorded here so the next audit does not re-litigate it. - `incidents/2026-10-26-planned-restart-orphaned-cron-run.md` — both `ea79a2_*` incident rows closed/resolved (c3ab2d784ba3 10-27, a4661554a3c2 closed 11-07 15:43), error string absent from retained executions window → status → resolved; drain-policy follow-ups 1/3/4 remain open design questions. - `reports/communities-wa-social-media-audit-2026.md` — live wa.gov.au check: remit verbatim-identical, five Ministers unchanged, site updated 18 Nov 2026 with fresh announcements. - Quartz rebuilt + deployed via second `wiki-maintain.sh` run (pattern 13). + +## 2026-11-20 00:5x — nightly audit (three windows: 2026-11-17, 11-18 and 11-19 AWST) + +**Why three windows:** the audit's own slots at **2026-11-18T00:00:08 → 00:01:43** and **2026-11-19T00:00:16 → 00:03:02** both closed `failed`, so **2026-11-17 and 2026-11-18 were never audited**. This run covers all three. + +- **New page — [[incidents/2026-11-18-wiki-audit-credit-outage]]:** `cbfe434a3b5e` (Wiki Audit — Nightly) is at `failure_streak: 2`, both failures on a chained-provider **credit/auth** fault (last clean run 11-17). Both failed slots carry `delivery_outcome: suppressed`, so **no failure notice ever reached Anthony** — the estate's own auditor was the one thing nobody heard about. Its incident row `cbfe43_fa12cca1ad4b` (`failure_type: auth`, opened 2026-11-18 00:01:42) is still `alerted` with **`alerted_at: None`**, the same silent-skip shape as the 11-10 429 incident. +- **New page — [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]:** `96eb95ce12f2` (Mnemosyne Watchdog) has failed **every** 15-minute slot since **2026-11-18 20:01:06**, reaching **115 consecutive failures** at **2026-11-20 00:31:16** (`failure_streak: 115`, `last_status: error`), for one reason: **59 unresolved conflicts against its threshold of 50**. A single durable trip, not 115 faults — but each slot delivers a fresh *failure notice* (9 of the 24 deliveries since 11-18 carry `for_failure: 1`), so the noisy channel is the estate's only channel, and every slot re-opens incident `96eb95_98ade422d195`. +- **Corrected — [[incidents/2026-11-14-mnemosyne-capture-stall]] and [[systems/mnemosyne]]:** the freeze is **over**. `working_memory` **1130 → 1131**, `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST** → the stall ran **105 h 52 m** and ended **with no intervention** (no config edit, restart or repair on the capture path). ⚠ **And the "capture is dead" reading was wrong:** `state.db` lists only **three** non-cron sessions since 11-14 00:00 — two `desktop` on **11-14 08:35 / 08:52** (both *before* the 09:33 boundary) and the **11-18 19:18** `desktop` whose turn *is* the captured row — so **every interactive session in the window was captured**. `mnemosyne_freshness_gate.py` measures the age of the newest non-`cron`/`subagent` message, so **silence and failure read identically**; that ambiguity is the real defect, and the monitor is still failing on it (`failure_streak: 2`; 11-19 00:01 was the only clean slot). Title and body reframed accordingly. +- **[[systems/scheduled-tasks]] — new Count bullet (11-17 / 11-18 / 11-19):** ledger **173 rows (171 `completed` / 2 `failed`)** / **172 (153/19)** / **176 (78/98)** — the last is the worst day in the retained window; 11-20 so far **8 (3 / 4 / 1 `running`)**. Deliveries since 11-18: **24 — 23 `delivered`** (9 `for_failure: 1`) / **1 `failed`** (a *content* delivery, `live adapter send failed: Timed out`, `2026-11-19 08:34:13`). Fleet **29 jobs (26 enabled / 3 disabled)** — a **twelfth consecutive window** at the same count (11-18 and 11-19 went unrecorded, see above) — but non-clean `last_status` **1 → 4**: `cbfe434a3b5e` `error` (2), `96eb95ce12f2` `error` (115), `ea79a2107e36` `error` (2), `08eb86f02d37` `delivery_failed` (0). `cron_incidents` unclosed **5 → 4**. The 03:00 stocktake (`1a8e90ea2271`) ran `completed` on all three days — **11 m 35 s / 23 m 07 s / 24 m 38 s** — so the estate's structural maintenance kept running while both alarm paths were red. +- **[[current-state]]** — new Recent Changes section for the 2026-11-17 → 2026-11-20 window. Frontmatter `updated` / `verified_on` bumped to **2026-11-20** across `index.md`, `current-state.md`, `systems/mnemosyne.md`, `systems/scheduled-tasks.md` and the two touched incident pages. +- **Mnemosyne consolidation is still zero-item:** newest `consolidation_log` row **`2026-11-19T16:20:14+00:00`** (= 2026-11-20 00:20 AWST, ~31 min cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle — timestamp updated for watchdog"`; **199 of the last 200 cycles are zero-item** (was **120** on the 11-17 read). The zero-item-streak threshold remains **proposed, not applied**. diff --git a/systems/mnemosyne.md b/systems/mnemosyne.md index 61f507a..30bb087 100644 --- a/systems/mnemosyne.md +++ b/systems/mnemosyne.md @@ -3,9 +3,9 @@ title: Mnemosyne type: system status: active created: 2026-07-22 -updated: 2026-11-17 -verified_on: 2026-11-17 -review_after: 2026-11-18 +updated: 2026-11-20 +verified_on: 2026-11-20 +review_after: 2026-11-21 confidence: high tags: [memory, mnemosyne, backend] sources: [live-probe, mnemosyne-db, config, ops-profile-config, live-secrets-scan, live-state-db, live-health-monitor-log] @@ -50,7 +50,7 @@ Primary agent memory store for Hermes. ## Known Issues -**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed; its `failure_type: auth` is the scheduler's classifier, not evidence of a credential fault). **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 slot saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, delivery `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run itself** (monitor log: header line 957 = 12:01:52 run, header line 1030 = 00:03:35 run, `-589.5` at line 1046, ❌ at 1047). `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`, default **120** min) but **activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last captured message was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min) and the first to postdate the stall was the **19:22 AWST desktop** turn — after the 12:00 slot. **The quiet was the `0 */12` cadence, not a sign-vs-magnitude bug.** **What did go unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, printed as `(0 items)` on every run yet tested only for age (`> 36 h`) — the candidate fix is a zero-item-streak threshold. ⚠ Related trap, verified: `consolidation_log.created_at` mixes formats (recent rows `+00:00`, older rows naive **AWST**), so treating the column as one axis misplaces events by 8 h. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). **Update 2026-11-16 00:18 (next audit): the freeze is total, not degraded** — `working_memory` **1130** / `episodic_memory` **1249**, *identical at every read of the stall*, so **0 writes in 38 h 46 m**; newest rows are still the `[MODEL_REFRESH_PROPOSAL]` maintenance writes at `09:32:58`/`09:32:59` and the newest *user-turn* capture is `2026-11-14 09:04:19`. The monitor failed twice more (`12:03:14`, `2026-11-16 00:03:15`) → `cron_incidents` **1 → 3 unclosed** (141 rows total), each a single observation (`first_seen_at == last_seen_at`) because **the signature embeds the elapsed-hours float** (`4.64157453` / `4.634034074444445` / `4.643481075555555`), which is never equal twice — one durable fault, a fresh row every slot, nothing ever escalates. ⚠ **And the `≈4.64 h` in the alert is not a stall duration:** `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message whose session source is not `cron`/`subagent` — the **Pulse/`desktop`** turns, ~6 h cadence — so **the alarm has understated severity on every fire** (mapping table on the incident page). The consolidator's `summary_preview` now reads `No-op consolidation cycle — timestamp updated for watchdog`: it **bumps its own timestamp to satisfy an age test**, so a zero-item streak is invisible to an age-only gate; streak now **~74 cycles** from `2026-11-14T02:04:42+00:00`. **Proposed, none applied:** a zero-item-streak threshold, and normalising the value out of the incident signature. **Update 2026-11-17 00:27 (third read):** nothing has moved — `working_memory` **1130** / `episodic_memory` **1249**, newest write `2026-11-14 09:32:59` AWST, so the stall is **62 h 54 m**; unclosed `alerted` incidents **5** (143 rows total), consolidator zero-item streak **120** cycles, monitor `failure_streak: 5`. → [[incidents/2026-11-14-mnemosyne-capture-stall]] +**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed; its `failure_type: auth` is the scheduler's classifier, not evidence of a credential fault). **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 slot saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, delivery `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run itself** (monitor log: header line 957 = 12:01:52 run, header line 1030 = 00:03:35 run, `-589.5` at line 1046, ❌ at 1047). `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`, default **120** min) but **activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last captured message was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min) and the first to postdate the stall was the **19:22 AWST desktop** turn — after the 12:00 slot. **The quiet was the `0 */12` cadence, not a sign-vs-magnitude bug.** **What did go unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, printed as `(0 items)` on every run yet tested only for age (`> 36 h`) — the candidate fix is a zero-item-streak threshold. ⚠ Related trap, verified: `consolidation_log.created_at` mixes formats (recent rows `+00:00`, older rows naive **AWST**), so treating the column as one axis misplaces events by 8 h. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). **Update 2026-11-16 00:18 (next audit): the freeze is total, not degraded** — `working_memory` **1130** / `episodic_memory` **1249**, *identical at every read of the stall*, so **0 writes in 38 h 46 m**; newest rows are still the `[MODEL_REFRESH_PROPOSAL]` maintenance writes at `09:32:58`/`09:32:59` and the newest *user-turn* capture is `2026-11-14 09:04:19`. The monitor failed twice more (`12:03:14`, `2026-11-16 00:03:15`) → `cron_incidents` **1 → 3 unclosed** (141 rows total), each a single observation (`first_seen_at == last_seen_at`) because **the signature embeds the elapsed-hours float** (`4.64157453` / `4.634034074444445` / `4.643481075555555`), which is never equal twice — one durable fault, a fresh row every slot, nothing ever escalates. ⚠ **And the `≈4.64 h` in the alert is not a stall duration:** `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message whose session source is not `cron`/`subagent` — the **Pulse/`desktop`** turns, ~6 h cadence — so **the alarm has understated severity on every fire** (mapping table on the incident page). The consolidator's `summary_preview` now reads `No-op consolidation cycle — timestamp updated for watchdog`: it **bumps its own timestamp to satisfy an age test**, so a zero-item streak is invisible to an age-only gate; streak now **~74 cycles** from `2026-11-14T02:04:42+00:00`. **Proposed, none applied:** a zero-item-streak threshold, and normalising the value out of the incident signature. **Update 2026-11-17 00:27 (third read):** nothing has moved — `working_memory` **1130** / `episodic_memory` **1249**, newest write `2026-11-14 09:32:59` AWST, so the stall is **62 h 54 m**; unclosed `alerted` incidents **5** (143 rows total), consolidator zero-item streak **120** cycles, monitor `failure_streak: 5`. **Update 2026-11-20 00:5x (fourth read) — the freeze ended by itself after 105 h 52 m, and the "capture dead" reading was wrong.** `working_memory` **1130 → 1131**, `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST** (stored UTC `11:25:03`) — the row is `session_id: hermes_20261118_191808_5c9e08`, content beginning `[USER] You are helping me migrate contex…`. Stall ran **105 h 52 m** and ended with **no intervention**. ⚠ Correction: `state.db` lists only **three** non-cron sessions since 11-14 — two `desktop` on 11-14 (**08:35**/40 msgs, **08:52**/41 msgs, both *before* the 09:33 boundary) and the 11-18 **19:18** `desktop` (9 msgs, *"Helping migrate context AI assistant"*) whose captured turn ends the freeze — so **every interactive session in the window was captured**, and the "stale pipeline" text was measuring **conversational silence** (the gate reads the age of the newest non-`cron`/`subagent` message). **Still red on that same metric:** monitor slots 11-18 00:01 and 12:01 failed, **11-19 00:01 `completed`**, 11-19 12:01 and 11-20 00:01 failed — all `STALE MEMORY PIPELINE`, `failure_streak: 2`, incidents `ea79a2_f2be2db3bfce` (11-19 12:01:58) and `ea79a2_aae4d914f3c9` (11-20 00:03:36), both `alerted`; their `failure_type: auth` is the scheduler's classifier, not a credential fault. Unclosed `cron_incidents` **5 → 4** (adds the audit's `cbfe43_fa12cca1ad4b`, *auth*, `alerted_at: None`). Consolidator newest row **`2026-11-19T16:20:14+00:00`** (running on ~31 min cadence), `items_consolidated: 0`, **199 of the last 200 cycles zero-item** (was 120 at the 11-17 read). Zero-item-streak threshold still **proposed, not applied**. The watchdog's separate fault is the conflict queue — **59 unresolved vs a threshold of 50** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]. → [[incidents/2026-11-14-mnemosyne-capture-stall]] **⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**. diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 51cc64b..5f17858 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,9 +3,9 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-11-17 -verified_on: 2026-11-17 -review_after: 2026-11-18 +updated: 2026-11-20 +verified_on: 2026-11-20 +review_after: 2026-11-21 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db] @@ -14,6 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db # Scheduled Tasks ## Count +- **Current as of 2026-11-20 00:5x (three-window audit — 2026-11-17, 11-18 and 11-19; the audits slotted at 11-18 00:00 and 11-19 00:00 both died, so 11-17 and 11-18 had never been audited; live `jobs.json` + `executions.db` + `deliveries.db` + `cron_incidents` + `mnemosyne.db`).** Ledger: **11-17 — 173 rows, 171 `completed` / 2 `failed`** (both `ea79a2107e36` *Mnemosyne Health Monitor*, `00:01` and `12:01`); **11-18 — 172 rows, 153 / 19** (this audit `00:00:08 → 00:01:43`, the monitor twice, and **16** *Mnemosyne Watchdog* slots from `20:01:06`); **11-19 — 176 rows, 78 / 98** — **the worst day in the retained window** (audit `00:00:16 → 00:03:02`, monitor `12:01`, **96** watchdog slots); **11-20 so far — 8 rows, 3 `completed` / 4 `failed` / 1 `running`** (this audit). Deliveries since 11-18: **24 — 23 `delivered` (9 `for_failure: 1`) / 1 `failed`** (`2026-11-19 08:34:13`, a *content* delivery: `live adapter send failed: Timed out; live adapter delivery to telegram`). Fleet **29 jobs (26 enabled / 3 disabled)** — a **twelfth consecutive window at the same count** (following the 11th, recorded on 11-17; the 11-18 and 11-19 windows went unrecorded because both audits died) — but non-clean `last_status` **1 → 4**: `cbfe434a3b5e` Wiki Audit `error`, `failure_streak` **2** (last clean run 11-17), `96eb95ce12f2` Watchdog `error`, streak **115**, `ea79a2107e36` Health Monitor `error`, streak **2**, `08eb86f02d37` Gmail Action Digest `delivery_failed`, streak **0**. **Two of the three jobs that keep the fleet safe are themselves red — and the 03:00 stocktake is not:** `1a8e90ea2271` ran `completed` on 11-17 (**11 m 35 s**), 11-18 (**23 m 07 s**) and 11-19 (**24 m 38 s**), so the estate's structural maintenance kept running while both alarm paths failed. `cron_incidents` unclosed **5 → 4**, none closed automatically: `cbfe43_fa12cca1ad4b` (*auth*, 2026-11-18 00:01:42, **`alerted_at: None`** — the audit's silent skip), `96eb95_98ade422d195` (*script*, 2026-11-18 20:01:15 → last_seen 2026-11-20 00:31:16), `ea79a2_f2be2db3bfce` (2026-11-19 12:01:58) and `ea79a2_aae4d914f3c9` (2026-11-20 00:03:36) — both monitor rows on the same `STALE MEMORY PIPELINE` metric that is now measuring a *quiet estate* rather than a broken capture (see [[systems/mnemosyne]]). Cadence and pins unchanged; this audit still runs `custom:omni-101` / `deepseek-paid-smart`, and the Mnemosyne Watchdog's model/provider fields remain unset (inherits global). Full detail: [[incidents/2026-11-18-wiki-audit-credit-outage]], [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]. - **Current as of 2026-11-17 00:27 (window 2026-11-16 00:00 -> 23:59; nightly audit).** Still **29 jobs** (26 enabled, 3 disabled) — 11th consecutive window unchanged. 11-16 ledger: **175 rows — 173 `completed`, 2 `failed`**; both failures are `ea79a2107e36` *Mnemosyne Health Monitor*