wiki audit 2026-11-20: three-window catch-up (11-17/11-18/11-19, both prior audit slots died)

- new: incidents/2026-11-18-wiki-audit-credit-outage (audit failed 11-18 + 11-19, deliveries suppressed, incident row alerted_at=None)
- new: incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm (115 consecutive slots, 59 unresolved conflicts vs threshold 50)
- corrected: incidents/2026-11-14-mnemosyne-capture-stall + systems/mnemosyne — freeze ENDED 2026-11-18 19:25 AWST (105h52m, no intervention); 'capture dead' reading was conversational silence (every interactive session captured)
- systems/scheduled-tasks: new Count bullet 11-17 173(171/2) / 11-18 172(153/19) / 11-19 176(78/98); fleet 29 (26/3); non-clean last_status 1->4; unclosed incidents 5->4; deliveries 24 (23 delivered / 1 failed)
- index.md feed + incidents + systems lines; log.md dated entry; current-state.md new section; frontmatter dates -> 2026-11-20
This commit is contained in:
Tony0410
2026-11-20 01:02:44 +08:00
parent 16f2c878c0
commit 226767f3ec
8 changed files with 310 additions and 22 deletions
+17 -4
View File
@@ -3,16 +3,29 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-11-18
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
confidence: high
tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
---
# Current State
Updated: 2026-11-17
Updated: 2026-11-20
## Recent Changes (2026-11-17 → 2026-11-20, three-window audit; live probes 2026-11-20 00:0x – 00:5x AWST)
**The estate's memory freeze ended by itself, its two alarm paths are still red, and the audits that would have reported either had silently died twice.**
- **Capture stall: over, unassisted.** `working_memory` **1130 → 1131**, `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST** — a **105 h 52 m** break that ended with no config change, restart or repair. ⚠ **And the "capture dead" reading was wrong:** `state.db` holds only **three** non-cron sessions since 11-14 00:00 — two `desktop` on **11-14 08:35 / 08:52** (both *before* the 09:33 boundary) and the **11-18 19:18** `desktop` whose turn *is* the captured row — so **every interactive session in the window was captured**. `mnemosyne_freshness_gate.py` reads the age of the newest non-`cron`/`subagent` message, so **silence and failure read identically**; that ambiguity is the real defect. The monitor is still failing on it (`failure_streak: 2`; slots 11-18 00:01 and 12:01, 11-19 12:01, 11-20 00:01 — 11-19 00:01 alone passed). → [[incidents/2026-11-14-mnemosyne-capture-stall]], [[systems/mnemosyne]]
- **The nightly audit missed two nights.** `cbfe434a3b5e` failed on **11-18** and **11-19** on a chained-provider credit/auth fault; **both deliveries `suppressed`**, and its incident row is still `alerted` with **`alerted_at: None`** → [[incidents/2026-11-18-wiki-audit-credit-outage]]. **This run covers 11-17, 11-18 and 11-19.**
- **The watchdog is in a threshold storm.** `96eb95ce12f2` failed **115 consecutive** 15-minute slots from **2026-11-18 20:01:06** to **2026-11-20 00:31:16** on **59 unresolved conflicts vs a threshold of 50** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]
- **Cron ledger.** 11-17 **173 rows (171 `completed` / 2 `failed`)**; 11-18 **172 (153/19)**; 11-19 **176 (78/98)** — the worst day in the retained window; 11-20 so far **8 (3 / 4 / 1 `running`)**. Deliveries since 11-18 **24 — 23 `delivered`** (9 `for_failure: 1`) / **1 `failed`** (content, Telegram timeout `2026-11-19 08:34:13`). Fleet **29 (26 enabled / 3 disabled)**, a twelfth consecutive window; non-clean `last_status` **1 → 4**; unclosed `cron_incidents` **5 → 4**. The **03:00 stocktake ran `completed` on all three days** (11 m 35 s / 23 m 07 s / 24 m 38 s).
- **Mnemosyne consolidation still zero-item:** newest `consolidation_log` row `2026-11-19T16:20:14+00:00`, `items_consolidated: 0`, **199 of the last 200 cycles zero-item** (was 120 on 11-17). Zero-item-streak threshold still **proposed, not applied**.
- **Pages touched:** [[incidents/2026-11-18-wiki-audit-credit-outage]] (new), [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] (new), [[incidents/2026-11-14-mnemosyne-capture-stall]] (new re-read + corrected title), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[index]], `log.md` — frontmatter dates → **2026-11-20**.
**Previous window:**
## Recent Changes (2026-11-16, nightly audit; live probes 2026-11-16 23:59 – 2026-11-17 00:27 AWST)
@@ -1,17 +1,21 @@
---
title: Mnemosyne capture stalled for 62+ hours (2026-11-14 09:33 AWST onward) — five unclosed alerts, freeze total on re-read
title: Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
type: incident
status: active
created: 2026-11-15
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-11-18
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
confidence: high
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
---
# Mnemosyne capture stalled for 62+ hours (from 2026-11-14 09:33 AWST)
# Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
> ⚠ **2026-11-20 re-read — the freeze ended by itself at 2026-11-18 19:25 AWST (105 h 52 m total), and every
> interactive session inside the window was captured.** The "capture is dead" reading below is a *quiet estate*,
> not a broken hook. Read the newest re-read section before the earlier readings.
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
@@ -144,6 +148,58 @@ total.**
`review_after` 2026-11-18 — this page is re-read at the next audit, and the monitor keeps alerting.
## Re-read 2026-11-20 00:5x AWST — the freeze ended by itself after **105 h 52 m**, and the "capture is dead" reading was wrong
**The capture freeze is over. The alarm is not.** Live reads of `mnemosyne.db` at 2026-11-20 00:0x–00:5x AWST:
| counter | 2026-11-17 00:27 | 2026-11-20 00:5x |
| --- | --- | --- |
| `working_memory` | 1130 | **1131** |
| `episodic_memory` | 1249 | **1250** |
| newest write | `2026-11-14 09:32:59` AWST | **`2026-11-18 19:25:03` AWST** (stored UTC `11:25:03`) |
The break coincides with nothing: no config edit, no restart and no repair is recorded against the capture path
in the window — the only nearby state change is the 11-18 19:18 desktop session itself. The single new
`working_memory` row carries `session_id: hermes_20261118_191808_5c9e08` and its content begins
`[USER] You are helping me migrate contex…`, which identifies the captured turn (the `desktop` session
**`20261118_191808_5c9e08`**, *"Helping migrate context AI assistant"*, 9 messages, started 19:18 AWST).
**Stall duration: 105 h 52 m** (2026-11-14 09:33 → 2026-11-18 19:25 AWST).
⚠ **The correction that matters — the window contained almost no conversation to capture.** `state.db` lists
**three** non-cron sessions since 2026-11-14 00:00 AWST: two `desktop` sessions on **11-14** (**08:35**, 40
messages; **08:52**, 41 messages — both *before* the 09:33 boundary), then **nothing** until the **11-18 19:18**
session, whose turn is the row that ends the freeze. Every interactive session in the window was captured, so
the earlier figures on this page (`0 writes in 38 h 46 m`, `62 h 54 m`) are real but measure **a quiet estate**,
not a broken hook: `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message
whose session source is not `cron`/`subagent`, so **silence and failure read identically**. That ambiguity — not
a broken capture path — is the durable defect on this page.
**The monitor is still failing on that same metric, after the break.** `ea79a2107e36` slots since 11-18:
`11-18 00:01:28` failed · `11-18 12:01:13` failed · **`11-19 00:01:34` `completed`** (the only clean slot since
11-17) · `11-19 12:01:17` failed · `11-20 00:01:22` failed — every failure `Script exited with code 1 / 🚨
Mnemosyne Health Monitor — exception(s): STALE MEMORY PIPELINE: capture …`, i.e. **the same signature firing on
a healthy pipeline** (`failure_streak: 2`; incidents `ea79a2_f2be2db3bfce` at 2026-11-19 12:01:58 and
`ea79a2_aae4d914f3c9` at 2026-11-20 00:03:36, both `alerted`, both `failure_type: auth` — the scheduler's
classifier, not a credential fault). The `11-19 00:01` pass and the `12:01` failure bracket the threshold: the
metric is *age of the newest captured conversational turn*, which only grows while the estate is silent.
**The other half of the original finding is unchanged, and larger.** The consolidator still consolidates zero
items: newest `consolidation_log` row **`2026-11-19T16:20:14.587267+00:00`** (= 2026-11-20 00:20 AWST, so it is
running on its ~31-minute cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle
— timestamp updated for watchdog"`. **199 of the last 200 cycles are zero-item** (was **120** at the 11-17 read
— consistent with ~31 min cadence over the three days). The zero-item-streak threshold remains **proposed, not
applied**.
**Unclosed incidents: 5 → 4** (one of the rows present at the 11-17 read has since closed). Current four, none
closed automatically: `cbfe43_fa12cca1ad4b` (*auth*, 2026-11-18 00:01:42, `alerted_at: None` — the wiki audit's
silent skip, see [[incidents/2026-11-18-wiki-audit-credit-outage]]), `96eb95_98ade422d195` (*script*,
2026-11-18 20:01:15 → last_seen 2026-11-20 00:31:16 — see
[[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]), `ea79a2_f2be2db3bfce` (2026-11-19
12:01:58) and `ea79a2_aae4d914f3c9` (2026-11-20 00:03:36).
**Standing open question, restated:** not *"why did capture stop"* but **"why does the gate treat conversational
silence as a fault?"** Until that is fixed, this page will keep re-opening on quiet days.
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
@@ -0,0 +1,111 @@
---
title: Mnemosyne Watchdog red for 115 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST)
type: incident
status: active
created: 2026-11-20
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
confidence: high
tags: [mnemosyne, memory, conflicts, watchdog, cron, alert-fatigue, incident]
sources: [live-cron-executions-db, live-cron-incidents, live-mnemosyne-db, live-deliveries-db, live-jobs-json]
---
# Mnemosyne Watchdog red for 115 consecutive slots (from 2026-11-18 20:01 AWST)
**Open.** `96eb95ce12f2` (*Mnemosyne Watchdog*, 15-minute cadence) has failed **every slot** since
**2026-11-18 20:01:06 AWST** — **115 consecutive failures** through **2026-11-20 00:31:16 AWST**, the last
recorded before this page was written (it was 114 when the audit began; the streak gained a slot mid-audit).
`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`**.
This is the estate's **only** red cron row with a large streak, and it is a *single durable condition*
re-reported every 15 minutes, not a series of separate faults.
## The failure is a fixed, self-describing threshold trip
```
Script exited with code 1
stdout:
🚨 Many unresolved conflicts: 59 (threshold: 50)
```
`cron_incidents` row **`96eb95_98ade422d195`** — `state: alerted`, `failure_type: script`,
`first_seen_at 2026-11-18T20:01:15.906837+08:00`, `last_seen_at 2026-11-20T00:31:16.326357+08:00`,
`alerted_at 2026-11-19T20:46:23.428143+08:00`, `closed_at: None`, output file
`/home/hermes/.hermes/cron/output/96eb95ce12f2/2026-11-20_00-16-15.md`.
Unlike the health monitor's rows (whose signatures embed a changing float and so fragment into a new row per
slot → [[incidents/2026-11-14-mnemosyne-capture-stall]]), **this one fixes on the constant count `59` and
therefore stays a single row.** The cost of that stability is the mirror image: it never closes, and it
re-alerts forever.
## Alerting works — and that is the problem
`deliveries.db`, window 2026-11-18 → 2026-11-20 00:0x — the watchdog's own failure notices:
| delivered | execution |
|---|---|
| 2026-11-18 20:01:16 | `f9865723b4f64d9fbfca7711ab053056` |
| 2026-11-19 02:16:16 | `7aa35681fd914a1e8db64b8dd0e09c84` |
| 2026-11-19 08:31:17 | `1ccc7e280d234287973edf5e0b35a5b7` |
| 2026-11-19 12:01:59 | `396940d9514b419eb024b83b3c10198d` |
| 2026-11-19 14:31:35 | `0766953f92dc4fc4b0d0404dca51683f` |
| 2026-11-19 20:46:19 | `db010a8294bf4cd78457a265a5fb167c` |
| 2026-11-20 00:03:37 | `64c6b5202463487c906f860bb746138b` |
All `status: delivered`, all `for_failure: 1`. So detection, the failure notice and the push path are **all
working**. The output is an alert roughly every 3–6 h for a threshold breach that has not moved in two days —
which is exactly how a red signal stops meaning anything: while this row is red, a *new* Mnemosyne fault
arriving on the same job is indistinguishable from the standing one.
## What the 59 conflicts actually are
Live read of `~/.hermes/mnemosyne/data/mnemosyne.db` (2026-11-20 00:3x AWST): **223 conflicts total, 59
unresolved.** Unresolved by creation day:
| created | unresolved |
|---|---|
| 2026-10-31 | 11 |
| 2026-11-02 | 2 |
| 2026-11-03 | 12 |
| 2026-11-06 | 1 |
| 2026-11-07 | 2 |
| 2026-11-08 | 1 |
| 2026-11-12 | 13 |
| 2026-11-13 | 3 |
| **2026-11-18** | **14** |
| **total** | **59** |
Two things stand out:
1. **Most of the backlog is old.** 42 of the 59 predate 2026-11-14; the oldest batch is from **2026-10-31**,
which is the same batch recorded on [[systems/mnemosyne]] as *"all created in one batch at 2026-10-31
19:26:39 and sharing a single `fact_a_id`, the extraction-noise signature"*. The documented remediation for
this class is to bulk-resolve them as `false_positive` (see the `mnemosyne-operations` skill) — **not
applied here**.
2. **The threshold has been raised before.** The 2026-10-18 and 2026-11-01 reads on [[systems/mnemosyne]] both
record *"1 unresolved conflict (benign, threshold 10)"*. It now reads **50**. So the tripwire was already
loosened 5×, and the backlog has since grown past even that. **UNVERIFIED:** when the threshold was changed
from 10 to 50, and by whom; basis: two dated page reads vs the live alert text — the config line itself was
not opened in this window.
## The 2026-11-18 batch coincides with the migration session
The 14 conflicts created on **2026-11-18** are the newest batch, and 2026-11-18 is also the day of the only
interactive session in this window — `20261118_191808_5c9e08` (desktop, 9 messages, title *"Helping migrate
context AI assistant"*), whose turns were captured into `working_memory` at **19:25:03 AWST** and summarised
episodically at **19:55:20 AWST**. Consistent with the standing assessment that most conflicts in this bank are
LLM-extraction artefacts rather than real contradictions, but **UNVERIFIED:** that the 14 rows are extraction
noise from that session's turns specifically; basis: same-day coincidence plus the page's prior noise analysis —
the fact pairs were not inspected.
## Not applied / open
- **Nothing was resolved, reclassified or deleted.** No conflict row was touched; no threshold was changed.
- The watchdog's cadence, signature construction and alert path were left as-is.
- Open question: should a *standing* threshold breach alert on state change rather than on every slot?
Same family as the zero-item-consolidation streak problem on
[[incidents/2026-11-14-mnemosyne-capture-stall]] — both are conditions that re-report continuously because
the gate tests a level, not a change.
→ [[systems/mnemosyne]], [[systems/scheduled-tasks]]
@@ -0,0 +1,89 @@
---
title: Wiki Audit failed two consecutive nights on a chained-provider credit/auth fault (2026-11-18, 2026-11-19) — 11-17 and 11-18 unaudited
type: incident
status: active
created: 2026-11-20
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
confidence: high
tags: [cron, wiki-audit, incident, model-routing, credentials, credits, router, silent-failure]
sources: [live-cron-executions-db, live-cron-incidents, live-jobs-json, live-deliveries-db, live-wiki-log]
---
# Wiki Audit failed two consecutive nights (2026-11-18, 2026-11-19) — 11-17 and 11-18 were never audited
**Open.** `cbfe434a3b5e` (**Wiki Audit — Nightly**, schedule `0 0 * * *`) failed on **two consecutive slots**,
so the audit that should have reviewed **2026-11-17** and the audit that should have reviewed **2026-11-18**
never ran. The **2026-11-20 00:00** slot is the first to fire since; at the time of writing its execution row
reads `running` (started `2026-11-20T00:00:08`).
This run therefore audits **three windows** — the standard last 24 h plus the two that were skipped.
## Evidence (live probes 2026-11-20 00:1x–00:4x AWST)
| slot | started | finished / last seen | duration | status |
|---|---|---|---|---|
| 2026-11-18 00:00 | `2026-11-18T00:00:08` | `2026-11-18T00:01:43` (incident `first_seen_at` `00:01:42.342149+08:00`) | ~1 m 35 s | `failed` |
| 2026-11-19 00:00 | `2026-11-19T00:00:16` | `2026-11-19T00:03:02` (incident `last_seen_at` `00:03:01.420319+08:00`) | ~2 m 46 s | `failed` |
| 2026-11-20 00:00 | `2026-11-20T00:00:08` | — | — | `running` |
Both failures carry the **same 314-character `RuntimeError`** — a concatenation of every target the router tried:
```
RuntimeError: command-code/deepseek/deepseek-v4-flash: model — [400]: You have insufficient credits to
make this request. Please purchase more credits to continue using the service. (HTTP 400);
openai-compatible-chat-conn:52258193/deepseek-v4-flash-0731: auth — [401]: invalid auth token (HTTP 401);
openai-compati…
```
The tail is cut off at the 314-char cap, mid-way through a further `openai-compatible-…` target, so the
**final** target's verdict is not recoverable from this field. The shape matters more than the wording: this is
not one bad pin — the **whole fallback chain** was refused, first on **credit exhaustion** (`command-code`),
then on an **invalid auth token** (the second target), then at least one more target whose error is truncated away.
- `cron/jobs.json`: `last_status: error`, `failure_streak: 2` — clean 0 at the 2026-11-17 00:27 run.
- `cron_incidents`: row **`cbfe43_fa12cca1ad4b`** — `state: detected`, `failure_type: auth`,
`first_seen_at 2026-11-18T00:01:42.342149+08:00`, `last_seen_at 2026-11-19T00:03:01.420319+08:00`,
**`alerted_at: None`**, `acked_at: None`, `closed_at: None`, output file
`/home/hermes/.hermes/cron/output/cbfe434a3b5e/2026-11-19_00-03-01.md`.
- `deliveries.db`, window since 2026-11-18: **24 rows — 23 `delivered` (9 of them `for_failure: 1`) and one
`failed`** (`2026-11-19 08:34:13`, a *content* delivery: `live adapter send failed: Timed out; live adapter
delivery to telegram`, consistent with *Gmail Action Digest* `08eb86f02d37` reading `delivery_failed`).
**No row is attributable to either failed audit slot** — both carry `delivery_outcome: suppressed` on the
execution row (as does the clean `2026-11-17` slot), i.e. this job's alerts are not wired to leave the box.
So the outage produced **no push to Anthony** and the red cron row was the only signal.
## Why this matters beyond one job
The wiki audit is this estate's own feedback loop: its output is the `log.md` entry and the page edits that keep
`current-state.md`, `systems/mnemosyne.md` and `systems/scheduled-tasks.md` honest. Two silent skips mean the wiki
stood still for two nights while the estate did not — including the window in which the **Mnemosyne watchdog went
red for 114 consecutive slots** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]. The gap is
visible in `index.md`: its newest feed entry was still **2026-11-16** on the morning of 11-20.
## Precedent — and a repeating pattern
Same class as [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] (**HTTP 429**, *"all targets were
skipped by pre-dispatch filters"*, 6 targets evaluated / **0 attempted**) and
[[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. Two observations worth carrying forward:
1. **The 11-10 failure self-cleared on the next slot** — so a single red night for this job is not necessarily a
standing fault. This one did **not** self-clear: two slots, identical error, ~1.5 and ~2.7 minutes each.
2. **`alerted_at: NULL` on both rows.** The 11-10 row was also left unalerted. Two independent failures of this
job have now produced incident rows that never alerted, which points at the job's failure-*delivery* path
rather than at the alert throttle.
## Not applied / open
- **Nothing was changed**: the job's pin, provider and schedule are untouched, no credential was rotated, no
cron entry was edited.
- No re-run was forced; the 2026-11-20 00:00 slot was left to fire on schedule (it did, and its result is the
entry appended to `log.md` the same night).
- **UNVERIFIED:** whether the credit exhaustion is on the `command-code` account specifically or the router's
shared pool — the concatenated error does not name which target a *successful* dispatch would have used;
basis: the truncated 314-char error string.
- **UNVERIFIED:** whether this job's failures are configured `suppressed`, or whether its failure path is simply
not wired to delivery; basis: absence of delivery rows, not a read of the job's delivery config.
→ [[systems/scheduled-tasks]], [[systems/omniroute]]
+13 -6
View File
File diff suppressed because one or more lines are too long
+11
View File
@@ -1864,3 +1864,14 @@ recorded here so the next audit does not re-litigate it.
- `incidents/2026-10-26-planned-restart-orphaned-cron-run.md` — both `ea79a2_*` incident rows closed/resolved (c3ab2d784ba3 10-27, a4661554a3c2 closed 11-07 15:43), error string absent from retained executions window → status → resolved; drain-policy follow-ups 1/3/4 remain open design questions.
- `reports/communities-wa-social-media-audit-2026.md` — live wa.gov.au check: remit verbatim-identical, five Ministers unchanged, site updated 18 Nov 2026 with fresh announcements.
- Quartz rebuilt + deployed via second `wiki-maintain.sh` run (pattern 13).
## 2026-11-20 00:5x — nightly audit (three windows: 2026-11-17, 11-18 and 11-19 AWST)
**Why three windows:** the audit's own slots at **2026-11-18T00:00:08 → 00:01:43** and **2026-11-19T00:00:16 → 00:03:02** both closed `failed`, so **2026-11-17 and 2026-11-18 were never audited**. This run covers all three.
- **New page — [[incidents/2026-11-18-wiki-audit-credit-outage]]:** `cbfe434a3b5e` (Wiki Audit — Nightly) is at `failure_streak: 2`, both failures on a chained-provider **credit/auth** fault (last clean run 11-17). Both failed slots carry `delivery_outcome: suppressed`, so **no failure notice ever reached Anthony** — the estate's own auditor was the one thing nobody heard about. Its incident row `cbfe43_fa12cca1ad4b` (`failure_type: auth`, opened 2026-11-18 00:01:42) is still `alerted` with **`alerted_at: None`**, the same silent-skip shape as the 11-10 429 incident.
- **New page — [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]:** `96eb95ce12f2` (Mnemosyne Watchdog) has failed **every** 15-minute slot since **2026-11-18 20:01:06**, reaching **115 consecutive failures** at **2026-11-20 00:31:16** (`failure_streak: 115`, `last_status: error`), for one reason: **59 unresolved conflicts against its threshold of 50**. A single durable trip, not 115 faults — but each slot delivers a fresh *failure notice* (9 of the 24 deliveries since 11-18 carry `for_failure: 1`), so the noisy channel is the estate's only channel, and every slot re-opens incident `96eb95_98ade422d195`.
- **Corrected — [[incidents/2026-11-14-mnemosyne-capture-stall]] and [[systems/mnemosyne]]:** the freeze is **over**. `working_memory` **1130 → 1131**, `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST** → the stall ran **105 h 52 m** and ended **with no intervention** (no config edit, restart or repair on the capture path). ⚠ **And the "capture is dead" reading was wrong:** `state.db` lists only **three** non-cron sessions since 11-14 00:00 — two `desktop` on **11-14 08:35 / 08:52** (both *before* the 09:33 boundary) and the **11-18 19:18** `desktop` whose turn *is* the captured row — so **every interactive session in the window was captured**. `mnemosyne_freshness_gate.py` measures the age of the newest non-`cron`/`subagent` message, so **silence and failure read identically**; that ambiguity is the real defect, and the monitor is still failing on it (`failure_streak: 2`; 11-19 00:01 was the only clean slot). Title and body reframed accordingly.
- **[[systems/scheduled-tasks]] — new Count bullet (11-17 / 11-18 / 11-19):** ledger **173 rows (171 `completed` / 2 `failed`)** / **172 (153/19)** / **176 (78/98)** — the last is the worst day in the retained window; 11-20 so far **8 (3 / 4 / 1 `running`)**. Deliveries since 11-18: **24 — 23 `delivered`** (9 `for_failure: 1`) / **1 `failed`** (a *content* delivery, `live adapter send failed: Timed out`, `2026-11-19 08:34:13`). Fleet **29 jobs (26 enabled / 3 disabled)** — a **twelfth consecutive window** at the same count (11-18 and 11-19 went unrecorded, see above) — but non-clean `last_status` **1 → 4**: `cbfe434a3b5e` `error` (2), `96eb95ce12f2` `error` (115), `ea79a2107e36` `error` (2), `08eb86f02d37` `delivery_failed` (0). `cron_incidents` unclosed **5 → 4**. The 03:00 stocktake (`1a8e90ea2271`) ran `completed` on all three days — **11 m 35 s / 23 m 07 s / 24 m 38 s** — so the estate's structural maintenance kept running while both alarm paths were red.
- **[[current-state]]** — new Recent Changes section for the 2026-11-17 → 2026-11-20 window. Frontmatter `updated` / `verified_on` bumped to **2026-11-20** across `index.md`, `current-state.md`, `systems/mnemosyne.md`, `systems/scheduled-tasks.md` and the two touched incident pages.
- **Mnemosyne consolidation is still zero-item:** newest `consolidation_log` row **`2026-11-19T16:20:14+00:00`** (= 2026-11-20 00:20 AWST, ~31 min cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle — timestamp updated for watchdog"`; **199 of the last 200 cycles are zero-item** (was **120** on the 11-17 read). The zero-item-streak threshold remains **proposed, not applied**.
+4 -4
View File
File diff suppressed because one or more lines are too long
+4 -3
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-11-18
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
@@ -14,6 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
# Scheduled Tasks
## Count
- **Current as of 2026-11-20 00:5x (three-window audit — 2026-11-17, 11-18 and 11-19; the audits slotted at 11-18 00:00 and 11-19 00:00 both died, so 11-17 and 11-18 had never been audited; live `jobs.json` + `executions.db` + `deliveries.db` + `cron_incidents` + `mnemosyne.db`).** Ledger: **11-17 — 173 rows, 171 `completed` / 2 `failed`** (both `ea79a2107e36` *Mnemosyne Health Monitor*, `00:01` and `12:01`); **11-18 — 172 rows, 153 / 19** (this audit `00:00:08 → 00:01:43`, the monitor twice, and **16** *Mnemosyne Watchdog* slots from `20:01:06`); **11-19 — 176 rows, 78 / 98** — **the worst day in the retained window** (audit `00:00:16 → 00:03:02`, monitor `12:01`, **96** watchdog slots); **11-20 so far — 8 rows, 3 `completed` / 4 `failed` / 1 `running`** (this audit). Deliveries since 11-18: **24 — 23 `delivered` (9 `for_failure: 1`) / 1 `failed`** (`2026-11-19 08:34:13`, a *content* delivery: `live adapter send failed: Timed out; live adapter delivery to telegram`). Fleet **29 jobs (26 enabled / 3 disabled)** — a **twelfth consecutive window at the same count** (following the 11th, recorded on 11-17; the 11-18 and 11-19 windows went unrecorded because both audits died) — but non-clean `last_status` **1 → 4**: `cbfe434a3b5e` Wiki Audit `error`, `failure_streak` **2** (last clean run 11-17), `96eb95ce12f2` Watchdog `error`, streak **115**, `ea79a2107e36` Health Monitor `error`, streak **2**, `08eb86f02d37` Gmail Action Digest `delivery_failed`, streak **0**. **Two of the three jobs that keep the fleet safe are themselves red — and the 03:00 stocktake is not:** `1a8e90ea2271` ran `completed` on 11-17 (**11 m 35 s**), 11-18 (**23 m 07 s**) and 11-19 (**24 m 38 s**), so the estate's structural maintenance kept running while both alarm paths failed. `cron_incidents` unclosed **5 → 4**, none closed automatically: `cbfe43_fa12cca1ad4b` (*auth*, 2026-11-18 00:01:42, **`alerted_at: None`** — the audit's silent skip), `96eb95_98ade422d195` (*script*, 2026-11-18 20:01:15 → last_seen 2026-11-20 00:31:16), `ea79a2_f2be2db3bfce` (2026-11-19 12:01:58) and `ea79a2_aae4d914f3c9` (2026-11-20 00:03:36) — both monitor rows on the same `STALE MEMORY PIPELINE` metric that is now measuring a *quiet estate* rather than a broken capture (see [[systems/mnemosyne]]). Cadence and pins unchanged; this audit still runs `custom:omni-101` / `deepseek-paid-smart`, and the Mnemosyne Watchdog's model/provider fields remain unset (inherits global). Full detail: [[incidents/2026-11-18-wiki-audit-credit-outage]], [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]].
- **Current as of 2026-11-17 00:27 (window 2026-11-16 00:00 -> 23:59; nightly audit).** Still **29
jobs** (26 enabled, 3 disabled) — 11th consecutive window unchanged. 11-16 ledger: **175 rows — 173
`completed`, 2 `failed`**; both failures are `ea79a2107e36` *Mnemosyne Health Monitor*