- new: incidents/2026-11-18-wiki-audit-credit-outage (audit failed 11-18 + 11-19, deliveries suppressed, incident row alerted_at=None) - new: incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm (115 consecutive slots, 59 unresolved conflicts vs threshold 50) - corrected: incidents/2026-11-14-mnemosyne-capture-stall + systems/mnemosyne — freeze ENDED 2026-11-18 19:25 AWST (105h52m, no intervention); 'capture dead' reading was conversational silence (every interactive session captured) - systems/scheduled-tasks: new Count bullet 11-17 173(171/2) / 11-18 172(153/19) / 11-19 176(78/98); fleet 29 (26/3); non-clean last_status 1->4; unclosed incidents 5->4; deliveries 24 (23 delivered / 1 failed) - index.md feed + incidents + systems lines; log.md dated entry; current-state.md new section; frontmatter dates -> 2026-11-20
5.7 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Wiki Audit failed two consecutive nights on a chained-provider credit/auth fault (2026-11-18, 2026-11-19) — 11-17 and 11-18 unaudited | incident | active | 2026-11-20 | 2026-11-20 | 2026-11-20 | 2026-11-21 | high |
|
|
Wiki Audit failed two consecutive nights (2026-11-18, 2026-11-19) — 11-17 and 11-18 were never audited
Open. cbfe434a3b5e (Wiki Audit — Nightly, schedule 0 0 * * *) failed on two consecutive slots,
so the audit that should have reviewed 2026-11-17 and the audit that should have reviewed 2026-11-18
never ran. The 2026-11-20 00:00 slot is the first to fire since; at the time of writing its execution row
reads running (started 2026-11-20T00:00:08).
This run therefore audits three windows — the standard last 24 h plus the two that were skipped.
Evidence (live probes 2026-11-20 00:1x–00:4x AWST)
| slot | started | finished / last seen | duration | status |
|---|---|---|---|---|
| 2026-11-18 00:00 | 2026-11-18T00:00:08 |
2026-11-18T00:01:43 (incident first_seen_at 00:01:42.342149+08:00) |
~1 m 35 s | failed |
| 2026-11-19 00:00 | 2026-11-19T00:00:16 |
2026-11-19T00:03:02 (incident last_seen_at 00:03:01.420319+08:00) |
~2 m 46 s | failed |
| 2026-11-20 00:00 | 2026-11-20T00:00:08 |
— | — | running |
Both failures carry the same 314-character RuntimeError — a concatenation of every target the router tried:
RuntimeError: command-code/deepseek/deepseek-v4-flash: model — [400]: You have insufficient credits to
make this request. Please purchase more credits to continue using the service. (HTTP 400);
openai-compatible-chat-conn:52258193/deepseek-v4-flash-0731: auth — [401]: invalid auth token (HTTP 401);
openai-compati…
The tail is cut off at the 314-char cap, mid-way through a further openai-compatible-… target, so the
final target's verdict is not recoverable from this field. The shape matters more than the wording: this is
not one bad pin — the whole fallback chain was refused, first on credit exhaustion (command-code),
then on an invalid auth token (the second target), then at least one more target whose error is truncated away.
cron/jobs.json:last_status: error,failure_streak: 2— clean 0 at the 2026-11-17 00:27 run.cron_incidents: rowcbfe43_fa12cca1ad4b—state: detected,failure_type: auth,first_seen_at 2026-11-18T00:01:42.342149+08:00,last_seen_at 2026-11-19T00:03:01.420319+08:00,alerted_at: None,acked_at: None,closed_at: None, output file/home/hermes/.hermes/cron/output/cbfe434a3b5e/2026-11-19_00-03-01.md.deliveries.db, window since 2026-11-18: 24 rows — 23delivered(9 of themfor_failure: 1) and onefailed(2026-11-19 08:34:13, a content delivery:live adapter send failed: Timed out; live adapter delivery to telegram, consistent with Gmail Action Digest08eb86f02d37readingdelivery_failed). No row is attributable to either failed audit slot — both carrydelivery_outcome: suppressedon the execution row (as does the clean2026-11-17slot), i.e. this job's alerts are not wired to leave the box. So the outage produced no push to Anthony and the red cron row was the only signal.
Why this matters beyond one job
The wiki audit is this estate's own feedback loop: its output is the log.md entry and the page edits that keep
current-state.md, systems/mnemosyne.md and systems/scheduled-tasks.md honest. Two silent skips mean the wiki
stood still for two nights while the estate did not — including the window in which the Mnemosyne watchdog went
red for 114 consecutive slots → incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm. The gap is
visible in index.md: its newest feed entry was still 2026-11-16 on the morning of 11-20.
Precedent — and a repeating pattern
Same class as incidents/2026-11-10-wiki-audit-429-all-targets-skipped (HTTP 429, "all targets were skipped by pre-dispatch filters", 6 targets evaluated / 0 attempted) and incidents/2026-10-21-omniroute-pre-dispatch-filter-skip. Two observations worth carrying forward:
- The 11-10 failure self-cleared on the next slot — so a single red night for this job is not necessarily a standing fault. This one did not self-clear: two slots, identical error, ~1.5 and ~2.7 minutes each.
alerted_at: NULLon both rows. The 11-10 row was also left unalerted. Two independent failures of this job have now produced incident rows that never alerted, which points at the job's failure-delivery path rather than at the alert throttle.
Not applied / open
- Nothing was changed: the job's pin, provider and schedule are untouched, no credential was rotated, no cron entry was edited.
- No re-run was forced; the 2026-11-20 00:00 slot was left to fire on schedule (it did, and its result is the
entry appended to
log.mdthe same night). - UNVERIFIED: whether the credit exhaustion is on the
command-codeaccount specifically or the router's shared pool — the concatenated error does not name which target a successful dispatch would have used; basis: the truncated 314-char error string. - UNVERIFIED: whether this job's failures are configured
suppressed, or whether its failure path is simply not wired to delivery; basis: absence of delivery rows, not a read of the job's delivery config.