--- title: Wiki Audit failed two consecutive nights on a chained-provider credit/auth fault (2026-11-18, 2026-11-19) — 11-17 and 11-18 unaudited type: incident status: active created: 2026-11-20 updated: 2026-11-20 verified_on: 2026-11-20 review_after: 2026-11-21 confidence: high tags: [cron, wiki-audit, incident, model-routing, credentials, credits, router, silent-failure] sources: [live-cron-executions-db, live-cron-incidents, live-jobs-json, live-deliveries-db, live-wiki-log] --- # Wiki Audit failed two consecutive nights (2026-11-18, 2026-11-19) — 11-17 and 11-18 were never audited **Open.** `cbfe434a3b5e` (**Wiki Audit — Nightly**, schedule `0 0 * * *`) failed on **two consecutive slots**, so the audit that should have reviewed **2026-11-17** and the audit that should have reviewed **2026-11-18** never ran. The **2026-11-20 00:00** slot is the first to fire since; at the time of writing its execution row reads `running` (started `2026-11-20T00:00:08`). This run therefore audits **three windows** — the standard last 24 h plus the two that were skipped. ## Evidence (live probes 2026-11-20 00:1x–00:4x AWST) | slot | started | finished / last seen | duration | status | |---|---|---|---|---| | 2026-11-18 00:00 | `2026-11-18T00:00:08` | `2026-11-18T00:01:43` (incident `first_seen_at` `00:01:42.342149+08:00`) | ~1 m 35 s | `failed` | | 2026-11-19 00:00 | `2026-11-19T00:00:16` | `2026-11-19T00:03:02` (incident `last_seen_at` `00:03:01.420319+08:00`) | ~2 m 46 s | `failed` | | 2026-11-20 00:00 | `2026-11-20T00:00:08` | — | — | `running` | Both failures carry the **same 314-character `RuntimeError`** — a concatenation of every target the router tried: ``` RuntimeError: command-code/deepseek/deepseek-v4-flash: model — [400]: You have insufficient credits to make this request. Please purchase more credits to continue using the service. (HTTP 400); openai-compatible-chat-conn:52258193/deepseek-v4-flash-0731: auth — [401]: invalid auth token (HTTP 401); openai-compati… ``` The tail is cut off at the 314-char cap, mid-way through a further `openai-compatible-…` target, so the **final** target's verdict is not recoverable from this field. The shape matters more than the wording: this is not one bad pin — the **whole fallback chain** was refused, first on **credit exhaustion** (`command-code`), then on an **invalid auth token** (the second target), then at least one more target whose error is truncated away. - `cron/jobs.json`: `last_status: error`, `failure_streak: 2` — clean 0 at the 2026-11-17 00:27 run. - `cron_incidents`: row **`cbfe43_fa12cca1ad4b`** — `state: detected`, `failure_type: auth`, `first_seen_at 2026-11-18T00:01:42.342149+08:00`, `last_seen_at 2026-11-19T00:03:01.420319+08:00`, **`alerted_at: None`**, `acked_at: None`, `closed_at: None`, output file `/home/hermes/.hermes/cron/output/cbfe434a3b5e/2026-11-19_00-03-01.md`. - `deliveries.db`, window since 2026-11-18: **24 rows — 23 `delivered` (9 of them `for_failure: 1`) and one `failed`** (`2026-11-19 08:34:13`, a *content* delivery: `live adapter send failed: Timed out; live adapter delivery to telegram`, consistent with *Gmail Action Digest* `08eb86f02d37` reading `delivery_failed`). **No row is attributable to either failed audit slot** — both carry `delivery_outcome: suppressed` on the execution row (as does the clean `2026-11-17` slot), i.e. this job's alerts are not wired to leave the box. So the outage produced **no push to Anthony** and the red cron row was the only signal. ## Why this matters beyond one job The wiki audit is this estate's own feedback loop: its output is the `log.md` entry and the page edits that keep `current-state.md`, `systems/mnemosyne.md` and `systems/scheduled-tasks.md` honest. Two silent skips mean the wiki stood still for two nights while the estate did not — including the window in which the **Mnemosyne watchdog went red for 114 consecutive slots** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]. The gap is visible in `index.md`: its newest feed entry was still **2026-11-16** on the morning of 11-20. ## Precedent — and a repeating pattern Same class as [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] (**HTTP 429**, *"all targets were skipped by pre-dispatch filters"*, 6 targets evaluated / **0 attempted**) and [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. Two observations worth carrying forward: 1. **The 11-10 failure self-cleared on the next slot** — so a single red night for this job is not necessarily a standing fault. This one did **not** self-clear: two slots, identical error, ~1.5 and ~2.7 minutes each. 2. **`alerted_at: NULL` on both rows.** The 11-10 row was also left unalerted. Two independent failures of this job have now produced incident rows that never alerted, which points at the job's failure-*delivery* path rather than at the alert throttle. ## Not applied / open - **Nothing was changed**: the job's pin, provider and schedule are untouched, no credential was rotated, no cron entry was edited. - No re-run was forced; the 2026-11-20 00:00 slot was left to fire on schedule (it did, and its result is the entry appended to `log.md` the same night). - **UNVERIFIED:** whether the credit exhaustion is on the `command-code` account specifically or the router's shared pool — the concatenated error does not name which target a *successful* dispatch would have used; basis: the truncated 314-char error string. - **UNVERIFIED:** whether this job's failures are configured `suppressed`, or whether its failure path is simply not wired to delivery; basis: absence of delivery rows, not a read of the job's delivery config. → [[systems/scheduled-tasks]], [[systems/omniroute]]