Files
agent-estate-wiki/incidents/2026-11-18-wiki-audit-credit-outage.md
Tony0410 226767f3ec wiki audit 2026-11-20: three-window catch-up (11-17/11-18/11-19, both prior audit slots died)
- new: incidents/2026-11-18-wiki-audit-credit-outage (audit failed 11-18 + 11-19, deliveries suppressed, incident row alerted_at=None)
- new: incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm (115 consecutive slots, 59 unresolved conflicts vs threshold 50)
- corrected: incidents/2026-11-14-mnemosyne-capture-stall + systems/mnemosyne — freeze ENDED 2026-11-18 19:25 AWST (105h52m, no intervention); 'capture dead' reading was conversational silence (every interactive session captured)
- systems/scheduled-tasks: new Count bullet 11-17 173(171/2) / 11-18 172(153/19) / 11-19 176(78/98); fleet 29 (26/3); non-clean last_status 1->4; unclosed incidents 5->4; deliveries 24 (23 delivered / 1 failed)
- index.md feed + incidents + systems lines; log.md dated entry; current-state.md new section; frontmatter dates -> 2026-11-20
2026-11-20 01:02:44 +08:00

5.7 KiB
Raw Permalink Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Wiki Audit failed two consecutive nights on a chained-provider credit/auth fault (2026-11-18, 2026-11-19) — 11-17 and 11-18 unaudited incident active 2026-11-20 2026-11-20 2026-11-20 2026-11-21 high
cron
wiki-audit
incident
model-routing
credentials
credits
router
silent-failure
live-cron-executions-db
live-cron-incidents
live-jobs-json
live-deliveries-db
live-wiki-log

Wiki Audit failed two consecutive nights (2026-11-18, 2026-11-19) — 11-17 and 11-18 were never audited

Open. cbfe434a3b5e (Wiki Audit — Nightly, schedule 0 0 * * *) failed on two consecutive slots, so the audit that should have reviewed 2026-11-17 and the audit that should have reviewed 2026-11-18 never ran. The 2026-11-20 00:00 slot is the first to fire since; at the time of writing its execution row reads running (started 2026-11-20T00:00:08).

This run therefore audits three windows — the standard last 24 h plus the two that were skipped.

Evidence (live probes 2026-11-20 00:1x–00:4x AWST)

slot started finished / last seen duration status
2026-11-18 00:00 2026-11-18T00:00:08 2026-11-18T00:01:43 (incident first_seen_at 00:01:42.342149+08:00) ~1 m 35 s failed
2026-11-19 00:00 2026-11-19T00:00:16 2026-11-19T00:03:02 (incident last_seen_at 00:03:01.420319+08:00) ~2 m 46 s failed
2026-11-20 00:00 2026-11-20T00:00:08 — — running

Both failures carry the same 314-character RuntimeError — a concatenation of every target the router tried:

RuntimeError: command-code/deepseek/deepseek-v4-flash: model — [400]: You have insufficient credits to
make this request. Please purchase more credits to continue using the service. (HTTP 400);
openai-compatible-chat-conn:52258193/deepseek-v4-flash-0731: auth — [401]: invalid auth token (HTTP 401);
openai-compati…

The tail is cut off at the 314-char cap, mid-way through a further openai-compatible-… target, so the final target's verdict is not recoverable from this field. The shape matters more than the wording: this is not one bad pin — the whole fallback chain was refused, first on credit exhaustion (command-code), then on an invalid auth token (the second target), then at least one more target whose error is truncated away.

  • cron/jobs.json: last_status: error, failure_streak: 2 — clean 0 at the 2026-11-17 00:27 run.
  • cron_incidents: row cbfe43_fa12cca1ad4b — state: detected, failure_type: auth, first_seen_at 2026-11-18T00:01:42.342149+08:00, last_seen_at 2026-11-19T00:03:01.420319+08:00, alerted_at: None, acked_at: None, closed_at: None, output file /home/hermes/.hermes/cron/output/cbfe434a3b5e/2026-11-19_00-03-01.md.
  • deliveries.db, window since 2026-11-18: 24 rows — 23 delivered (9 of them for_failure: 1) and one failed (2026-11-19 08:34:13, a content delivery: live adapter send failed: Timed out; live adapter delivery to telegram, consistent with Gmail Action Digest 08eb86f02d37 reading delivery_failed). No row is attributable to either failed audit slot — both carry delivery_outcome: suppressed on the execution row (as does the clean 2026-11-17 slot), i.e. this job's alerts are not wired to leave the box. So the outage produced no push to Anthony and the red cron row was the only signal.

Why this matters beyond one job

The wiki audit is this estate's own feedback loop: its output is the log.md entry and the page edits that keep current-state.md, systems/mnemosyne.md and systems/scheduled-tasks.md honest. Two silent skips mean the wiki stood still for two nights while the estate did not — including the window in which the Mnemosyne watchdog went red for 114 consecutive slots → incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm. The gap is visible in index.md: its newest feed entry was still 2026-11-16 on the morning of 11-20.

Precedent — and a repeating pattern

Same class as incidents/2026-11-10-wiki-audit-429-all-targets-skipped (HTTP 429, "all targets were skipped by pre-dispatch filters", 6 targets evaluated / 0 attempted) and incidents/2026-10-21-omniroute-pre-dispatch-filter-skip. Two observations worth carrying forward:

  1. The 11-10 failure self-cleared on the next slot — so a single red night for this job is not necessarily a standing fault. This one did not self-clear: two slots, identical error, ~1.5 and ~2.7 minutes each.
  2. alerted_at: NULL on both rows. The 11-10 row was also left unalerted. Two independent failures of this job have now produced incident rows that never alerted, which points at the job's failure-delivery path rather than at the alert throttle.

Not applied / open

  • Nothing was changed: the job's pin, provider and schedule are untouched, no credential was rotated, no cron entry was edited.
  • No re-run was forced; the 2026-11-20 00:00 slot was left to fire on schedule (it did, and its result is the entry appended to log.md the same night).
  • UNVERIFIED: whether the credit exhaustion is on the command-code account specifically or the router's shared pool — the concatenated error does not name which target a successful dispatch would have used; basis: the truncated 314-char error string.
  • UNVERIFIED: whether this job's failures are configured suppressed, or whether its failure path is simply not wired to delivery; basis: absence of delivery rows, not a read of the job's delivery config.

→ systems/scheduled-tasks, systems/omniroute