Files
agent-estate-wiki/incidents/2026-11-18-wiki-audit-credit-outage.md
T

7.6 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Wiki Audit failed two consecutive nights on a chained-provider credit/auth fault (2026-11-18, 2026-11-19) — 11-17 and 11-18 unaudited incident resolved 2026-11-20 2026-11-21 2026-11-21 2026-11-28 high
cron
wiki-audit
incident
model-routing
credentials
credits
router
silent-failure
live-cron-executions-db
live-cron-incidents
live-jobs-json
live-deliveries-db
live-wiki-log

Wiki Audit failed two consecutive nights (2026-11-18, 2026-11-19) — 11-17 and 11-18 were never audited

Open. cbfe434a3b5e (Wiki Audit — Nightly, schedule 0 0 * * *) failed on two consecutive slots, so the audit that should have reviewed 2026-11-17 and the audit that should have reviewed 2026-11-18 never ran. The 2026-11-20 00:00 slot is the first to fire since; at the time of writing its execution row reads running (started 2026-11-20T00:00:08).

This run therefore audits three windows — the standard last 24 h plus the two that were skipped.

Evidence (live probes 2026-11-20 00:1x–00:4x AWST)

slot started finished / last seen duration status
2026-11-18 00:00 2026-11-18T00:00:08 2026-11-18T00:01:43 (incident first_seen_at 00:01:42.342149+08:00) ~1 m 35 s failed
2026-11-19 00:00 2026-11-19T00:00:16 2026-11-19T00:03:02 (incident last_seen_at 00:03:01.420319+08:00) ~2 m 46 s failed
2026-11-20 00:00 2026-11-20T00:00:08 — — running

Both failures carry the same 314-character RuntimeError — a concatenation of every target the router tried:

RuntimeError: command-code/deepseek/deepseek-v4-flash: model — [400]: You have insufficient credits to
make this request. Please purchase more credits to continue using the service. (HTTP 400);
openai-compatible-chat-conn:52258193/deepseek-v4-flash-0731: auth — [401]: invalid auth token (HTTP 401);
openai-compati…

The tail is cut off at the 314-char cap, mid-way through a further openai-compatible-… target, so the final target's verdict is not recoverable from this field. The shape matters more than the wording: this is not one bad pin — the whole fallback chain was refused, first on credit exhaustion (command-code), then on an invalid auth token (the second target), then at least one more target whose error is truncated away.

  • cron/jobs.json: last_status: error, failure_streak: 2 — clean 0 at the 2026-11-17 00:27 run.
  • cron_incidents: row cbfe43_fa12cca1ad4b — state: detected, failure_type: auth, first_seen_at 2026-11-18T00:01:42.342149+08:00, last_seen_at 2026-11-19T00:03:01.420319+08:00, alerted_at: None, acked_at: None, closed_at: None, output file /home/hermes/.hermes/cron/output/cbfe434a3b5e/2026-11-19_00-03-01.md.
  • deliveries.db, window since 2026-11-18: 24 rows — 23 delivered (9 of them for_failure: 1) and one failed (2026-11-19 08:34:13, a content delivery: live adapter send failed: Timed out; live adapter delivery to telegram, consistent with Gmail Action Digest 08eb86f02d37 reading delivery_failed). No row is attributable to either failed audit slot — both carry delivery_outcome: suppressed on the execution row (as does the clean 2026-11-17 slot), i.e. this job's alerts are not wired to leave the box. So the outage produced no push to Anthony and the red cron row was the only signal.

Why this matters beyond one job

The wiki audit is this estate's own feedback loop: its output is the log.md entry and the page edits that keep current-state.md, systems/mnemosyne.md and systems/scheduled-tasks.md honest. Two silent skips mean the wiki stood still for two nights while the estate did not — including the window in which the Mnemosyne watchdog went red for 114 consecutive slots → incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm. The gap is visible in index.md: its newest feed entry was still 2026-11-16 on the morning of 11-20.

Precedent — and a repeating pattern

Same class as incidents/2026-11-10-wiki-audit-429-all-targets-skipped (HTTP 429, "all targets were skipped by pre-dispatch filters", 6 targets evaluated / 0 attempted) and incidents/2026-10-21-omniroute-pre-dispatch-filter-skip. Two observations worth carrying forward:

  1. The 11-10 failure self-cleared on the next slot — so a single red night for this job is not necessarily a standing fault. This one did not self-clear: two slots, identical error, ~1.5 and ~2.7 minutes each.
  2. alerted_at: NULL on both rows. The 11-10 row was also left unalerted. Two independent failures of this job have now produced incident rows that never alerted, which points at the job's failure-delivery path rather than at the alert throttle.

Re-read 2026-11-21 00:0x AWST — resolved; and the ledger shows the outage was two nights, not one

  • The job is clean again. cbfe434a3b5e reads last_status: ok, failure_streak = 0. The execution ledger dates the outage exactly:

    Slot Execution Outcome
    2026-11-18 00:00:08 d6323ba3b7ca4d8a95d0f57bf20b4807 failed (delivery_outcome: suppressed)
    2026-11-19 00:00:16 668984226ccc47b09cb66fe65ae89ddc failed (delivery_outcome: suppressed)
    2026-11-20 00:00:08 b88bd7d8007248399f57cccfc98b536a completed (report not delivered)
    2026-11-21 00:00:08 ec65ca362c6847fcb9405daa15d1f6ed running — the run authoring this note
  • Two nights were lost, not one. The 11-18 and 11-19 runs both failed, so windows 11-17 and 11-18 have no audit of their own. The 11-20 run absorbed both via its three-window catch-up (2026-11-17 → 2026-11-20), read from the cron ledger and state.db rather than from conversation — state.db holds no interactive sessions in those windows, so there was no conversation to miss.

  • No re-run of either lost slot was forced; recovery came from the next scheduled slot running normally.

  • Not re-tested: whether the credit/auth fault can recur. The job's pin, provider and schedule are unchanged and no credential was rotated, so a recurrence would present identically.

  • The silent-skip class this page identified is now observed, not inferred: both failed nights carry delivery_outcome: suppressed and the incident row cbfe43_fa12cca1ad4b has alerted_at: NULL — two consecutive failed nights produced no alert and no report, which is why the outage only surfaced when the 11-20 audit read the ledger. Whether that suppression is job-level config or framework behaviour is still UNVERIFIED (the job's delivery config was not read).

Not applied / open

  • Nothing was changed: the job's pin, provider and schedule are untouched, no credential was rotated, no cron entry was edited.
  • No re-run was forced; the 2026-11-20 00:00 slot was left to fire on schedule (it did, and its result is the entry appended to log.md the same night).
  • UNVERIFIED: whether the credit exhaustion is on the command-code account specifically or the router's shared pool — the concatenated error does not name which target a successful dispatch would have used; basis: the truncated 314-char error string.
  • UNVERIFIED: whether this job's failures are configured suppressed, or whether its failure path is simply not wired to delivery; basis: absence of delivery rows, not a read of the job's delivery config.

→ systems/scheduled-tasks, systems/omniroute