audit(2026-11-13): third zero-failed day and a fully clean fleet; the Uber Eats fix clears the last non-clean cron row — and the pre_tool_call hook timeout recurred at load 12.08 (incident reopened resolved -> active, state.db 2.04 GB)

This commit is contained in:
Tony0410
2026-11-14 00:27:05 +08:00
parent a0ec3a1ec5
commit b0e4a54f8a
5 changed files with 133 additions and 15 deletions
+58
View File
@@ -2547,3 +2547,61 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted*
`incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md`, `systems/hermes-hq-push.md`,
`systems/scheduled-tasks.md`, `current-state.md`, `index.md`, `log.md`, `systems/skills-index.md`
(regenerated). **No page deleted, no decision reversed, no secret written.**
## 2026-11-13 23:5x – 2026-11-14 00:2x AWST — Nightly audit (window 2026-11-12 23:59 → 2026-11-13 23:59)
**Headline: a third consecutive `failed`-free day and a fully clean fleet — then the `pre_tool_call`
hook timeout came back and reopened its incident.**
- **Ledger — 2026-11-13: 174 rows, 173 `completed` / 0 `failed` / 1 `running`** (this audit,
`cbfe434a3b5e`, started 23:59:41). Trend: 11-11 **172 / 172**, 11-12 **176 / 176**, 11-13 **174 / 173**
— the **third consecutive zero-`failed` day**. Deliveries **8 rows / 8 `delivered` / 0 `for_failure`**
(11-12: 9 / 9). *cron_incidents* **0 unclosed**.
- **Fleet 29 (26 enabled / 3 disabled)** — an **eighth** consecutive window at the same count, nothing
added, retired, disabled or repinned. **Every job now reads `last_status: ok` with `failure_streak` 0:
the fleet has no non-clean row at all.** The last one cleared because **`066bb5191280` (Uber Eats Monthly
Spend Report) was fixed end-to-end on 2026-11-13** — so [[systems/scheduled-tasks]]'s *"the fleet's only
non-clean `last_status`"* is now a historical statement, not a current one.
- **Uber Eats fix, recorded for the audit trail.** Two stacked root causes: (1) the job's **runtime prompt
had grown to 88 k chars** (it carried `camofox-browser-automation` + `browser-backend-verification`), so
the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest (*"remaining 4 in
the report"*) plus forbidden run commentary; (2) the `delivery_failed` label was a **stale
`TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart — which happened
**2026-11-13 13:14**. Fix: a deterministic digest printer
**`~/.hermes/cron/scripts/uber-eats-digest.py`**, a rewritten prompt with a hard output contract, and the
skills cleared (**88 k → 1.8 k chars**); the lesson went into the `cron-job-maintenance` skill. Verified on
the **18:51 → 18:58** slot — **154 unique / 153 completed** orders, Oct **9 orders / $301.04** (unchanged),
delivery row `delivered`, `hermes cron doctor` clean across all 26 enabled jobs. Pages already regenerated
in-window by that work: [[reports/uber-eats/index]], `reports/uber-eats/2026-10.md` + `.html`
(commits `bc35c4b`, `a0ec3a1`). **This audit updated that job's row in [[systems/scheduled-tasks]].**
- ⚠ **[[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] REOPENED — `resolved` → `active`.** The
hook overran its **raised 120 s** budget at **2026-11-14 00:05:40**, followed by **10** *"skipped after
previous timeout or while still running"* lines through **00:06:38**. It refused **this audit's own** first
tool calls (first-hand: `skill_view` and `terminal` both returned the plugin timeout) for ~1 minute before
dispatch recovered unaided. `errors.log` single-source counts: **11-12 0 / 0, 11-13 0 / 0, 11-14 1 / 10** —
the **first `pre_tool_call` timeout in the retained log since the 2026-11-08 cascade**. **No cron run
failed.** ⚠ **The 2026-11-08 lines have rotated out of `errors.log`** (which now starts **2026-11-09
13:18:39**), so the page's 58-timeout figure **stands on the 2026-11-09 read and was not re-verifiable**
this window — recorded on the page rather than silently recopied.
- ⚠ **Load is the trigger and it has roughly doubled.** `load average` **12.08 / 13.02 / 12.33** on **2 vCPU**
(2026-11-08 read: **5.89**). **`state.db` is now 2.04 GB** (**2,036,908,032 B**; **2,977 sessions /
207,577 messages** — was **1.89 GB / 2,901 / 197,682** on 11-09), and the root filesystem is **88 % used**.
The **30 → 120 s fix is proven insufficient**, which settles the page's open hypothesis in the confirmatory
direction: at load ~12 the callback needs >120 s, so **there is no headroom left in the timeout**, and
re-raising it would only mask the fault. **`state.db` checkpoint/retention is now the estate's most urgent
open item.**
- **Freshness (2026-11-13 03:00 stocktake; file mtime 03:06:30): 0 stale / 131 fresh / 3 no-date of 134
pages.** Nothing needed retiring. → `raw/stocktake/freshness-20261113.txt`
- **Stocktake 2026-11-13 recovered to a normal window:** `1a8e90ea2271` ran **03:01:07 → 03:17:14 =
16 m 07 s**, `completed`, `last_status ok`, `failure_streak 0` (11-12: 22 m 10 s; the 11-08 failure was
1 h 34 m). It found **0 stale pages**, so there was nothing to re-verify and it **left no commit of its own
and no section in this log** — its freshness output was byte-identical to the copy already committed by
`e033a08`. Recorded as a **minor process gap, not a failure**: unlike the 2026-11-09 *"completes with zero
edits"* incident, there was genuinely nothing in scope.
- **`executions.db` retention floor moved to 2026-11-08T00:01:20+08:00** (was 2026-11-06T05:09:30), so the
11-07/11-08 cascade **runs** remain readable even though their log lines have rotated.
- **Pages touched:** `incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md` (frontmatter `resolved` →
`active`, plus a **Recurrence (2026-11-14)** section), `systems/scheduled-tasks.md` (new *Current as of
2026-11-13* section, the Uber Eats job row, frontmatter), `current-state.md` (new *Recent Changes
(2026-11-13)* section), `index.md` (new blurb, the incident's RESOLVED → REOPENED label, frontmatter),
`log.md` (this entry). **No page deleted, no decision reversed, no secret written.**