audit(2026-11-13): third zero-failed day and a fully clean fleet; the Uber Eats fix clears the last non-clean cron row — and the pre_tool_call hook timeout recurred at load 12.08 (incident reopened resolved -> active, state.db 2.04 GB)
This commit is contained in:
+12
-3
@@ -3,9 +3,9 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-13
|
||||
verified_on: 2026-11-13
|
||||
review_after: 2026-11-13
|
||||
updated: 2026-11-14
|
||||
verified_on: 2026-11-14
|
||||
review_after: 2026-11-14
|
||||
confidence: high
|
||||
tags: [system]
|
||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
|
||||
@@ -14,6 +14,15 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
|
||||
# Current State
|
||||
Updated: 2026-11-13
|
||||
|
||||
## Recent Changes (2026-11-13, nightly audit; live probes 2026-11-13 23:59 – 2026-11-14 00:2x AWST)
|
||||
|
||||
- ✅ **2026-11-13 was the third consecutive zero-`failed` cron day: 174 rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit, `cbfe434a3b5e`, started 23:59:41). Trend: 11-11 **172 / 172**, 11-12 **176 / 176**, 11-13 **174 / 173**. **Deliveries 11-13: 8 rows, 8 / 8 `delivered`, 0 `for_failure`** (11-12: 9 / 9). Fleet **29 jobs (26 enabled / 3 disabled)** — an **eighth consecutive window** unchanged — and **every job reads `last_status: ok` with `failure_streak` 0, so the fleet has no non-clean row at all** for the first time in weeks. `cron_incidents` **0 unclosed**. → [[systems/scheduled-tasks]]
|
||||
- ⚠ **The `pre_tool_call` hook timeout is back, and the raised budget did not hold it.** `_pre_tool_call timed out after 120s` at **2026-11-14 00:05:40**, then **10** `skipped after previous timeout or while still running` lines through **00:06:38** — refusing this audit's own first tool calls (first-hand: `skill_view` and `terminal` both returned the plugin timeout) for ~1 minute before dispatch recovered unaided. `errors.log` by day: 11-12 **0 / 0**, 11-13 **0 / 0**, 11-14 **1 / 10**; it is the **first `pre_tool_call` timeout in the retained log since the 2026-11-08 cascade** (whose own lines have since rotated out — `errors.log` now starts **2026-11-09 13:18:39**). **No cron run failed** — 11-13 settled `failed`-free. **The variable is load, now ~2× the level recorded at diagnosis (`load average 12.08 / 13.02 / 12.33` on 2 vCPU vs 5.89).** The 30 → 120 s fix is **proven insufficient**; the incident is **reopened, `resolved` → `active`**. → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
- ⚠ **`state.db` is now 2.04 GB** (**2,036,908,032 B**; **2,977 sessions / 207,577 messages**) — up from **1.89 GB / 2,901 / 197,682** on 2026-11-09 — and it is the load source behind the reopened hook fault above. **Root filesystem 88 % used.** `state.db` **checkpoint/retention remains undecided** and is now the estate's most urgent open item; re-raising the hook budget again would only mask the fault. Still `journal_mode=wal`, `wal_autocheckpoint=2048`. → [[systems/hermes-agent]], [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
- ✅ **The Uber Eats Monthly Spend Report was fixed end-to-end on 2026-11-13**, clearing the fleet's last non-clean row. Two stacked root causes: (1) the job's **runtime prompt was 88 k chars** because it carried two heavy browser skills, so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest (*"remaining 4 in the report"* plus forbidden run commentary); (2) the `delivery_failed` label was a **stale `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart — which happened **2026-11-13 13:14**. Fix: a deterministic digest printer **`~/.hermes/cron/scripts/uber-eats-digest.py`**, a rewritten prompt with a hard output contract, and the skills cleared (**88 k → 1.8 k chars**); the lesson went into the `cron-job-maintenance` skill. Verified on the **18:51 → 18:58** slot: **154 unique / 153 completed** orders, Oct = **9 orders / $301.04** (unchanged), delivery row `delivered`, `hermes cron doctor` clean across all 26 enabled jobs. → [[reports/uber-eats/index]], [[systems/scheduled-tasks]]
|
||||
- **Freshness scan (2026-11-13 03:00 stocktake): 0 stale / 131 fresh / 3 no-date of 134 pages** — nothing needed retiring. → `raw/stocktake/freshness-20261113.txt`
|
||||
- **`executions.db` retention floor moved to 2026-11-08T00:01:20+08:00** (was 2026-11-06T05:09:30). The 11-07/11-08 cascade **runs** are still readable, but their **`errors.log` lines are not** — so the incident page's 58-timeout figure stands on the 2026-11-09 read and was **not re-verifiable** this window. → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
|
||||
## Recent Changes (2026-11-12, nightly audit; live probes 2026-11-12 23:59 – 2026-11-13 00:4x AWST)
|
||||
|
||||
**Clean day. The estate is incident-clean, and the only open item is an upgrade that cannot be installed.**
|
||||
|
||||
Reference in New Issue
Block a user