From b0e4a54f8ae8e1e3df4beae577ef8de4a6a102c5 Mon Sep 17 00:00:00 2001 From: Tony0410 <47945770+Tony0410@users.noreply.github.com> Date: Sat, 14 Nov 2026 00:27:05 +0800 Subject: [PATCH] =?UTF-8?q?audit(2026-11-13):=20third=20zero-failed=20day?= =?UTF-8?q?=20and=20a=20fully=20clean=20fleet;=20the=20Uber=20Eats=20fix?= =?UTF-8?q?=20clears=20the=20last=20non-clean=20cron=20row=20=E2=80=94=20a?= =?UTF-8?q?nd=20the=20pre=5Ftool=5Fcall=20hook=20timeout=20recurred=20at?= =?UTF-8?q?=20load=2012.08=20(incident=20reopened=20resolved=20->=20active?= =?UTF-8?q?,=20state.db=202.04=20GB)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- current-state.md | 15 ++++- ...1-08-pre-tool-call-hook-timeout-cascade.md | 38 ++++++++++-- index.md | 10 ++-- log.md | 58 +++++++++++++++++++ systems/scheduled-tasks.md | 27 +++++++-- 5 files changed, 133 insertions(+), 15 deletions(-) diff --git a/current-state.md b/current-state.md index 896fe22..c2c7f0e 100644 --- a/current-state.md +++ b/current-state.md @@ -3,9 +3,9 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-11-13 -verified_on: 2026-11-13 -review_after: 2026-11-13 +updated: 2026-11-14 +verified_on: 2026-11-14 +review_after: 2026-11-14 confidence: high tags: [system] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db] @@ -14,6 +14,15 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv # Current State Updated: 2026-11-13 +## Recent Changes (2026-11-13, nightly audit; live probes 2026-11-13 23:59 – 2026-11-14 00:2x AWST) + +- ✅ **2026-11-13 was the third consecutive zero-`failed` cron day: 174 rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit, `cbfe434a3b5e`, started 23:59:41). Trend: 11-11 **172 / 172**, 11-12 **176 / 176**, 11-13 **174 / 173**. **Deliveries 11-13: 8 rows, 8 / 8 `delivered`, 0 `for_failure`** (11-12: 9 / 9). Fleet **29 jobs (26 enabled / 3 disabled)** — an **eighth consecutive window** unchanged — and **every job reads `last_status: ok` with `failure_streak` 0, so the fleet has no non-clean row at all** for the first time in weeks. `cron_incidents` **0 unclosed**. → [[systems/scheduled-tasks]] +- ⚠ **The `pre_tool_call` hook timeout is back, and the raised budget did not hold it.** `_pre_tool_call timed out after 120s` at **2026-11-14 00:05:40**, then **10** `skipped after previous timeout or while still running` lines through **00:06:38** — refusing this audit's own first tool calls (first-hand: `skill_view` and `terminal` both returned the plugin timeout) for ~1 minute before dispatch recovered unaided. `errors.log` by day: 11-12 **0 / 0**, 11-13 **0 / 0**, 11-14 **1 / 10**; it is the **first `pre_tool_call` timeout in the retained log since the 2026-11-08 cascade** (whose own lines have since rotated out — `errors.log` now starts **2026-11-09 13:18:39**). **No cron run failed** — 11-13 settled `failed`-free. **The variable is load, now ~2× the level recorded at diagnosis (`load average 12.08 / 13.02 / 12.33` on 2 vCPU vs 5.89).** The 30 → 120 s fix is **proven insufficient**; the incident is **reopened, `resolved` → `active`**. → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] +- ⚠ **`state.db` is now 2.04 GB** (**2,036,908,032 B**; **2,977 sessions / 207,577 messages**) — up from **1.89 GB / 2,901 / 197,682** on 2026-11-09 — and it is the load source behind the reopened hook fault above. **Root filesystem 88 % used.** `state.db` **checkpoint/retention remains undecided** and is now the estate's most urgent open item; re-raising the hook budget again would only mask the fault. Still `journal_mode=wal`, `wal_autocheckpoint=2048`. → [[systems/hermes-agent]], [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] +- ✅ **The Uber Eats Monthly Spend Report was fixed end-to-end on 2026-11-13**, clearing the fleet's last non-clean row. Two stacked root causes: (1) the job's **runtime prompt was 88 k chars** because it carried two heavy browser skills, so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest (*"remaining 4 in the report"* plus forbidden run commentary); (2) the `delivery_failed` label was a **stale `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart — which happened **2026-11-13 13:14**. Fix: a deterministic digest printer **`~/.hermes/cron/scripts/uber-eats-digest.py`**, a rewritten prompt with a hard output contract, and the skills cleared (**88 k → 1.8 k chars**); the lesson went into the `cron-job-maintenance` skill. Verified on the **18:51 → 18:58** slot: **154 unique / 153 completed** orders, Oct = **9 orders / $301.04** (unchanged), delivery row `delivered`, `hermes cron doctor` clean across all 26 enabled jobs. → [[reports/uber-eats/index]], [[systems/scheduled-tasks]] +- **Freshness scan (2026-11-13 03:00 stocktake): 0 stale / 131 fresh / 3 no-date of 134 pages** — nothing needed retiring. → `raw/stocktake/freshness-20261113.txt` +- **`executions.db` retention floor moved to 2026-11-08T00:01:20+08:00** (was 2026-11-06T05:09:30). The 11-07/11-08 cascade **runs** are still readable, but their **`errors.log` lines are not** — so the incident page's 58-timeout figure stands on the 2026-11-09 read and was **not re-verifiable** this window. → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] + ## Recent Changes (2026-11-12, nightly audit; live probes 2026-11-12 23:59 – 2026-11-13 00:4x AWST) **Clean day. The estate is incident-clean, and the only open item is an upgrade that cannot be installed.** diff --git a/incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md b/incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md index e5b85b3..f717afd 100644 --- a/incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md +++ b/incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md @@ -1,11 +1,11 @@ --- title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08) type: incident -status: resolved +status: active created: 2026-11-09 -updated: 2026-11-09 -verified_on: 2026-11-09 -review_after: 2026-11-16 +updated: 2026-11-14 +verified_on: 2026-11-14 +review_after: 2026-11-17 confidence: high tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor] sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json] @@ -98,6 +98,36 @@ the error string does not name the hook. - **2026-11-09 was the first `failed`-free cron day since 2026-11-05** (11-06: 1 failed, 11-07: 3, 11-08: 1, 11-09: **0 of 174**), and unclosed incident rows went 3 → 0. - **Residual:** the two long-runners above are the same 9-hour-turn environment this bug leaves behind, and one (`wiki-stocktake-maintenance`) is pinned to `free-stack` — a combo recorded broken on 2026-08-14. The job-iteration budget stays in force. +## Recurrence (2026-11-14 nightly audit) — the raised budget did not hold + +**Reopened.** The `pre_tool_call` hook overran its **raised 120 s** budget again at **2026-11-14 00:05:40**, +on the wiki-audit job's own tool calls. The 30 -> 120 s fix of 2026-11-08 is therefore **proven +insufficient at current load**, and this page is no longer `resolved`. + +| time (AWST) | line | +|---|---| +| 2026-11-14 00:05:40,043 | `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` | +| 2026-11-14 00:05:43 -> 00:06:38 | **10** x `skipped after previous timeout or while still running`, ~6 s apart | + +- **Source:** `~/.hermes/logs/errors.log` (single-source count; `agent.log` carries the same lines). + By day: **2026-11-12 = 0 timeouts / 0 skips; 2026-11-13 = 0 / 0; 2026-11-14 = 1 / 10.** It is the + **first `pre_tool_call` timeout in the retained log since the 2026-11-08 cascade** — the 2026-11-08 + lines themselves have since **rotated out** of `errors.log`, so this page's original 58-timeout figure + still stands on the 2026-11-09 read and was **not re-verifiable** this window. +- **Cost this time:** this audit's first `skill_view` and `terminal` calls were **refused outright** by the + plugin (first-hand evidence), with dispatch recovering on its own by ~00:06:40 — roughly **one minute** + of hard suppression. It is a **single-event recurrence, not the 18-hour cascade**. **No cron run failed:** + 2026-11-13 settled **173 `completed` / 0 `failed`**. +- **Load is the variable, and it is now ~2x the level recorded when the fault was diagnosed:** + `load average **12.08 / 13.02 / 12.33**` on **2 vCPU** (2026-11-08 read: **5.89**). `state.db` has grown + **1.89 GB -> 2.04 GB** (**2,036,908,032 B**; **2,977 sessions / 207,577 messages**, was 2,901 / 197,682), + and the root filesystem is **88 % used**. +- **The earlier open hypothesis is now settled.** *"If it recurs while the hook budget is 120 s, this + hypothesis is dead"* — it recurred, so the load trigger is **confirmed** and there is **no headroom left + in the timeout**: at load ~12 the callback needs >120 s. **Re-raising the budget would only mask the + fault.** The remaining levers are unchanged and now urgent — **`state.db` checkpoint/retention** and + reducing host load (see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]). + ## See also - [[systems/conduit]] diff --git a/index.md b/index.md index 13fae20..193e774 100644 --- a/index.md +++ b/index.md @@ -3,8 +3,8 @@ title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 -updated: 2026-11-12 -verified_on: 2026-11-12 +updated: 2026-11-14 +verified_on: 2026-11-14 confidence: high tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout] sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake] @@ -12,7 +12,9 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-11-13. +> model routing and automation. Last updated: 2026-11-14. +> **2026-11-13 (nightly audit; live probes 2026-11-13 23:59 - 2026-11-14 00:2x AWST) — the fleet went fully clean, then the hook came back.** **2026-11-13: 174 ledger rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:41) — the **third consecutive zero-`failed` day** (11-11 172/172, 11-12 176/176); deliveries **8 / 8 `delivered`**, 0 `for_failure`; `cron_incidents` **0 unclosed**. **Every job now reads `last_status: ok` with `failure_streak` 0 — the fleet has no non-clean row at all**, because **`066bb5191280` (Uber Eats Monthly Spend Report) was fixed end-to-end on 2026-11-13**, clearing a `delivery_failed` that had stood since 2026-11-01. Two stacked causes: the job's **runtime prompt had grown to 88 k chars** (it carried two heavy browser skills) so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest, and the `delivery_failed` label was a **stale `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart (**2026-11-13 13:14**). Fix: deterministic printer **`~/.hermes/cron/scripts/uber-eats-digest.py`**, rewritten prompt with a hard output contract, skills cleared (**88 k -> 1.8 k**); verified on the 18:51 -> 18:58 slot (**154 unique / 153 completed** orders, Oct **9 orders / $301.04**), lesson saved to `cron-job-maintenance`. Fleet **29 (26 / 3)** — an **eighth** window unchanged. ⚠ **But the window's fault is the `pre_tool_call` hook, and it returned:** `_pre_tool_call timed out after 120s` at **2026-11-14 00:05:40** plus **10** suppression lines through **00:06:38**, refusing this audit's own tool calls for ~1 minute (first-hand). `errors.log` by day: 11-12 **0/0**, 11-13 **0/0**, 11-14 **1/10** — the first such timeout in the retained log since the 2026-11-08 cascade (**`errors.log` now starts 2026-11-09 13:18:39**, so the 11-08 lines have rotated out and the page's 58-count stands on the 2026-11-09 read). **Load is the trigger and it has ~doubled: `load average 12.08 / 13.02 / 12.33` on 2 vCPU** (vs **5.89** at diagnosis), with **`state.db` now 2.04 GB** (**2,977 sessions / 207,577 messages**, was 1.89 GB / 2,901 / 197,682) and the root filesystem **88 % used**. The 30 -> 120 s fix is **proven insufficient**; [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] is **reopened, `resolved` -> `active`**, and `state.db` checkpoint/retention — not a bigger timeout — is now the estate's most urgent open item. Freshness (03:00 stocktake): **0 stale / 131 fresh / 3 no-date of 134**. +> > **2026-11-11 → 2026-11-12 (nightly audit; live probes 2026-11-12 23:59 – 2026-11-13 00:4x AWST) — a clean day, and an upgrade that could not be installed.** The 429 row that was the estate's only open incident **closed `resolved` at 2026-11-12T00:25:36** (the next slot of the same job ran clean, carrying both windows); *cron_incidents* now reads **0 unclosed of 137**. **2026-11-12: 176 ledger rows — 175 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:54), deliveries **9 / 9 `delivered`**; **11-11: 172 / 172 `completed` / 0 `failed`**, deliveries 7 / 7. Fleet **29 (26 / 3)**, a seventh window unchanged. The window's one new open item is not a failure: the attempt to take **`hermes-hq-push`** past its installed **0.9.42** was **refused by the pre-install plugin security scan** — *"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."* The critical is the plugin's **own** `skills/dashboard-tile/SKILL.md` one-liner; **no per-verdict override exists** (`plugins.scan_on_install: false` would blind the scan for *every* future install, and `plugin_guard.py` hardcodes `trust_level="community"`). The fix is **PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file), pushed from a fork because `gh` is **read-only** upstream. Installed 0.9.42 keeps delivering. Also: the alleged *"stored hold-and-ask instruction"* for scan warnings **does not exist on disk** (184,716-file sweep, zero matches in editable text; the sentence lives only in `state.db` history) — nothing was removed or reverted, and the hunt is now the skill `stored-instruction-search-and-removal` (**skills 299 → 300**). `config.yaml` is **byte-identical to its 2026-11-12 18:14:52 backup**; the only drift since 11-09 is the **model catalogue block** (6 out, 9 in). **New page:** [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]]. > > @@ -77,7 +79,7 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi ## Incidents - [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]] - **OPEN — the `hermes-hq-push` upgrade past v0.9.42 was refused by the pre-install plugin security scan.** `hermes plugins install mrcharlesiv/hermes-hq-plugins/hermes-hq-push --enable --force` returned **"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."** The sole critical is the plugin's **own** `skills/dashboard-tile/SKILL.md:41` shell one-liner (`echo '' | python3 …` = *obfuscation*); the other 38 findings are low/medium and alone would have installed as `caution`. **`--force` lifts `caution` only**; a `dangerous` verdict is an unconditional block (`tools/plugin_guard.py:406-418`), and **no per-verdict override exists** — the only key is `plugins.scan_on_install` (default `true`, currently unset), which disables scanning for *every* future install; the `trust_level`/allowlist machinery lives in `tools/skills_guard.py` (the skills path) and `plugin_guard.py` hardcodes `trust_level="community"`. **Installed state unchanged:** v0.9.42, `enabled`, still delivering. **Fix = PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc` → `main`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file, created 2026-11-12T10:00:01Z) — pushed from a fork because this host's `gh` account has **read-only** access upstream; local clone `/home/hermes/workspace/hq-plugins-fix` (`90c9d28`). - [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **RESOLVED 2026-11-12T00:25:36** (self-cleared on the next slot's clean completion; *cron_incidents* now 0 unclosed of 137) **— the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and is **still open with `alerted_at: NULL` — the estate's only open incident row**. Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]). -- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - **RESOLVED (same day) — the hook was the fault, not the schedule.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** → [[systems/conduit]], [[systems/hermes-agent]] +- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - ⚠ **REOPENED 2026-11-14 — the hook was the fault, and the 30 → 120 s fix did not hold at current load.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** ⚠ **Recurred 2026-11-14 00:05:40** — **1 timeout + 10 skips**, overrunning the **120 s** budget at `load average 12.08` on 2 vCPU (**~2×** the 5.89 at diagnosis), with `state.db` grown to **2.04 GB**. Status **`resolved` → `active`**; the lever is now urgent. → [[systems/conduit]], [[systems/hermes-agent]] - [[incidents/2026-11-09-stocktake-completes-with-zero-edits]] (resolved 2026-11-11) - **RESOLVED BY ITSELF.** The nightly stocktake closed `completed` after **9 h 10 m** having landed **zero edits and no commit** — `failure_streak` 3 → 0 while the work stayed undone, and the top-priority item on its brief had been the monotonically worsening failure streak. Third consecutive day its findings never landed. - [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]] - **NO INCIDENT RAISED.** Job Radar Stage 1 overran to **9 h 09 m**, skipped its own verification, wrote **no handoff file**, and both Stage-2 slots ran before it finished and declined silently. No incident row was created — the completion-time-only alerting gap leaves the failure invisible. - [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] - **OPEN.** A restart requested at **~04:29:52** sat in *after-turn* mode for the full **1800 s**, then timed out and drained at **05:00:43/47** — the same `#77184` behaviour as the 10-31 reload, but this time it killed **two** in-flight cron executions: *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42.918 -> 05:02:22.870) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31.515 -> 05:02:22.174), both closed **`failed`** on `lost its durable fire claim ownership`; systemd restarted the unit at **05:01:00** (`restart counter is at 3`). A **third** failure that night was the agent-log ERROR watchdog correctly catching the event 19 s later. The request landed **inside the `conduit_push` self-update window** (04:27:55 -> 04:32:32, its own ledger recording `restarting_gateway` = *failed*) — the **working hypothesis for the requester, not proven**; the 0.22.5 -> 0.23.0 updater is excluded (receipt starts 05:05:13). ⚠ **Closed 0 -> 2 incident rows and neither alerted**: both `1a8e90_*` rows sit `detected` with `alerted_at: null`. The stocktake slot has now failed **two days running** (`failure_streak: 2`). Job Radar ran clean. → [[systems/scheduled-tasks]], [[systems/hermes-agent]], [[systems/conduit]], [[systems/mnemosyne]] diff --git a/log.md b/log.md index 1984cc6..393fd3e 100644 --- a/log.md +++ b/log.md @@ -2547,3 +2547,61 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted* `incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md`, `systems/hermes-hq-push.md`, `systems/scheduled-tasks.md`, `current-state.md`, `index.md`, `log.md`, `systems/skills-index.md` (regenerated). **No page deleted, no decision reversed, no secret written.** + +## 2026-11-13 23:5x – 2026-11-14 00:2x AWST — Nightly audit (window 2026-11-12 23:59 → 2026-11-13 23:59) + +**Headline: a third consecutive `failed`-free day and a fully clean fleet — then the `pre_tool_call` +hook timeout came back and reopened its incident.** + +- **Ledger — 2026-11-13: 174 rows, 173 `completed` / 0 `failed` / 1 `running`** (this audit, + `cbfe434a3b5e`, started 23:59:41). Trend: 11-11 **172 / 172**, 11-12 **176 / 176**, 11-13 **174 / 173** + — the **third consecutive zero-`failed` day**. Deliveries **8 rows / 8 `delivered` / 0 `for_failure`** + (11-12: 9 / 9). *cron_incidents* **0 unclosed**. +- **Fleet 29 (26 enabled / 3 disabled)** — an **eighth** consecutive window at the same count, nothing + added, retired, disabled or repinned. **Every job now reads `last_status: ok` with `failure_streak` 0: + the fleet has no non-clean row at all.** The last one cleared because **`066bb5191280` (Uber Eats Monthly + Spend Report) was fixed end-to-end on 2026-11-13** — so [[systems/scheduled-tasks]]'s *"the fleet's only + non-clean `last_status`"* is now a historical statement, not a current one. +- **Uber Eats fix, recorded for the audit trail.** Two stacked root causes: (1) the job's **runtime prompt + had grown to 88 k chars** (it carried `camofox-browser-automation` + `browser-backend-verification`), so + the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest (*"remaining 4 in + the report"*) plus forbidden run commentary; (2) the `delivery_failed` label was a **stale + `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart — which happened + **2026-11-13 13:14**. Fix: a deterministic digest printer + **`~/.hermes/cron/scripts/uber-eats-digest.py`**, a rewritten prompt with a hard output contract, and the + skills cleared (**88 k → 1.8 k chars**); the lesson went into the `cron-job-maintenance` skill. Verified on + the **18:51 → 18:58** slot — **154 unique / 153 completed** orders, Oct **9 orders / $301.04** (unchanged), + delivery row `delivered`, `hermes cron doctor` clean across all 26 enabled jobs. Pages already regenerated + in-window by that work: [[reports/uber-eats/index]], `reports/uber-eats/2026-10.md` + `.html` + (commits `bc35c4b`, `a0ec3a1`). **This audit updated that job's row in [[systems/scheduled-tasks]].** +- ⚠ **[[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] REOPENED — `resolved` → `active`.** The + hook overran its **raised 120 s** budget at **2026-11-14 00:05:40**, followed by **10** *"skipped after + previous timeout or while still running"* lines through **00:06:38**. It refused **this audit's own** first + tool calls (first-hand: `skill_view` and `terminal` both returned the plugin timeout) for ~1 minute before + dispatch recovered unaided. `errors.log` single-source counts: **11-12 0 / 0, 11-13 0 / 0, 11-14 1 / 10** — + the **first `pre_tool_call` timeout in the retained log since the 2026-11-08 cascade**. **No cron run + failed.** ⚠ **The 2026-11-08 lines have rotated out of `errors.log`** (which now starts **2026-11-09 + 13:18:39**), so the page's 58-timeout figure **stands on the 2026-11-09 read and was not re-verifiable** + this window — recorded on the page rather than silently recopied. +- ⚠ **Load is the trigger and it has roughly doubled.** `load average` **12.08 / 13.02 / 12.33** on **2 vCPU** + (2026-11-08 read: **5.89**). **`state.db` is now 2.04 GB** (**2,036,908,032 B**; **2,977 sessions / + 207,577 messages** — was **1.89 GB / 2,901 / 197,682** on 11-09), and the root filesystem is **88 % used**. + The **30 → 120 s fix is proven insufficient**, which settles the page's open hypothesis in the confirmatory + direction: at load ~12 the callback needs >120 s, so **there is no headroom left in the timeout**, and + re-raising it would only mask the fault. **`state.db` checkpoint/retention is now the estate's most urgent + open item.** +- **Freshness (2026-11-13 03:00 stocktake; file mtime 03:06:30): 0 stale / 131 fresh / 3 no-date of 134 + pages.** Nothing needed retiring. → `raw/stocktake/freshness-20261113.txt` +- **Stocktake 2026-11-13 recovered to a normal window:** `1a8e90ea2271` ran **03:01:07 → 03:17:14 = + 16 m 07 s**, `completed`, `last_status ok`, `failure_streak 0` (11-12: 22 m 10 s; the 11-08 failure was + 1 h 34 m). It found **0 stale pages**, so there was nothing to re-verify and it **left no commit of its own + and no section in this log** — its freshness output was byte-identical to the copy already committed by + `e033a08`. Recorded as a **minor process gap, not a failure**: unlike the 2026-11-09 *"completes with zero + edits"* incident, there was genuinely nothing in scope. +- **`executions.db` retention floor moved to 2026-11-08T00:01:20+08:00** (was 2026-11-06T05:09:30), so the + 11-07/11-08 cascade **runs** remain readable even though their log lines have rotated. +- **Pages touched:** `incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md` (frontmatter `resolved` → + `active`, plus a **Recurrence (2026-11-14)** section), `systems/scheduled-tasks.md` (new *Current as of + 2026-11-13* section, the Uber Eats job row, frontmatter), `current-state.md` (new *Recent Changes + (2026-11-13)* section), `index.md` (new blurb, the incident's RESOLVED → REOPENED label, frontmatter), + `log.md` (this entry). **No page deleted, no decision reversed, no secret written.** diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index c21c6c9..0b4db5f 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,9 +3,9 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-11-13 -verified_on: 2026-11-13 -review_after: 2026-11-13 +updated: 2026-11-14 +verified_on: 2026-11-14 +review_after: 2026-11-14 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db] @@ -14,6 +14,25 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db # Scheduled Tasks ## Count +- **Current as of 2026-11-13 (window 2026-11-12 23:59 -> 2026-11-13 23:59; nightly audit).** **A third + consecutive zero-`failed` day — and the fleet went fully clean.** Fleet **29 (26 enabled / 3 disabled)**, + an **eighth consecutive window at the same count** (nothing added, retired, disabled or repinned). + Ledger — **2026-11-13: 174 rows, 173 `completed` / 0 `failed` / 1 `running`** (this audit, `cbfe434a3b5e`, + started 23:59:41). Trend: **11-11 172 / 172, 11-12 176 / 176, 11-13 174 / 173**. + **Deliveries 11-13: 8 rows, 8 / 8 `delivered`, 0 `for_failure`** (11-12: 9 / 9). **Every job now reads + `last_status: ok` and `failure_streak` 0 — the fleet has no non-clean row at all**, clearing + `066bb5191280` (**Uber Eats Monthly Spend Report**), which had carried `delivery_failed` since + **2026-11-01** and was **fixed end-to-end on 2026-11-13** (see the job table). + **Unclosed incident rows: 0** (unchanged). + ⚠ **The window's one fault is not a job: the `pre_tool_call` hook timeout returned.** `_pre_tool_call + timed out after 120s` at **2026-11-14 00:05:40**, then **10** suppression lines through **00:06:38** + — refusing this audit's own tool calls for ~1 minute. `errors.log` by day: 11-12 **0 / 0**, + 11-13 **0 / 0**, 11-14 **1 / 10**. **Load is the variable and it has ~doubled** (`load average 12.08 / + 13.02 / 12.33` on **2 vCPU**; 2026-11-08 read 5.89) with `state.db` at **2.04 GB** + (**2,977 sessions / 207,577 messages**, was 1.89 GB / 2,901 / 197,682) and the root filesystem **88 % used**. + The 30 -> 120 s fix is **proven insufficient**; the incident is **reopened**, `resolved` -> `active` + -> [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. Freshness scan (03:00 stocktake): + **0 stale / 131 fresh / 3 no-date of 134 pages** (`raw/stocktake/freshness-20261113.txt`). - **Current as of 2026-11-12 (window 2026-11-11 23:59 → 2026-11-12 23:59; nightly audit).** **A clean day — and an upgrade that could not happen.** Fleet **29 (26 enabled / 3 disabled)**, a seventh consecutive window at the same count. Ledger — **2026-11-11: 172 rows, 172 `completed` / 0 `failed`**; @@ -246,7 +265,7 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne | Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | ⚠ Two node faults, no code fault: 2026-10-15 08:06 `RuntimeError: Connection error.` (`0e5198_be2291809e56`) and **2026-10-19 08:01 `HTTP 503 … resource pressure`** (new incident `0e5198_ec2fac98a91d`, `alerted`). Both are open; `failure_streak: 2`. Next slot Thu 2026-10-22 08:00 | ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "`failure_streak: 2`" note above is **superseded**; the job cleared on its Mon/Thu 08:00 slots after 2026-10-19. | | Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) | | Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | ⚠ **2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `85edf6_256c5e150bc4`, `alerted`) — its first failure since its 10-15 connection-refused red cleared on 10-16. Live `last_status: error`, streak 1, next slot 2026-10-20 07:00 | ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, streak 1" note above is **superseded**; the job cleared on its own daily 07:00 slot after 2026-10-19. | -| Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram:1793951355 | camofox-browser-automation, browser-backend-verification | — | ✅ **Fixed 2026-11-01:** delivery retargeted to `telegram:1793951355` (was a bare `telegram` target that inherited the dead `thread_id 34795` → "configured thread_id 34795 … was not found; delivered without thread_id"). Also repaired the cron prompt, which pointed at **port 9093** (nonexistent) instead of **9377**, and wired in the tested pipeline `scrape-uber-eats-orders.py` → `build-uber-eats-report.py` → `generate-uber-eats-report.py`. The Oct 2026 report was generated and verified against the live Uber DOM → [[reports/uber-eats/index]]. | +| Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram:1793951355 | — (two heavy browser skills **cleared 2026-11-13**, 88k -> 1.8k chars) | — | ✅ **Fixed 2026-11-01:** delivery retargeted to `telegram:1793951355` (was a bare `telegram` target that inherited the dead `thread_id 34795` → "configured thread_id 34795 … was not found; delivered without thread_id"). Also repaired the cron prompt, which pointed at **port 9093** (nonexistent) instead of **9377**, and wired in the tested pipeline `scrape-uber-eats-orders.py` → `build-uber-eats-report.py` → `generate-uber-eats-report.py`. The Oct 2026 report was generated and verified against the live Uber DOM → [[reports/uber-eats/index]]. ✅ **Fixed again 2026-11-13 (two stacked causes).** (1) The runtime prompt was **88 k chars** because the job carried two heavy browser skills, so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest (*"remaining 4 in the report"* plus forbidden run commentary). (2) The `delivery_failed` label was a **stale `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart — which happened **2026-11-13 13:14**. Fix: a deterministic digest printer **`~/.hermes/cron/scripts/uber-eats-digest.py`**, a rewritten prompt with a hard output contract, and the skills cleared. Verified on the **18:51 -> 18:58** slot: **154 unique / 153 completed** orders, Oct = **9 orders / $301.04** (unchanged), delivery row `delivered`, and `last_status` now **`ok`**. Lesson recorded in the `cron-job-maintenance` skill. | | Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the `cliprox/terra` pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ **Repinned 2026-10-05 23:39** → `custom:omni-101` / `free-stack` and **verified by a live manual run**; the incident auto-resolved. Now `last_status: ok`, streak 0 | | weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 | | Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes. ⚠ **The 2026-10-16 23:59 slot failed** on the `omni-101` 503, so the 2026-10-15 → 2026-10-17 span went unaudited; incident `cbfe43_9b89947a046a` is **`detected`** and clears only on the next successful 23:59 run (the 2026-10-17 run, which produced this entry). **Not** a stale session lock — the execution row carries the explicit 503 error text |