audit(2026-11-13): third zero-failed day and a fully clean fleet; the Uber Eats fix clears the last non-clean cron row — and the pre_tool_call hook timeout recurred at load 12.08 (incident reopened resolved -> active, state.db 2.04 GB)

This commit is contained in:
Tony0410
2026-11-14 00:27:05 +08:00
parent a0ec3a1ec5
commit b0e4a54f8a
5 changed files with 133 additions and 15 deletions
+23 -4
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-11-13
verified_on: 2026-11-13
review_after: 2026-11-13
updated: 2026-11-14
verified_on: 2026-11-14
review_after: 2026-11-14
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
@@ -14,6 +14,25 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
# Scheduled Tasks
## Count
- **Current as of 2026-11-13 (window 2026-11-12 23:59 -> 2026-11-13 23:59; nightly audit).** **A third
consecutive zero-`failed` day — and the fleet went fully clean.** Fleet **29 (26 enabled / 3 disabled)**,
an **eighth consecutive window at the same count** (nothing added, retired, disabled or repinned).
Ledger — **2026-11-13: 174 rows, 173 `completed` / 0 `failed` / 1 `running`** (this audit, `cbfe434a3b5e`,
started 23:59:41). Trend: **11-11 172 / 172, 11-12 176 / 176, 11-13 174 / 173**.
**Deliveries 11-13: 8 rows, 8 / 8 `delivered`, 0 `for_failure`** (11-12: 9 / 9). **Every job now reads
`last_status: ok` and `failure_streak` 0 — the fleet has no non-clean row at all**, clearing
`066bb5191280` (**Uber Eats Monthly Spend Report**), which had carried `delivery_failed` since
**2026-11-01** and was **fixed end-to-end on 2026-11-13** (see the job table).
**Unclosed incident rows: 0** (unchanged).
⚠ **The window's one fault is not a job: the `pre_tool_call` hook timeout returned.** `_pre_tool_call
timed out after 120s` at **2026-11-14 00:05:40**, then **10** suppression lines through **00:06:38**
— refusing this audit's own tool calls for ~1 minute. `errors.log` by day: 11-12 **0 / 0**,
11-13 **0 / 0**, 11-14 **1 / 10**. **Load is the variable and it has ~doubled** (`load average 12.08 /
13.02 / 12.33` on **2 vCPU**; 2026-11-08 read 5.89) with `state.db` at **2.04 GB**
(**2,977 sessions / 207,577 messages**, was 1.89 GB / 2,901 / 197,682) and the root filesystem **88 % used**.
The 30 -> 120 s fix is **proven insufficient**; the incident is **reopened**, `resolved` -> `active`
-> [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. Freshness scan (03:00 stocktake):
**0 stale / 131 fresh / 3 no-date of 134 pages** (`raw/stocktake/freshness-20261113.txt`).
- **Current as of 2026-11-12 (window 2026-11-11 23:59 → 2026-11-12 23:59; nightly audit).** **A clean day
— and an upgrade that could not happen.** Fleet **29 (26 enabled / 3 disabled)**, a seventh consecutive
window at the same count. Ledger — **2026-11-11: 172 rows, 172 `completed` / 0 `failed`**;
@@ -246,7 +265,7 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne
| Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | ⚠ Two node faults, no code fault: 2026-10-15 08:06 `RuntimeError: Connection error.` (`0e5198_be2291809e56`) and **2026-10-19 08:01 `HTTP 503 … resource pressure`** (new incident `0e5198_ec2fac98a91d`, `alerted`). Both are open; `failure_streak: 2`. Next slot Thu 2026-10-22 08:00 | ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "`failure_streak: 2`" note above is **superseded**; the job cleared on its Mon/Thu 08:00 slots after 2026-10-19. |
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) |
| Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | ⚠ **2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `85edf6_256c5e150bc4`, `alerted`) — its first failure since its 10-15 connection-refused red cleared on 10-16. Live `last_status: error`, streak 1, next slot 2026-10-20 07:00 | ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, streak 1" note above is **superseded**; the job cleared on its own daily 07:00 slot after 2026-10-19. |
| Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram:1793951355 | camofox-browser-automation, browser-backend-verification | — | ✅ **Fixed 2026-11-01:** delivery retargeted to `telegram:1793951355` (was a bare `telegram` target that inherited the dead `thread_id 34795` → "configured thread_id 34795 … was not found; delivered without thread_id"). Also repaired the cron prompt, which pointed at **port 9093** (nonexistent) instead of **9377**, and wired in the tested pipeline `scrape-uber-eats-orders.py` → `build-uber-eats-report.py` → `generate-uber-eats-report.py`. The Oct 2026 report was generated and verified against the live Uber DOM → [[reports/uber-eats/index]]. |
| Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram:1793951355 | — (two heavy browser skills **cleared 2026-11-13**, 88k -> 1.8k chars) | — | ✅ **Fixed 2026-11-01:** delivery retargeted to `telegram:1793951355` (was a bare `telegram` target that inherited the dead `thread_id 34795` → "configured thread_id 34795 … was not found; delivered without thread_id"). Also repaired the cron prompt, which pointed at **port 9093** (nonexistent) instead of **9377**, and wired in the tested pipeline `scrape-uber-eats-orders.py` → `build-uber-eats-report.py` → `generate-uber-eats-report.py`. The Oct 2026 report was generated and verified against the live Uber DOM → [[reports/uber-eats/index]]. ✅ **Fixed again 2026-11-13 (two stacked causes).** (1) The runtime prompt was **88 k chars** because the job carried two heavy browser skills, so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest (*"remaining 4 in the report"* plus forbidden run commentary). (2) The `delivery_failed` label was a **stale `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart — which happened **2026-11-13 13:14**. Fix: a deterministic digest printer **`~/.hermes/cron/scripts/uber-eats-digest.py`**, a rewritten prompt with a hard output contract, and the skills cleared. Verified on the **18:51 -> 18:58** slot: **154 unique / 153 completed** orders, Oct = **9 orders / $301.04** (unchanged), delivery row `delivered`, and `last_status` now **`ok`**. Lesson recorded in the `cron-job-maintenance` skill. |
| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the `cliprox/terra` pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ **Repinned 2026-10-05 23:39** → `custom:omni-101` / `free-stack` and **verified by a live manual run**; the incident auto-resolved. Now `last_status: ok`, streak 0 |
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes. ⚠ **The 2026-10-16 23:59 slot failed** on the `omni-101` 503, so the 2026-10-15 → 2026-10-17 span went unaudited; incident `cbfe43_9b89947a046a` is **`detected`** and clears only on the next successful 23:59 run (the 2026-10-17 run, which produced this entry). **Not** a stale session lock — the execution row carries the explicit 503 error text |