audit 2026-11-02: Weekly Review lost to gateway handoff failure; three cron workers SIGTERM'd

Window 2026-11-01 00:00 -> 2026-11-02 00:4x AWST. 167 completed / 1 running / 2 unknown.

NEW incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (active)
NEW incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md (resolved)
UPDATED systems/scheduled-tasks.md (Count -> 2026-11-02 window, Re-verified section)
UPDATED systems/messaging-integrations.md (Telegram flapping, relay adapter absent ~29h)
UPDATED index.md, log.md
This commit is contained in:
Tony0410
2026-11-02 00:27:09 +08:00
parent 8bb3d196b1
commit fda56e53fe
6 changed files with 383 additions and 9 deletions
+44
View File
@@ -1810,3 +1810,47 @@ recorded wrong.**
CT201:8102 returns **307 → 200** on `/board`, so the dashboard page's "live, 200" claim still holds. CT201's
`:8443/` root returns **502** (upstream 8082 has no listener) and `:8443/mgr` returns **200** — both still as
the homepage-dashboard page records. No secrets, tokens, passwords or keys written.
## [2026-11-02] audit | Nightly review — the Weekly Review never ran, three workers killed by `-15`, and a red status with no row behind it
**Window:** 2026-11-01 00:00 → 2026-11-02 00:4x AWST. Live probes 00:05–00:55 on `state.db` (sessions),
`~/.hermes/cron/{jobs.json,executions.db,deliveries.db}`, `gateway.log` / `agent.log.1`, and both git trees.
- **NEW — `incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md`** (`status: active`):
the Sunday 18:00 `79f9bb41ecf5` closed `unknown` at 18:05:13 with *"Gateway connection failed; no local
fallback"*, **delivered nothing**, and its incident row `79f9bb_9c443b602eec` is **still `alerted`**.
`deliveries.db` has **no row after 11:01:12**. Batch-mate `b63ed5817212` hit the identical error and
self-healed in 68 s. The same page records the **inverse** silent failure: `ecc1fc13655a` reads
`error`, `last_run 11-01T12:01:00`, `failure_streak: 1` — with **no execution row and no incident row**.
- **NEW — `incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md`** (`status: resolved`): three
workers `SIGTERM`ed (`-15`) after adopting their claim (`85edf6772bca` 07:30:46; `96eb95ce12f2` +
`0e753e9720fd` 09:02:18) → *"whether side effects ran is unknown"*. **Fourth variant of the orphan
class, and the first with no gateway restart/reload/drain anywhere in the window** (PID `175269` up
since 10-31 19:24:40; the relay counter ran monotonic through both signals). Probable trigger: a
**30 s worker-ack window vs a ~70 s cold start** (`Plugin discovery complete: 83 found, 69 enabled`).
All three incident rows `resolved`; one execution left `unknown`.
- **UPDATED — `systems/scheduled-tasks.md`**: `## Count` advanced to the 2026-11-02 window (**167
`completed` / 1 `running` / 2 `unknown`** vs 161/4 on 10-31, 138/0 on 10-30, 177/0 on 10-29 — the
first non-clean day in four); fleet **29 jobs / 26 enabled / 3 disabled** unchanged; **unclosed
incident rows 1 → 1 but a different row** (all three rows the 10-31 audit left open closed on 11-01,
including **`0e753e_e7f8194a7d06`**, open since **2026-10-01** — 31 days); new `## Re-verified
2026-11-02` section covering the three failure modes, the 09:00 cold-start pair and the ledger.
**Non-clean `last_status` 4 → 2** (`ea79a2107e36`, `79f9bb41ecf5`).
- **UPDATED — `systems/messaging-integrations.md`**: Telegram degraded all day (dual-stack
`api.telegram.org` `ConnectTimeout` → IPv4-literal fallback; disconnect 20:51:53, polling restarts
22:33:30 / 23:12:23), and `getent hosts` currently returns **only an AAAA record** for
`api.telegram.org` — consistent with, but not proof of, a systematic timeout on the path DNS hands
the client. The **`relay` platform has had no adapter for ~29 h** (`Reconnecting relay` attempt
**147 @10-31 18:39:24 → 336 @11-02 00:10:47**, counter never reset — itself the proof of no gateway
restart). `066bb5191280`'s `thread_id 34795` home-DM miss is the **second confirmed variant** of the
thread-mismatch bug: the thread-less fallback fires, the content lands, and the **run is recorded
`failed` anyway**. Day's delivery health: **8 rows / 7 `delivered` / 1 `failed`**.
- **UPDATED — `index.md`**: `updated`/`verified_on` → 2026-11-02, the audit summary line rewritten for
this window, and the two new incident links added to the incident list.
- **Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned;
no decision reversed in the window. A full vault mirror pass was deliberately skipped (drift counts
below instead of a bulk copy) — an audit's mandate is smaller than a full sync.
- **DRIFT (context, not findings):** the authoring tree was clean and in sync with Gitea
(`origin/master..HEAD` empty) before this audit; touched pages were mirrored into the vault's
existing conventions and hash-verified. **No secrets, passwords, tokens or keys written or quoted** —
every job id, PID, thread id and timestamp above is a non-sensitive operational identifier.