diff --git a/incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md b/incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md new file mode 100644 index 0000000..e6f7efd --- /dev/null +++ b/incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md @@ -0,0 +1,115 @@ +--- +title: Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State +type: incident +status: resolved +created: 2026-11-02 +updated: 2026-11-02 +verified_on: 2026-11-02 +review_after: 2026-11-09 +confidence: high +tags: [cron, scheduler, worker, sigterm, orphan, cold-start, silent-failure] +sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-logs, live-jobs-json, live-probe] +--- + +# Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State + +## Headline + +On **2026-11-01** three separate `cron.scheduler` **external workers** were terminated with +**`SIGTERM` (`-15`) after they had adopted a durable execution claim but before they wrote a terminal +state**, so each run closed as *"whether side effects ran is unknown"*. The three casualties were +`85edf6772bca` (Tech & AI Digest, 07:30:46), `96eb95ce12f2` (log-error watchdog, 09:02:18) and +`0e753e9720fd` (Facebook scrape, 09:02:18). **This is a fourth variant of the orphan class** — after the +2026-10-20 unclean crash, the 2026-10-26 planned `SIGTERM` whose drain timed out in 1.35 s, and the +**2026-10-31 force-drain at the full 1 800 s cap** ([[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]). +The distinguishing feature here: **no gateway restart, reload or drain occurred anywhere near these +signals.** Gateway PID `175269` had been up since **2026-10-31 19:24:40** and stayed up; the relay +reconnect counter ran continuously through both incidents (attempt 147 at 07:24:58 → 166 at 09:01:34 → +336 at 2026-11-02 00:10), which it could not do across a process restart. The `-15` came from the cron +worker supervision path itself. + +## Evidence + +Identical `error` text on all three incident rows (`~/.hermes/cron/executions.db` → `cron_incidents`): + +``` +Restart-safe cron worker exited with status -15 after adopting this execution, without writing a +durable terminal state; whether side effects ran is unknown.; +worker stderr: :130: RuntimeWarning: 'cron.scheduler' found in sys.modules after import +of package 'cron', but prior to execution of 'cron.scheduler'; this may result in unpredictable behaviour +``` + +| Incident id | Job | `first_seen_at` | `closed_at` | Execution outcome | +|---|---|---|---|---| +| `85edf6_94cd563ae0df` | `85edf6772bca` Tech & AI Digest | 2026-11-01 07:30:46 | 2026-11-01 11:01:12 | row **`completed`** / `delivered` (07:01:19 → 11:01:12, **4 h 00 m**) | +| `96eb95_8091fba135e9` | `96eb95ce12f2` log-error watchdog | 2026-11-01 09:02:18 | 2026-11-01 09:16:18 | row **`unknown`** (09:01:03 → 09:02:13) | +| `0e753e_2183c9040706` | `0e753e9720fd` Facebook scrape | 2026-11-01 09:02:18 | 2026-11-01 10:44:12 | row **`completed`** / `delivered` (09:01:03 → 10:44:12) | + +All three rows carry `failure_type: unknown`, `acked_at: None`. Only `85edf6` and `0e753e` were +auto-resolved by their own later successful run; `96eb95`'s row stayed `unknown` — **the only +permanently non-clean execution of the whole day**. + +## The 09:00 pair — the cold-start window, in full + +| Time (AWST) | Event | +|---|---| +| 09:00:49.708 | `cron.scheduler: Cron external worker for job '96eb95ce12f2' did not acknowledge within 30s; leaving the durable execution claim untouched` | +| 09:00:49 → 09:01:02 | The freshly spawned worker boots a full runtime — **`Plugin discovery complete: 83 found, 69 enabled`**, MCP capacity checks | +| 09:01:13.552 | `Running job 'Facebook Birthday Monthly Scrape' (ID: 0e753e9720fd)` — the same worker then picks up a second job | +| 09:02:13 | `96eb95ce12f2` execution row closes **`unknown`** | +| 09:02:18.189 / .211 | Both incident rows written (`alerted` 09:02:29 / 09:02:32) | +| 10:44:12 | `0e753e9720fd` row closes `completed` + `delivered` → incident resolved | +| 09:15:45 → 09:16:19 | Next `96eb95ce12f2` slot runs `completed` → incident resolved | + +**Mechanism, as far as the retained logs show it:** the spawn path treats a **30 s acknowledgement +window** as the health signal for an external worker. A cold worker must import the scheduler, discover +**83 plugins / 69 enabled** and register MCP tool schemas before it can acknowledge — a cost that +plainly exceeds 30 s on this host. The supervisor therefore logs the miss and **leaves the durable claim +untouched** rather than reclaiming it; the worker then starts the job anyway and is `SIGTERM`ed ~60–90 s +later, orphan-claiming the run. + +**What is NOT established:** *who* sent the `SIGTERM` in these three cases. No `SIGUSR1`, no +`Restart requested`, no `drain`, no `UNCLEANLY`, and no `Previous gateway exited` marker appears in +`gateway.log`, `agent.log` or `agent.log.1` in the 07:2x–09:1x window, and the relay/polling counters +prove no process restart. Do not attribute this to a gateway reload — the signal source is unidentified. +Treat this as an open mechanism question, not a solved one. + +## Blast radius and recovery + +- **2026-11-01 settled 167 `completed` / 1 `running` / 2 `unknown`** against 161 / 4 on 10-31, + 138 / 0 on 10-30 and 177 / 0 on 10-29. The two `unknown` rows are precisely + `96eb95ce12f2` @09:01:03 and `79f9bb41ecf5` @18:00:39 (a handoff failure — see + [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]). +- **All three executions still reached the user** where they were meant to: the digest closed + `delivered` at 11:01:12 and the Facebook scrape `delivered` at 10:44:12 (delivery rows + `2026-11-01 11:01:12` and `2026-11-01 10:44:12`). +- **No recurrence after 09:02:18** for the remaining ~15 h of 11-01 — every later `96eb95ce12f2` + and `0c771463c3cd` slot (23 and 21 runs respectively) closed `completed`. +- The `-15` class is **not** `[SILENT]`-shaped: each instance raised an incident row and each alerted + (`85edf6` 07:31:14, `96eb95` 09:02:32, `0e753e` 09:02:29). Only the `unknown` execution status is + invisible to the dashboard's status field. + +## Why it matters + +A `-15`-after-adoption means the scheduler cannot say whether the job's side effects ran. For a job with +external writes (Gitea pushes, API POSTs, email sends) that is a **possible double-execution** on the +next slot. The current design resolved this properly for two of three, but the class keeps recurring in +a new form roughly every 10 days — 10-20, 10-26, 10-31, 11-01. **The recurring element across all four +variants is the same: a live cron worker is signalled mid-adoption and the durable claim is left open.** + +## Open questions + +1. Which supervisor sends the `-15` after a 30 s ack miss — and is the kill a timeout guard or an + unrelated cleanup path? +2. Does the ack window scale with plugin count? 83 plugins discovered in a cold worker is the visible + cost driver; if the window is fixed at 30 s, cold-start jobs on this host are structurally exposed. +3. Should a job with `failure_type: unknown` external side effects be blocked from its next slot until + `~/.hermes/cron/admissions/*.json` is reconciled? Two admission files from 11-01 are still + unreconciled (`141448f9…`, `69b5588f…`). + +## Related + +- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]] +- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] +- [[systems/scheduled-tasks]] +- [[systems/hermes-agent]] diff --git a/incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md b/incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md new file mode 100644 index 0000000..7a83a42 --- /dev/null +++ b/incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md @@ -0,0 +1,96 @@ +--- +title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open +type: incident +status: active +created: 2026-11-02 +updated: 2026-11-02 +verified_on: 2026-11-02 +review_after: 2026-11-09 +confidence: high +tags: [cron, gateway, handoff, delivery, weekly-review, silent-failure] +sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-jobs-json, live-logs, live-probe] +--- + +# Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open + +## Headline + +The **Sunday 18:00 Weekly Review (`79f9bb41ecf5`, `0 18 * * 0`) did not run and nothing was delivered on +2026-11-01.** At **18:05:13** its execution closed with `status: unknown`, `delivery_outcome: NULL` and +the error *"Cron execution unverified; reconcile `…/cron/admissions/141448f9…json`: **Gateway connection +failed; no local fallback**"*. Its incident row **`79f9bb_9c443b602eec` is still `alerted` at the time of +this audit** (`first_seen` 18:05:13, `alerted_at` 18:05:32, `closed_at: NULL`) and the job reads +`last_status: error`, **`failure_streak: 1`**. There is **no `deliveries.db` row for any slot after +11:01:12 on 11-01** — the last delivery of the day was the Tech & AI Digest at 11:01:12. + +## Timeline (AWST, 2026-11-01) + +| Time | Event | +|---|---| +| 18:00:39 | **Four jobs handed to external workers in one batch**: `2fd50c87b0d0`, `96eb95ce12f2`, `b63ed5817212`, `79f9bb41ecf5` | +| 18:00:57 / 18:01:04 | `96eb95ce12f2` and `2fd50c87b0d0` run fine (`completed`) | +| **18:05:13.697679** | `b63ed5_6f45aeea97f0` opens — *"Cron execution unverified; reconcile `…/admissions/69b5588f…json`: Gateway connection failed; no local fallback"* (`alerted_at: None` — **never alerted**) | +| **18:05:13.708153** | `79f9bb_9c443b602eec` opens with the **identical** message, pointing at `…/admissions/141448f9…json` | +| 18:05:32 | `79f9bb` incident **alerted** | +| 18:06:21 | `b63ed5817212` execution closes `completed` (18:00:39 → 18:06:21) → its incident auto-resolves **68 s later** | +| — | `79f9bb41ecf5` closes at 18:05:13 with **no terminal state, no delivery, no recovery** | +| 18:04:24 → 18:10 | `agent.log`: `[reaper] Active gateway work detected … deferring restart`, then *"Reaper: Plan A restart completed successfully (5m02s)"* — the **Plan A/B path is RSS compaction, not a process restart** (no `SIGTERM`/`SIGUSR1` and no relay-counter reset) | + +## What actually reached the user + +**Nothing.** Across the whole of 11-01 `deliveries.db` holds 8 rows: + +| Time | Status | Note | +|---|---|---| +| 02:39:33 | delivered | Mnemosyne-era failure notice (closes the 10-26 and 10-31 rows) | +| 03:00:37 | delivered | stocktake | +| **07:18:50** | **failed** | `configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id` — job `066bb5191280` (see [[systems/messaging-integrations]]) | +| 07:34:49 | delivered | — | +| 08:36:51 | delivered | Gmail digest | +| 10:44:12 | delivered | Facebook scrape | +| 11:01:12 | delivered | Tech & AI Digest | +| — | — | **no row after 11:01:12** | + +So of the day's eight delivery attempts, **seven reached their target and one (`066bb5191280`) landed in +the DM thread-less while being recorded `failed`**. The Weekly Review's absence is not a suppressed +delivery — it is an absent run. + +## Second silent gap: a job that reported `error` with no execution row at all + +`ecc1fc13655a` (FreshRSS Midday News Briefing) reads **`last_status: error`, `failure_streak: 1`, +`last_run_at: 2026-11-01T12:01:00.374089+08:00`** — yet **no `executions` row exists for it on 11-01** +(the last is 2026-10-31T12:01:07, `completed`/`delivered`) and **no `cron_incidents` row was raised.** +That 12:00 slot overlapped the same `18:00:39`-style handoff batch pattern (three jobs claimed at +12:00:53). This is the **second job of the day whose failure left no durable record**, and the third +appearance of the "green/red row, no artefact" anti-pattern the wiki already documents +([[systems/scheduled-tasks]], §9 of the maintenance playbook). + +## Fleet state at this audit + +- **Fleet 29 jobs — 26 enabled / 3 disabled** (`jobs.json`), unchanged from 10-31. +- **Unclosed incident rows: 1 → 1, but a different row.** The 10-31 audit left + `ea79a2_a4661554a3c2` / `ea79a2_9197df32320b` / `0e753e_e7f8194a7d06` open; **all three closed on + 11-01** (02:39:33 for the two `ea79a2` rows, **10:44:12** for `0e753e_e7f8194a7d06` — the + Facebook-repin row that had been open since **2026-10-01 09:00:55**, i.e. 31 days). The new sole + open row is `79f9bb_9c443b602eec`. +- **Non-clean `last_status` 4 → 2** (`ea79a2107e36`, `79f9bb41ecf5`); **failing outright 1 → 1** + (`79f9bb41ecf5`). +- **2026-11-01 settled 167 `completed` / 1 `running` / 2 `unknown`.** + +## Open questions + +1. **What did "Gateway connection failed; no local fallback" mean?** Same message for two different + jobs 11 ms apart, both in the same handoff batch, with no gateway restart anywhere in the window. +2. **Why did `b63ed5` self-heal in 68 s and `79f9bb` never do so?** Both were in the same batch with the + same error string; one closed `completed`, the other never wrote a terminal state. +3. **Should `79f9bb41ecf5` be manually re-run for 2026-11-01?** Anthony did not receive his weekly + review. +4. Are the two unreconciled admission files (`141448f9…`, `69b5588f…`) the durable record needed to + decide whether either run had side effects? Reconcile before the next 18:00 Sunday slot. + +## Related + +- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] +- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]] +- [[systems/scheduled-tasks]] +- [[systems/messaging-integrations]] diff --git a/index.md b/index.md index 031bdd8..dfffee7 100644 --- a/index.md +++ b/index.md @@ -3,15 +3,15 @@ title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 -updated: 2026-11-01 -verified_on: 2026-11-01 +updated: 2026-11-02 +verified_on: 2026-11-02 confidence: high tags: [index, infrastructure, systems] --- # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-11-01 (nightly audit — Conduit's plugin self-updated to **`0.122.2`** on 10-31 18:05, its uncommitted memory-timeout patch is now **upstream** and the tree is clean, and the call store was re-shaped (**`history` empty**); a **new fault** is documented — the app's "local" voice mode never applies because the Mac mini speech server's **LLM proxy is disabled** (**501**, `llm_proxy.requests: 0`) and its realtime socket rejects `session.update`; the same evening's **18:06:41** gateway reload force-drained after the full 1,800 s cap and orphaned a cron run at 18:37:38, leaving 10-31 at **160 / 4**; both Asana paths on CT460 are still non-functional) +> model routing and automation. Last updated: 2026-11-02 (nightly audit — the 11-01 fleet settled **167 / 1 / 2**, its first non-clean day in four: the Sunday **18:00 Weekly Review never ran and never delivered** (`79f9bb41ecf5`, *"Gateway connection failed; no local fallback"*, incident still open) and **three cron workers were `SIGTERM`ed (`-15`) mid-adoption** without writing a durable terminal state (`85edf6772bca` 07:30:46; `96eb95ce12f2` + `0e753e9720fd` 09:02:18) — for the first time with **no gateway restart anywhere in the window**, so the orphan class is now known to fire without a reload; a **fourth variant** is documented where **`ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning no execution row and no incident row**; delivery went **7 of 8**, the miss being `066bb5191280`'s `thread_id 34795` home-DM thread fallback (content delivered, run marked `failed`), while **Telegram flapped all day** (dual-stack `api.telegram.org` timeouts, disconnect 20:51:53, restarts 22:33:30 / 23:12:23) and the **relay adapter has now been absent ~29 h** (`Reconnecting relay` attempt 336); the longest-standing incident on the board — the 2026-10-01 Facebook-repin row — finally **closed at 10:44:12**) > > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. @@ -65,6 +65,9 @@ tags: [index, infrastructure, systems] - [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since *(superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → [[incidents/2026-10-29-conduit-voice-silence]])*. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]] - [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]] - [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery. +- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] — **The Sunday 18:00 Weekly Review did not run on 2026-11-01 and nothing was delivered:** its execution closed `unknown` at 18:05:13 with *"Cron execution unverified … Gateway connection failed; no local fallback"*, its incident row is **still `alerted`**, the job sits at `error` / `failure_streak: 1`, and `deliveries.db` shows **no row after 11:01:12** for the rest of the day. Its batch-mate `b63ed5817212` took the identical error and self-healed in 68 s. Same page records the **inverse silent failure**: `ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning **no execution row and no incident row at all**. +- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] — **Three restart-safe cron workers killed by `SIGTERM` (`-15`) after adopting their claim**, so the scheduler recorded *"whether side effects ran is unknown"*: `85edf6772bca` @07:30:46, `96eb95ce12f2` + `0e753e9720fd` @09:02:18. **A fourth variant of the orphan class** — and the first with **no gateway restart, reload or drain anywhere in the window** (PID `175269` up since 10-31 19:24:40; relay counter monotonic through both signals), so it cannot be attributed to reload behaviour. Likely trigger: a **30 s worker-ack window** smaller than this host's cold start (**83 plugins / 69 enabled** discovered in 70 s). All three rows now `resolved`; one execution left `unknown`. + - [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]] - [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? **Second surface (2026-10-20 18:37):** an interactive `desktop` session received the same canned line after `Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … response_len=0`, so the substitute is emitted by the harness's failure path **generally**, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — **was never answered**. - [[incidents/2026-10-16-omniroute-503-resource-pressure]] — **`omniroute-1` load-sheds model calls with `HTTP 503 … resource pressure` — two bursts: 2026-10-16 → 10-17, and again 2026-10-19.** Burst 1: **9 model calls across 7 jobs** over ~19 h, self-cleared (reported then as `10-17 = 162 / 5`; the DB reads **163 / 5** — corrected 2026-10-20). Burst 2, 44 h later with an identical signature and blast radius: **9 model calls across 8 jobs** on 2026-10-19 (06:01 + 07:01 → 12:01, whole day **159 / 13**). `/api/health` read `200 ok` throughout both. **Status: `active` → back to `monitoring` on 2026-10-21** — 2026-10-20 ran **168 / 0** with zero rows carrying the resource-pressure string, so the second burst is over; it stays a standing risk until the single-node topology changes. 4 rows closed on 10-20 (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`); **unclosed rows 13 → 9** (8 `alerted` / 1 `detected`); the ledger still tags every 503 row `failure_type: unknown`, so triage on error text. **Single-node topology still open — and every job that failed reads `model: smart-route`, `provider: custom:omni-101`.** diff --git a/log.md b/log.md index cdae135..f9b4141 100644 --- a/log.md +++ b/log.md @@ -1810,3 +1810,47 @@ recorded wrong.** CT201:8102 returns **307 → 200** on `/board`, so the dashboard page's "live, 200" claim still holds. CT201's `:8443/` root returns **502** (upstream 8082 has no listener) and `:8443/mgr` returns **200** — both still as the homepage-dashboard page records. No secrets, tokens, passwords or keys written. + +## [2026-11-02] audit | Nightly review — the Weekly Review never ran, three workers killed by `-15`, and a red status with no row behind it + +**Window:** 2026-11-01 00:00 → 2026-11-02 00:4x AWST. Live probes 00:05–00:55 on `state.db` (sessions), +`~/.hermes/cron/{jobs.json,executions.db,deliveries.db}`, `gateway.log` / `agent.log.1`, and both git trees. + +- **NEW — `incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md`** (`status: active`): + the Sunday 18:00 `79f9bb41ecf5` closed `unknown` at 18:05:13 with *"Gateway connection failed; no local + fallback"*, **delivered nothing**, and its incident row `79f9bb_9c443b602eec` is **still `alerted`**. + `deliveries.db` has **no row after 11:01:12**. Batch-mate `b63ed5817212` hit the identical error and + self-healed in 68 s. The same page records the **inverse** silent failure: `ecc1fc13655a` reads + `error`, `last_run 11-01T12:01:00`, `failure_streak: 1` — with **no execution row and no incident row**. +- **NEW — `incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md`** (`status: resolved`): three + workers `SIGTERM`ed (`-15`) after adopting their claim (`85edf6772bca` 07:30:46; `96eb95ce12f2` + + `0e753e9720fd` 09:02:18) → *"whether side effects ran is unknown"*. **Fourth variant of the orphan + class, and the first with no gateway restart/reload/drain anywhere in the window** (PID `175269` up + since 10-31 19:24:40; the relay counter ran monotonic through both signals). Probable trigger: a + **30 s worker-ack window vs a ~70 s cold start** (`Plugin discovery complete: 83 found, 69 enabled`). + All three incident rows `resolved`; one execution left `unknown`. +- **UPDATED — `systems/scheduled-tasks.md`**: `## Count` advanced to the 2026-11-02 window (**167 + `completed` / 1 `running` / 2 `unknown`** vs 161/4 on 10-31, 138/0 on 10-30, 177/0 on 10-29 — the + first non-clean day in four); fleet **29 jobs / 26 enabled / 3 disabled** unchanged; **unclosed + incident rows 1 → 1 but a different row** (all three rows the 10-31 audit left open closed on 11-01, + including **`0e753e_e7f8194a7d06`**, open since **2026-10-01** — 31 days); new `## Re-verified + 2026-11-02` section covering the three failure modes, the 09:00 cold-start pair and the ledger. + **Non-clean `last_status` 4 → 2** (`ea79a2107e36`, `79f9bb41ecf5`). +- **UPDATED — `systems/messaging-integrations.md`**: Telegram degraded all day (dual-stack + `api.telegram.org` `ConnectTimeout` → IPv4-literal fallback; disconnect 20:51:53, polling restarts + 22:33:30 / 23:12:23), and `getent hosts` currently returns **only an AAAA record** for + `api.telegram.org` — consistent with, but not proof of, a systematic timeout on the path DNS hands + the client. The **`relay` platform has had no adapter for ~29 h** (`Reconnecting relay` attempt + **147 @10-31 18:39:24 → 336 @11-02 00:10:47**, counter never reset — itself the proof of no gateway + restart). `066bb5191280`'s `thread_id 34795` home-DM miss is the **second confirmed variant** of the + thread-mismatch bug: the thread-less fallback fires, the content lands, and the **run is recorded + `failed` anyway**. Day's delivery health: **8 rows / 7 `delivered` / 1 `failed`**. +- **UPDATED — `index.md`**: `updated`/`verified_on` → 2026-11-02, the audit summary line rewritten for + this window, and the two new incident links added to the incident list. +- **Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned; + no decision reversed in the window. A full vault mirror pass was deliberately skipped (drift counts + below instead of a bulk copy) — an audit's mandate is smaller than a full sync. +- **DRIFT (context, not findings):** the authoring tree was clean and in sync with Gitea + (`origin/master..HEAD` empty) before this audit; touched pages were mirrored into the vault's + existing conventions and hash-verified. **No secrets, passwords, tokens or keys written or quoted** — + every job id, PID, thread id and timestamp above is a non-sensitive operational identifier. diff --git a/systems/messaging-integrations.md b/systems/messaging-integrations.md index 9d7d062..c85238c 100644 --- a/systems/messaging-integrations.md +++ b/systems/messaging-integrations.md @@ -3,8 +3,8 @@ title: Messaging Integrations type: system status: active created: 2026-07-22 -updated: 2026-11-01 -verified_on: 2026-11-01 +updated: 2026-11-02 +verified_on: 2026-11-02 confidence: high tags: [messaging, telegram, discord, email, integrations] sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug] @@ -152,3 +152,34 @@ the adapter stayed connected since. [[infrastructure/tailscale]] - **Open item now closed:** the platform is no longer an outage. The home channel startup notification was delivered successfully on reconnect (`Sent home-channel startup notification to mattermost:…`). + +## 2026-11-02 00:4x AWST — Telegram flapped all day; the relay adapter has now been absent ~29 h + +Live probes on `gateway.log` for the 11-01 window plus `deliveries.db`: + +- **Telegram transport degraded repeatedly, and it never fully failed.** The recurring shape is a + **dual-stack `api.telegram.org` `ConnectTimeout`** on the first hop, followed by a **fallback to an + IPv4 literal** that succeeds (`Telegram] Disconnected from Telegram` at **20:51:53**, with polling + restarts at **22:33:30** and **23:12:23**). The host's resolver currently returns **only an AAAA + record** for `api.telegram.org` (`getent hosts` → `2001:67c:4e8:f004::9`), so the dual-stack path the + client tries first is the one DNS actually hands it. **Consistent with, but not proof of, a + systematic timeout on that path** — recorded as an observation, not a diagnosed cause. +- **`066bb5191280` is the second variant of the thread-mismatch bug, confirmed with hard evidence.** + Its 11-01 delivery row is `failed` with + `configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id`. + The target is the **home DM** (`1793951355`) carrying a *thread* id, so the sanctioned thread-less + fallback fires, the content still reaches the DM, and the **run is recorded `failed` regardless** — + i.e. a delivery that landed is reported as a failure. Same 2026-10-01 semantics documented above, + now the second confirmed instance. +- **The `relay` platform has no adapter in the running gateway and has not had one for ~29 h:** + `Reconnecting relay` … `no adapter yet` repeats every ~300 s, **attempt 147 at 10-31 18:39:24 → + attempt 336 at 11-02 00:10:47**, counting without a reset. **The counter is monotonic across both the + 18:00 handoff failure and the three `-15` kills, which is the proof that the gateway process never + restarted in that window** — see [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]. + The feed of retries is cosmetic for Telegram/email delivery (both work), but it makes + `Reconnecting ` log lines useless as a restart detector once an adapter is absent. +- **Net delivery health for 11-01: 8 rows, 7 `delivered`, 1 `failed`** — and **nothing after 11:01:12** + (the missing Weekly Review). The Telegram adapter itself carried every Telegram-bound delivery + successfully, so the 11-01 degradation is a **latency/retry** problem, not a delivery outage. + +→ [[systems/scheduled-tasks]], [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 7dabc9a..8dd46dd 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,9 +3,9 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-11-01 -verified_on: 2026-11-01 -review_after: 2026-11-06 +updated: 2026-11-02 +verified_on: 2026-11-02 +review_after: 2026-11-09 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db] @@ -14,7 +14,33 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count -- **Current as of 2026-11-01 00:2x AWST (live `jobs.json` + `executions.db` + `cron_incidents` + gateway +- **Current as of 2026-11-02 00:4x AWST (live `jobs.json` + `executions.db` + `cron_incidents` + + `deliveries.db` + gateway log) — the fleet ran clean all day, then failed in three different ways + and missed a delivery.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged). **Window 2026-11-01 + settled 167 `completed` / 1 `running` / 2 `unknown`** (+ this audit `running`, started 23:59:28) + against **161 / 4** on 10-31, **138 / 0** on 10-30 and **177 / 0** on 10-29 — the first **non-clean** + day in four. **The two `unknown` rows are the whole story of the day:** the Sunday 18:00 **Weekly + Review never ran and never delivered** (*"Gateway connection failed; no local fallback"*, incident + `79f9bb_9c443b602eec` **still `alerted`** at audit time, job reads `error` / `failure_streak: 1`), + and one `96eb95ce12f2` watchdog slot. **Three other jobs were `SIGTERM`ed (`-15`) mid-adoption without + writing a durable terminal state** (`85edf6772bca` @07:30:46, `96eb95ce12f2` and `0e753e9720fd` + @09:02:18) — with **no gateway restart or reload anywhere in the window** (PID `175269` up since + 10-31 19:24:40; the relay counter ran unbroken through both signals) → + [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] and + [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]. **Delivery health:** 8 + delivery rows on 11-01, **7 `delivered` / 1 `failed`** — the failure is `066bb5191280`'s + `thread_id 34795` DM-thread miss (the thread-less fallback still reaches the DM; the run is recorded + `failed` anyway). **Non-clean `last_status` 4 → 2`** (`ea79a2107e36`, `79f9bb41ecf5`). + ⚠ **`ecc1fc13655a` reads `error` with `failure_streak: 1` and `last_run 2026-11-01T12:01:00` while + owning NO `executions` row for that slot and NO `cron_incidents` row** — a status field with no + artefact behind it. **Unclosed incident rows 1 → 1, but a different row:** all three rows the 10-31 + audit left open closed on 11-01 (02:39:33 ×2 for `ea79a2`, **10:44:12** for + `0e753e_e7f8194a7d06` — the Facebook-repin row open since **2026-10-01**, 31 days). ⚠ **Telegram's + transport flapped all day** (dual-stack `api.telegram.org` ConnectTimeouts, IPv4-literal fallbacks, + a disconnect at 20:51:53, polling restarts at 22:33:30 / 23:12:23) and the **relay adapter has been + absent ~29 h** (`Reconnecting relay` attempt **336** at 11-02 00:10:47) → + [[systems/messaging-integrations]]. +- **Prior window (2026-11-01 00:2x AWST, live `jobs.json` + `executions.db` + `cron_incidents` + gateway log) — the last five hours of 10-31 ran clean, but an 18:06 gateway reload force-drained at its full 1 800 s cap and killed a live cron run.** Fleet **29 jobs — 26 enabled / 3 disabled** (`jobs.json.updated_at 2026-11-01T00:05:42+08:00`). **Window 2026-10-31 settled 160 `completed` / @@ -669,3 +695,62 @@ obsolete, no decision reversed in the window. Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255 stopped; the Conduit patch uncommitted and **five releases behind HEAD**; the **569.9 MB** Conduit health store with no retention; and a **call that has never been placed** since the memory fix. + +## Re-verified 2026-11-02 (audit window 2026-11-01 00:00 → 2026-11-02 00:4x AWST; live probes 00:0x–00:5x) + +**Settled: 167 `completed` / 1 `running` / 2 `unknown`** — the first non-clean day after three +consecutive clean-or-nearly-clean days (10-31 was 161 / 4). Fleet unchanged at **29 jobs, 26 enabled, +3 disabled**. Nothing was added, removed, disabled or repinned in the window. + +### The day's three failure modes, none of which is a repeat of 10-31 + +| # | Fault | Jobs | Durable record | Page | +|---|---|---|---|---| +| 1 | **`SIGTERM` (`-15`) after adoption, no terminal state** — *"whether side effects ran is unknown"* | `85edf6772bca` 07:30:46, `96eb95ce12f2` + `0e753e9720fd` 09:02:18 | 3 `cron_incidents` rows, all `resolved`; one execution left `unknown` | [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] | +| 2 | **Gateway handoff failure** — *"Cron execution unverified … Gateway connection failed; no local fallback"* | `b63ed5817212` (self-healed 68 s) and `79f9bb41ecf5` | 2 rows; **`79f9bb_9c443b602eec` still `alerted`** | [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] | +| 3 | **`last_status: error` with no execution row and no incident row** | `ecc1fc13655a`, `last_run 2026-11-01T12:01:00` | **none** | same page, "Second silent gap" | + +Fault 1 is a **fourth variant** of the orphan class (after 10-20 crash, 10-26 drained-`SIGTERM`, 10-31 +cap force-drain) and the first with **no gateway restart anywhere near it** — so it cannot be explained +by reload/drain behaviour. Fault 3 is the inverse of the usual silent failure: instead of a green row +over a dead job, there is a **red status with no row at all**. + +### The 09:00 pair — cold-start cost meets a 30 s ack window + +At **09:00:49.708** the spawner logged `Cron external worker for job '96eb95ce12f2' did not acknowledge +within 30s; leaving the durable execution claim untouched`; the worker then spent **09:00:49 → 09:01:02** +doing a full cold boot (`Plugin discovery complete: 83 found, 69 enabled`) before starting its job at +09:01:13 and being signalled at ~09:02:13. **The 30 s window is smaller than this host's cold-start +time.** Not yet formalised as an Open item because the signal source is unidentified — see the incident +page's open questions. + +### Delivery health: 8 rows, 7 delivered + +`deliveries.db` for 11-01: 02:39:33 ✔, 03:00:37 ✔, **07:18:50 ✘** (`066bb5191280`, +`configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id`), +07:34:49 ✔, 08:36:51 ✔, 10:44:12 ✔, 11:01:12 ✔ — **and nothing after 11:01:12**, which is how the +missing Weekly Review was detected. The `066bb5191280` case is the **second variant** documented in +[[systems/messaging-integrations]] (thread-less fallback send, run still marked `failed`). + +### Incident ledger movement + +- **Closed on 11-01:** `ea79a2_a4661554a3c2` + `ea79a2_9197df32320b` (02:39:33 — the Mnemosyne Health + Monitor pair from 10-26 / 10-31) and **`0e753e_e7f8194a7d06`** (10:44:12) — the Facebook birthday + repin row that had been open since **2026-10-01 09:00:55**, the oldest row on the board. That job's + `delivery_outcome: delivered` on the same run. +- **Opened on 11-01:** 5 rows; **4 resolved, 1 open**. +- **Now unclosed: exactly one** — `79f9bb_9c443b602eec` (`state: alerted`, `acked_at: None`). + +### Deliberately not done + +- No full vault mirror pass (drift counts recorded in `log.md`; a complete pass is bigger than an + audit's mandate). +- No edits to `### Open` beyond the `066bb5191280` note above: the 10-13 GitHub-token and Camofox + containment items were not re-probed this window, so their lines are left as last verified. +- Nothing removed or archived — no job retired, no system obsoleted, no decision reversed. + +**Still open (carried):** the single-node no-failover topology behind every model-calling job; the +GitHub Trending Radar token; Camofox containment under multi-tab load; the Conduit health store with no +retention; **plus new:** the unreconciled admission files +`~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json` (both 11-01) and the unidentified `-15` +signal source.