audit 2026-11-02: Weekly Review lost to gateway handoff failure; three cron workers SIGTERM'd

Window 2026-11-01 00:00 -> 2026-11-02 00:4x AWST. 167 completed / 1 running / 2 unknown.

NEW incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (active)
NEW incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md (resolved)
UPDATED systems/scheduled-tasks.md (Count -> 2026-11-02 window, Re-verified section)
UPDATED systems/messaging-integrations.md (Telegram flapping, relay adapter absent ~29h)
UPDATED index.md, log.md
This commit is contained in:
Tony0410
2026-11-02 00:27:09 +08:00
parent 8bb3d196b1
commit fda56e53fe
6 changed files with 383 additions and 9 deletions
@@ -0,0 +1,115 @@
---
title: Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State
type: incident
status: resolved
created: 2026-11-02
updated: 2026-11-02
verified_on: 2026-11-02
review_after: 2026-11-09
confidence: high
tags: [cron, scheduler, worker, sigterm, orphan, cold-start, silent-failure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-logs, live-jobs-json, live-probe]
---
# Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State
## Headline
On **2026-11-01** three separate `cron.scheduler` **external workers** were terminated with
**`SIGTERM` (`-15`) after they had adopted a durable execution claim but before they wrote a terminal
state**, so each run closed as *"whether side effects ran is unknown"*. The three casualties were
`85edf6772bca` (Tech & AI Digest, 07:30:46), `96eb95ce12f2` (log-error watchdog, 09:02:18) and
`0e753e9720fd` (Facebook scrape, 09:02:18). **This is a fourth variant of the orphan class** — after the
2026-10-20 unclean crash, the 2026-10-26 planned `SIGTERM` whose drain timed out in 1.35 s, and the
**2026-10-31 force-drain at the full 1 800 s cap** ([[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]).
The distinguishing feature here: **no gateway restart, reload or drain occurred anywhere near these
signals.** Gateway PID `175269` had been up since **2026-10-31 19:24:40** and stayed up; the relay
reconnect counter ran continuously through both incidents (attempt 147 at 07:24:58 → 166 at 09:01:34 →
336 at 2026-11-02 00:10), which it could not do across a process restart. The `-15` came from the cron
worker supervision path itself.
## Evidence
Identical `error` text on all three incident rows (`~/.hermes/cron/executions.db` → `cron_incidents`):
```
Restart-safe cron worker exited with status -15 after adopting this execution, without writing a
durable terminal state; whether side effects ran is unknown.;
worker stderr: <frozen runpy>:130: RuntimeWarning: 'cron.scheduler' found in sys.modules after import
of package 'cron', but prior to execution of 'cron.scheduler'; this may result in unpredictable behaviour
```
| Incident id | Job | `first_seen_at` | `closed_at` | Execution outcome |
|---|---|---|---|---|
| `85edf6_94cd563ae0df` | `85edf6772bca` Tech & AI Digest | 2026-11-01 07:30:46 | 2026-11-01 11:01:12 | row **`completed`** / `delivered` (07:01:19 → 11:01:12, **4 h 00 m**) |
| `96eb95_8091fba135e9` | `96eb95ce12f2` log-error watchdog | 2026-11-01 09:02:18 | 2026-11-01 09:16:18 | row **`unknown`** (09:01:03 → 09:02:13) |
| `0e753e_2183c9040706` | `0e753e9720fd` Facebook scrape | 2026-11-01 09:02:18 | 2026-11-01 10:44:12 | row **`completed`** / `delivered` (09:01:03 → 10:44:12) |
All three rows carry `failure_type: unknown`, `acked_at: None`. Only `85edf6` and `0e753e` were
auto-resolved by their own later successful run; `96eb95`'s row stayed `unknown` — **the only
permanently non-clean execution of the whole day**.
## The 09:00 pair — the cold-start window, in full
| Time (AWST) | Event |
|---|---|
| 09:00:49.708 | `cron.scheduler: Cron external worker for job '96eb95ce12f2' did not acknowledge within 30s; leaving the durable execution claim untouched` |
| 09:00:49 → 09:01:02 | The freshly spawned worker boots a full runtime — **`Plugin discovery complete: 83 found, 69 enabled`**, MCP capacity checks |
| 09:01:13.552 | `Running job 'Facebook Birthday Monthly Scrape' (ID: 0e753e9720fd)` — the same worker then picks up a second job |
| 09:02:13 | `96eb95ce12f2` execution row closes **`unknown`** |
| 09:02:18.189 / .211 | Both incident rows written (`alerted` 09:02:29 / 09:02:32) |
| 10:44:12 | `0e753e9720fd` row closes `completed` + `delivered` → incident resolved |
| 09:15:45 → 09:16:19 | Next `96eb95ce12f2` slot runs `completed` → incident resolved |
**Mechanism, as far as the retained logs show it:** the spawn path treats a **30 s acknowledgement
window** as the health signal for an external worker. A cold worker must import the scheduler, discover
**83 plugins / 69 enabled** and register MCP tool schemas before it can acknowledge — a cost that
plainly exceeds 30 s on this host. The supervisor therefore logs the miss and **leaves the durable claim
untouched** rather than reclaiming it; the worker then starts the job anyway and is `SIGTERM`ed ~60–90 s
later, orphan-claiming the run.
**What is NOT established:** *who* sent the `SIGTERM` in these three cases. No `SIGUSR1`, no
`Restart requested`, no `drain`, no `UNCLEANLY`, and no `Previous gateway exited` marker appears in
`gateway.log`, `agent.log` or `agent.log.1` in the 07:2x–09:1x window, and the relay/polling counters
prove no process restart. Do not attribute this to a gateway reload — the signal source is unidentified.
Treat this as an open mechanism question, not a solved one.
## Blast radius and recovery
- **2026-11-01 settled 167 `completed` / 1 `running` / 2 `unknown`** against 161 / 4 on 10-31,
138 / 0 on 10-30 and 177 / 0 on 10-29. The two `unknown` rows are precisely
`96eb95ce12f2` @09:01:03 and `79f9bb41ecf5` @18:00:39 (a handoff failure — see
[[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]).
- **All three executions still reached the user** where they were meant to: the digest closed
`delivered` at 11:01:12 and the Facebook scrape `delivered` at 10:44:12 (delivery rows
`2026-11-01 11:01:12` and `2026-11-01 10:44:12`).
- **No recurrence after 09:02:18** for the remaining ~15 h of 11-01 — every later `96eb95ce12f2`
and `0c771463c3cd` slot (23 and 21 runs respectively) closed `completed`.
- The `-15` class is **not** `[SILENT]`-shaped: each instance raised an incident row and each alerted
(`85edf6` 07:31:14, `96eb95` 09:02:32, `0e753e` 09:02:29). Only the `unknown` execution status is
invisible to the dashboard's status field.
## Why it matters
A `-15`-after-adoption means the scheduler cannot say whether the job's side effects ran. For a job with
external writes (Gitea pushes, API POSTs, email sends) that is a **possible double-execution** on the
next slot. The current design resolved this properly for two of three, but the class keeps recurring in
a new form roughly every 10 days — 10-20, 10-26, 10-31, 11-01. **The recurring element across all four
variants is the same: a live cron worker is signalled mid-adoption and the durable claim is left open.**
## Open questions
1. Which supervisor sends the `-15` after a 30 s ack miss — and is the kill a timeout guard or an
unrelated cleanup path?
2. Does the ack window scale with plugin count? 83 plugins discovered in a cold worker is the visible
cost driver; if the window is fixed at 30 s, cold-start jobs on this host are structurally exposed.
3. Should a job with `failure_type: unknown` external side effects be blocked from its next slot until
`~/.hermes/cron/admissions/*.json` is reconciled? Two admission files from 11-01 are still
unreconciled (`141448f9…`, `69b5588f…`).
## Related
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]
- [[systems/scheduled-tasks]]
- [[systems/hermes-agent]]
@@ -0,0 +1,96 @@
---
title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open
type: incident
status: active
created: 2026-11-02
updated: 2026-11-02
verified_on: 2026-11-02
review_after: 2026-11-09
confidence: high
tags: [cron, gateway, handoff, delivery, weekly-review, silent-failure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-jobs-json, live-logs, live-probe]
---
# Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open
## Headline
The **Sunday 18:00 Weekly Review (`79f9bb41ecf5`, `0 18 * * 0`) did not run and nothing was delivered on
2026-11-01.** At **18:05:13** its execution closed with `status: unknown`, `delivery_outcome: NULL` and
the error *"Cron execution unverified; reconcile `…/cron/admissions/141448f9…json`: **Gateway connection
failed; no local fallback**"*. Its incident row **`79f9bb_9c443b602eec` is still `alerted` at the time of
this audit** (`first_seen` 18:05:13, `alerted_at` 18:05:32, `closed_at: NULL`) and the job reads
`last_status: error`, **`failure_streak: 1`**. There is **no `deliveries.db` row for any slot after
11:01:12 on 11-01** — the last delivery of the day was the Tech & AI Digest at 11:01:12.
## Timeline (AWST, 2026-11-01)
| Time | Event |
|---|---|
| 18:00:39 | **Four jobs handed to external workers in one batch**: `2fd50c87b0d0`, `96eb95ce12f2`, `b63ed5817212`, `79f9bb41ecf5` |
| 18:00:57 / 18:01:04 | `96eb95ce12f2` and `2fd50c87b0d0` run fine (`completed`) |
| **18:05:13.697679** | `b63ed5_6f45aeea97f0` opens — *"Cron execution unverified; reconcile `…/admissions/69b5588f…json`: Gateway connection failed; no local fallback"* (`alerted_at: None` — **never alerted**) |
| **18:05:13.708153** | `79f9bb_9c443b602eec` opens with the **identical** message, pointing at `…/admissions/141448f9…json` |
| 18:05:32 | `79f9bb` incident **alerted** |
| 18:06:21 | `b63ed5817212` execution closes `completed` (18:00:39 → 18:06:21) → its incident auto-resolves **68 s later** |
| — | `79f9bb41ecf5` closes at 18:05:13 with **no terminal state, no delivery, no recovery** |
| 18:04:24 → 18:10 | `agent.log`: `[reaper] Active gateway work detected … deferring restart`, then *"Reaper: Plan A restart completed successfully (5m02s)"* — the **Plan A/B path is RSS compaction, not a process restart** (no `SIGTERM`/`SIGUSR1` and no relay-counter reset) |
## What actually reached the user
**Nothing.** Across the whole of 11-01 `deliveries.db` holds 8 rows:
| Time | Status | Note |
|---|---|---|
| 02:39:33 | delivered | Mnemosyne-era failure notice (closes the 10-26 and 10-31 rows) |
| 03:00:37 | delivered | stocktake |
| **07:18:50** | **failed** | `configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id` — job `066bb5191280` (see [[systems/messaging-integrations]]) |
| 07:34:49 | delivered | — |
| 08:36:51 | delivered | Gmail digest |
| 10:44:12 | delivered | Facebook scrape |
| 11:01:12 | delivered | Tech & AI Digest |
| — | — | **no row after 11:01:12** |
So of the day's eight delivery attempts, **seven reached their target and one (`066bb5191280`) landed in
the DM thread-less while being recorded `failed`**. The Weekly Review's absence is not a suppressed
delivery — it is an absent run.
## Second silent gap: a job that reported `error` with no execution row at all
`ecc1fc13655a` (FreshRSS Midday News Briefing) reads **`last_status: error`, `failure_streak: 1`,
`last_run_at: 2026-11-01T12:01:00.374089+08:00`** — yet **no `executions` row exists for it on 11-01**
(the last is 2026-10-31T12:01:07, `completed`/`delivered`) and **no `cron_incidents` row was raised.**
That 12:00 slot overlapped the same `18:00:39`-style handoff batch pattern (three jobs claimed at
12:00:53). This is the **second job of the day whose failure left no durable record**, and the third
appearance of the "green/red row, no artefact" anti-pattern the wiki already documents
([[systems/scheduled-tasks]], §9 of the maintenance playbook).
## Fleet state at this audit
- **Fleet 29 jobs — 26 enabled / 3 disabled** (`jobs.json`), unchanged from 10-31.
- **Unclosed incident rows: 1 → 1, but a different row.** The 10-31 audit left
`ea79a2_a4661554a3c2` / `ea79a2_9197df32320b` / `0e753e_e7f8194a7d06` open; **all three closed on
11-01** (02:39:33 for the two `ea79a2` rows, **10:44:12** for `0e753e_e7f8194a7d06` — the
Facebook-repin row that had been open since **2026-10-01 09:00:55**, i.e. 31 days). The new sole
open row is `79f9bb_9c443b602eec`.
- **Non-clean `last_status` 4 → 2** (`ea79a2107e36`, `79f9bb41ecf5`); **failing outright 1 → 1**
(`79f9bb41ecf5`).
- **2026-11-01 settled 167 `completed` / 1 `running` / 2 `unknown`.**
## Open questions
1. **What did "Gateway connection failed; no local fallback" mean?** Same message for two different
jobs 11 ms apart, both in the same handoff batch, with no gateway restart anywhere in the window.
2. **Why did `b63ed5` self-heal in 68 s and `79f9bb` never do so?** Both were in the same batch with the
same error string; one closed `completed`, the other never wrote a terminal state.
3. **Should `79f9bb41ecf5` be manually re-run for 2026-11-01?** Anthony did not receive his weekly
review.
4. Are the two unreconciled admission files (`141448f9…`, `69b5588f…`) the durable record needed to
decide whether either run had side effects? Reconcile before the next 18:00 Sunday slot.
## Related
- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]]
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
- [[systems/scheduled-tasks]]
- [[systems/messaging-integrations]]
+6 -3
View File
@@ -3,15 +3,15 @@ title: Anthony's Systems Wiki
type: collection type: collection
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-01 updated: 2026-11-02
verified_on: 2026-11-01 verified_on: 2026-11-02
confidence: high confidence: high
tags: [index, infrastructure, systems] tags: [index, infrastructure, systems]
--- ---
# Anthony's Systems Wiki # Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure, > Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-11-01 (nightly audit — Conduit's plugin self-updated to **`0.122.2`** on 10-31 18:05, its uncommitted memory-timeout patch is now **upstream** and the tree is clean, and the call store was re-shaped (**`history` empty**); a **new fault** is documented — the app's "local" voice mode never applies because the Mac mini speech server's **LLM proxy is disabled** (**501**, `llm_proxy.requests: 0`) and its realtime socket rejects `session.update`; the same evening's **18:06:41** gateway reload force-drained after the full 1,800 s cap and orphaned a cron run at 18:37:38, leaving 10-31 at **160 / 4**; both Asana paths on CT460 are still non-functional) > model routing and automation. Last updated: 2026-11-02 (nightly audit — the 11-01 fleet settled **167 / 1 / 2**, its first non-clean day in four: the Sunday **18:00 Weekly Review never ran and never delivered** (`79f9bb41ecf5`, *"Gateway connection failed; no local fallback"*, incident still open) and **three cron workers were `SIGTERM`ed (`-15`) mid-adoption** without writing a durable terminal state (`85edf6772bca` 07:30:46; `96eb95ce12f2` + `0e753e9720fd` 09:02:18) — for the first time with **no gateway restart anywhere in the window**, so the orphan class is now known to fire without a reload; a **fourth variant** is documented where **`ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning no execution row and no incident row**; delivery went **7 of 8**, the miss being `066bb5191280`'s `thread_id 34795` home-DM thread fallback (content delivered, run marked `failed`), while **Telegram flapped all day** (dual-stack `api.telegram.org` timeouts, disconnect 20:51:53, restarts 22:33:30 / 23:12:23) and the **relay adapter has now been absent ~29 h** (`Reconnecting relay` attempt 336); the longest-standing incident on the board — the 2026-10-01 Facebook-repin row — finally **closed at 10:44:12**)
> >
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
@@ -65,6 +65,9 @@ tags: [index, infrastructure, systems]
- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since *(superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → [[incidents/2026-10-29-conduit-voice-silence]])*. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]] - [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since *(superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → [[incidents/2026-10-29-conduit-voice-silence]])*. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]]
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]] - [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]]
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery. - [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery.
- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] — **The Sunday 18:00 Weekly Review did not run on 2026-11-01 and nothing was delivered:** its execution closed `unknown` at 18:05:13 with *"Cron execution unverified … Gateway connection failed; no local fallback"*, its incident row is **still `alerted`**, the job sits at `error` / `failure_streak: 1`, and `deliveries.db` shows **no row after 11:01:12** for the rest of the day. Its batch-mate `b63ed5817212` took the identical error and self-healed in 68 s. Same page records the **inverse silent failure**: `ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning **no execution row and no incident row at all**.
- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] — **Three restart-safe cron workers killed by `SIGTERM` (`-15`) after adopting their claim**, so the scheduler recorded *"whether side effects ran is unknown"*: `85edf6772bca` @07:30:46, `96eb95ce12f2` + `0e753e9720fd` @09:02:18. **A fourth variant of the orphan class** — and the first with **no gateway restart, reload or drain anywhere in the window** (PID `175269` up since 10-31 19:24:40; relay counter monotonic through both signals), so it cannot be attributed to reload behaviour. Likely trigger: a **30 s worker-ack window** smaller than this host's cold start (**83 plugins / 69 enabled** discovered in 70 s). All three rows now `resolved`; one execution left `unknown`.
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]] - [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]]
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? **Second surface (2026-10-20 18:37):** an interactive `desktop` session received the same canned line after `Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … response_len=0`, so the substitute is emitted by the harness's failure path **generally**, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — **was never answered**. - [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? **Second surface (2026-10-20 18:37):** an interactive `desktop` session received the same canned line after `Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … response_len=0`, so the substitute is emitted by the harness's failure path **generally**, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — **was never answered**.
- [[incidents/2026-10-16-omniroute-503-resource-pressure]] — **`omniroute-1` load-sheds model calls with `HTTP 503 … resource pressure` — two bursts: 2026-10-16 → 10-17, and again 2026-10-19.** Burst 1: **9 model calls across 7 jobs** over ~19 h, self-cleared (reported then as `10-17 = 162 / 5`; the DB reads **163 / 5** — corrected 2026-10-20). Burst 2, 44 h later with an identical signature and blast radius: **9 model calls across 8 jobs** on 2026-10-19 (06:01 + 07:01 → 12:01, whole day **159 / 13**). `/api/health` read `200 ok` throughout both. **Status: `active` → back to `monitoring` on 2026-10-21** — 2026-10-20 ran **168 / 0** with zero rows carrying the resource-pressure string, so the second burst is over; it stays a standing risk until the single-node topology changes. 4 rows closed on 10-20 (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`); **unclosed rows 13 → 9** (8 `alerted` / 1 `detected`); the ledger still tags every 503 row `failure_type: unknown`, so triage on error text. **Single-node topology still open — and every job that failed reads `model: smart-route`, `provider: custom:omni-101`.** - [[incidents/2026-10-16-omniroute-503-resource-pressure]] — **`omniroute-1` load-sheds model calls with `HTTP 503 … resource pressure` — two bursts: 2026-10-16 → 10-17, and again 2026-10-19.** Burst 1: **9 model calls across 7 jobs** over ~19 h, self-cleared (reported then as `10-17 = 162 / 5`; the DB reads **163 / 5** — corrected 2026-10-20). Burst 2, 44 h later with an identical signature and blast radius: **9 model calls across 8 jobs** on 2026-10-19 (06:01 + 07:01 → 12:01, whole day **159 / 13**). `/api/health` read `200 ok` throughout both. **Status: `active` → back to `monitoring` on 2026-10-21** — 2026-10-20 ran **168 / 0** with zero rows carrying the resource-pressure string, so the second burst is over; it stays a standing risk until the single-node topology changes. 4 rows closed on 10-20 (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`); **unclosed rows 13 → 9** (8 `alerted` / 1 `detected`); the ledger still tags every 503 row `failure_type: unknown`, so triage on error text. **Single-node topology still open — and every job that failed reads `model: smart-route`, `provider: custom:omni-101`.**
+44
View File
@@ -1810,3 +1810,47 @@ recorded wrong.**
CT201:8102 returns **307 → 200** on `/board`, so the dashboard page's "live, 200" claim still holds. CT201's CT201:8102 returns **307 → 200** on `/board`, so the dashboard page's "live, 200" claim still holds. CT201's
`:8443/` root returns **502** (upstream 8082 has no listener) and `:8443/mgr` returns **200** — both still as `:8443/` root returns **502** (upstream 8082 has no listener) and `:8443/mgr` returns **200** — both still as
the homepage-dashboard page records. No secrets, tokens, passwords or keys written. the homepage-dashboard page records. No secrets, tokens, passwords or keys written.
## [2026-11-02] audit | Nightly review — the Weekly Review never ran, three workers killed by `-15`, and a red status with no row behind it
**Window:** 2026-11-01 00:00 → 2026-11-02 00:4x AWST. Live probes 00:05–00:55 on `state.db` (sessions),
`~/.hermes/cron/{jobs.json,executions.db,deliveries.db}`, `gateway.log` / `agent.log.1`, and both git trees.
- **NEW — `incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md`** (`status: active`):
the Sunday 18:00 `79f9bb41ecf5` closed `unknown` at 18:05:13 with *"Gateway connection failed; no local
fallback"*, **delivered nothing**, and its incident row `79f9bb_9c443b602eec` is **still `alerted`**.
`deliveries.db` has **no row after 11:01:12**. Batch-mate `b63ed5817212` hit the identical error and
self-healed in 68 s. The same page records the **inverse** silent failure: `ecc1fc13655a` reads
`error`, `last_run 11-01T12:01:00`, `failure_streak: 1` — with **no execution row and no incident row**.
- **NEW — `incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md`** (`status: resolved`): three
workers `SIGTERM`ed (`-15`) after adopting their claim (`85edf6772bca` 07:30:46; `96eb95ce12f2` +
`0e753e9720fd` 09:02:18) → *"whether side effects ran is unknown"*. **Fourth variant of the orphan
class, and the first with no gateway restart/reload/drain anywhere in the window** (PID `175269` up
since 10-31 19:24:40; the relay counter ran monotonic through both signals). Probable trigger: a
**30 s worker-ack window vs a ~70 s cold start** (`Plugin discovery complete: 83 found, 69 enabled`).
All three incident rows `resolved`; one execution left `unknown`.
- **UPDATED — `systems/scheduled-tasks.md`**: `## Count` advanced to the 2026-11-02 window (**167
`completed` / 1 `running` / 2 `unknown`** vs 161/4 on 10-31, 138/0 on 10-30, 177/0 on 10-29 — the
first non-clean day in four); fleet **29 jobs / 26 enabled / 3 disabled** unchanged; **unclosed
incident rows 1 → 1 but a different row** (all three rows the 10-31 audit left open closed on 11-01,
including **`0e753e_e7f8194a7d06`**, open since **2026-10-01** — 31 days); new `## Re-verified
2026-11-02` section covering the three failure modes, the 09:00 cold-start pair and the ledger.
**Non-clean `last_status` 4 → 2** (`ea79a2107e36`, `79f9bb41ecf5`).
- **UPDATED — `systems/messaging-integrations.md`**: Telegram degraded all day (dual-stack
`api.telegram.org` `ConnectTimeout` → IPv4-literal fallback; disconnect 20:51:53, polling restarts
22:33:30 / 23:12:23), and `getent hosts` currently returns **only an AAAA record** for
`api.telegram.org` — consistent with, but not proof of, a systematic timeout on the path DNS hands
the client. The **`relay` platform has had no adapter for ~29 h** (`Reconnecting relay` attempt
**147 @10-31 18:39:24 → 336 @11-02 00:10:47**, counter never reset — itself the proof of no gateway
restart). `066bb5191280`'s `thread_id 34795` home-DM miss is the **second confirmed variant** of the
thread-mismatch bug: the thread-less fallback fires, the content lands, and the **run is recorded
`failed` anyway**. Day's delivery health: **8 rows / 7 `delivered` / 1 `failed`**.
- **UPDATED — `index.md`**: `updated`/`verified_on` → 2026-11-02, the audit summary line rewritten for
this window, and the two new incident links added to the incident list.
- **Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned;
no decision reversed in the window. A full vault mirror pass was deliberately skipped (drift counts
below instead of a bulk copy) — an audit's mandate is smaller than a full sync.
- **DRIFT (context, not findings):** the authoring tree was clean and in sync with Gitea
(`origin/master..HEAD` empty) before this audit; touched pages were mirrored into the vault's
existing conventions and hash-verified. **No secrets, passwords, tokens or keys written or quoted** —
every job id, PID, thread id and timestamp above is a non-sensitive operational identifier.
+33 -2
View File
@@ -3,8 +3,8 @@ title: Messaging Integrations
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-01 updated: 2026-11-02
verified_on: 2026-11-01 verified_on: 2026-11-02
confidence: high confidence: high
tags: [messaging, telegram, discord, email, integrations] tags: [messaging, telegram, discord, email, integrations]
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug] sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
@@ -152,3 +152,34 @@ the adapter stayed connected since.
[[infrastructure/tailscale]] [[infrastructure/tailscale]]
- **Open item now closed:** the platform is no longer an outage. The home channel startup notification - **Open item now closed:** the platform is no longer an outage. The home channel startup notification
was delivered successfully on reconnect (`Sent home-channel startup notification to mattermost:…`). was delivered successfully on reconnect (`Sent home-channel startup notification to mattermost:…`).
## 2026-11-02 00:4x AWST — Telegram flapped all day; the relay adapter has now been absent ~29 h
Live probes on `gateway.log` for the 11-01 window plus `deliveries.db`:
- **Telegram transport degraded repeatedly, and it never fully failed.** The recurring shape is a
**dual-stack `api.telegram.org` `ConnectTimeout`** on the first hop, followed by a **fallback to an
IPv4 literal** that succeeds (`Telegram] Disconnected from Telegram` at **20:51:53**, with polling
restarts at **22:33:30** and **23:12:23**). The host's resolver currently returns **only an AAAA
record** for `api.telegram.org` (`getent hosts` → `2001:67c:4e8:f004::9`), so the dual-stack path the
client tries first is the one DNS actually hands it. **Consistent with, but not proof of, a
systematic timeout on that path** — recorded as an observation, not a diagnosed cause.
- **`066bb5191280` is the second variant of the thread-mismatch bug, confirmed with hard evidence.**
Its 11-01 delivery row is `failed` with
`configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id`.
The target is the **home DM** (`1793951355`) carrying a *thread* id, so the sanctioned thread-less
fallback fires, the content still reaches the DM, and the **run is recorded `failed` regardless** —
i.e. a delivery that landed is reported as a failure. Same 2026-10-01 semantics documented above,
now the second confirmed instance.
- **The `relay` platform has no adapter in the running gateway and has not had one for ~29 h:**
`Reconnecting relay` … `no adapter yet` repeats every ~300 s, **attempt 147 at 10-31 18:39:24 →
attempt 336 at 11-02 00:10:47**, counting without a reset. **The counter is monotonic across both the
18:00 handoff failure and the three `-15` kills, which is the proof that the gateway process never
restarted in that window** — see [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]].
The feed of retries is cosmetic for Telegram/email delivery (both work), but it makes
`Reconnecting <platform>` log lines useless as a restart detector once an adapter is absent.
- **Net delivery health for 11-01: 8 rows, 7 `delivered`, 1 `failed`** — and **nothing after 11:01:12**
(the missing Weekly Review). The Telegram adapter itself carried every Telegram-bound delivery
successfully, so the 11-01 degradation is a **latency/retry** problem, not a delivery outage.
→ [[systems/scheduled-tasks]], [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]]
+89 -4
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-01 updated: 2026-11-02
verified_on: 2026-11-01 verified_on: 2026-11-02
review_after: 2026-11-06 review_after: 2026-11-09
confidence: high confidence: high
tags: [cron, scheduled, automation] tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db] sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -14,7 +14,33 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks # Scheduled Tasks
## Count ## Count
- **Current as of 2026-11-01 00:2x AWST (live `jobs.json` + `executions.db` + `cron_incidents` + gateway - **Current as of 2026-11-02 00:4x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
`deliveries.db` + gateway log) — the fleet ran clean all day, then failed in three different ways
and missed a delivery.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged). **Window 2026-11-01
settled 167 `completed` / 1 `running` / 2 `unknown`** (+ this audit `running`, started 23:59:28)
against **161 / 4** on 10-31, **138 / 0** on 10-30 and **177 / 0** on 10-29 — the first **non-clean**
day in four. **The two `unknown` rows are the whole story of the day:** the Sunday 18:00 **Weekly
Review never ran and never delivered** (*"Gateway connection failed; no local fallback"*, incident
`79f9bb_9c443b602eec` **still `alerted`** at audit time, job reads `error` / `failure_streak: 1`),
and one `96eb95ce12f2` watchdog slot. **Three other jobs were `SIGTERM`ed (`-15`) mid-adoption without
writing a durable terminal state** (`85edf6772bca` @07:30:46, `96eb95ce12f2` and `0e753e9720fd`
@09:02:18) — with **no gateway restart or reload anywhere in the window** (PID `175269` up since
10-31 19:24:40; the relay counter ran unbroken through both signals) →
[[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] and
[[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]. **Delivery health:** 8
delivery rows on 11-01, **7 `delivered` / 1 `failed`** — the failure is `066bb5191280`'s
`thread_id 34795` DM-thread miss (the thread-less fallback still reaches the DM; the run is recorded
`failed` anyway). **Non-clean `last_status` 4 → 2`** (`ea79a2107e36`, `79f9bb41ecf5`).
⚠ **`ecc1fc13655a` reads `error` with `failure_streak: 1` and `last_run 2026-11-01T12:01:00` while
owning NO `executions` row for that slot and NO `cron_incidents` row** — a status field with no
artefact behind it. **Unclosed incident rows 1 → 1, but a different row:** all three rows the 10-31
audit left open closed on 11-01 (02:39:33 ×2 for `ea79a2`, **10:44:12** for
`0e753e_e7f8194a7d06` — the Facebook-repin row open since **2026-10-01**, 31 days). ⚠ **Telegram's
transport flapped all day** (dual-stack `api.telegram.org` ConnectTimeouts, IPv4-literal fallbacks,
a disconnect at 20:51:53, polling restarts at 22:33:30 / 23:12:23) and the **relay adapter has been
absent ~29 h** (`Reconnecting relay` attempt **336** at 11-02 00:10:47) →
[[systems/messaging-integrations]].
- **Prior window (2026-11-01 00:2x AWST, live `jobs.json` + `executions.db` + `cron_incidents` + gateway
log) — the last five hours of 10-31 ran clean, but an 18:06 gateway reload force-drained at its full log) — the last five hours of 10-31 ran clean, but an 18:06 gateway reload force-drained at its full
1 800 s cap and killed a live cron run.** Fleet **29 jobs — 26 enabled / 3 disabled** 1 800 s cap and killed a live cron run.** Fleet **29 jobs — 26 enabled / 3 disabled**
(`jobs.json.updated_at 2026-11-01T00:05:42+08:00`). **Window 2026-10-31 settled 160 `completed` / (`jobs.json.updated_at 2026-11-01T00:05:42+08:00`). **Window 2026-10-31 settled 160 `completed` /
@@ -669,3 +695,62 @@ obsolete, no decision reversed in the window.
Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255 Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255
stopped; the Conduit patch uncommitted and **five releases behind HEAD**; the **569.9 MB** Conduit stopped; the Conduit patch uncommitted and **five releases behind HEAD**; the **569.9 MB** Conduit
health store with no retention; and a **call that has never been placed** since the memory fix. health store with no retention; and a **call that has never been placed** since the memory fix.
## Re-verified 2026-11-02 (audit window 2026-11-01 00:00 → 2026-11-02 00:4x AWST; live probes 00:0x–00:5x)
**Settled: 167 `completed` / 1 `running` / 2 `unknown`** — the first non-clean day after three
consecutive clean-or-nearly-clean days (10-31 was 161 / 4). Fleet unchanged at **29 jobs, 26 enabled,
3 disabled**. Nothing was added, removed, disabled or repinned in the window.
### The day's three failure modes, none of which is a repeat of 10-31
| # | Fault | Jobs | Durable record | Page |
|---|---|---|---|---|
| 1 | **`SIGTERM` (`-15`) after adoption, no terminal state** — *"whether side effects ran is unknown"* | `85edf6772bca` 07:30:46, `96eb95ce12f2` + `0e753e9720fd` 09:02:18 | 3 `cron_incidents` rows, all `resolved`; one execution left `unknown` | [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] |
| 2 | **Gateway handoff failure** — *"Cron execution unverified … Gateway connection failed; no local fallback"* | `b63ed5817212` (self-healed 68 s) and `79f9bb41ecf5` | 2 rows; **`79f9bb_9c443b602eec` still `alerted`** | [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] |
| 3 | **`last_status: error` with no execution row and no incident row** | `ecc1fc13655a`, `last_run 2026-11-01T12:01:00` | **none** | same page, "Second silent gap" |
Fault 1 is a **fourth variant** of the orphan class (after 10-20 crash, 10-26 drained-`SIGTERM`, 10-31
cap force-drain) and the first with **no gateway restart anywhere near it** — so it cannot be explained
by reload/drain behaviour. Fault 3 is the inverse of the usual silent failure: instead of a green row
over a dead job, there is a **red status with no row at all**.
### The 09:00 pair — cold-start cost meets a 30 s ack window
At **09:00:49.708** the spawner logged `Cron external worker for job '96eb95ce12f2' did not acknowledge
within 30s; leaving the durable execution claim untouched`; the worker then spent **09:00:49 → 09:01:02**
doing a full cold boot (`Plugin discovery complete: 83 found, 69 enabled`) before starting its job at
09:01:13 and being signalled at ~09:02:13. **The 30 s window is smaller than this host's cold-start
time.** Not yet formalised as an Open item because the signal source is unidentified — see the incident
page's open questions.
### Delivery health: 8 rows, 7 delivered
`deliveries.db` for 11-01: 02:39:33 ✔, 03:00:37 ✔, **07:18:50 ✘** (`066bb5191280`,
`configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id`),
07:34:49 ✔, 08:36:51 ✔, 10:44:12 ✔, 11:01:12 ✔ — **and nothing after 11:01:12**, which is how the
missing Weekly Review was detected. The `066bb5191280` case is the **second variant** documented in
[[systems/messaging-integrations]] (thread-less fallback send, run still marked `failed`).
### Incident ledger movement
- **Closed on 11-01:** `ea79a2_a4661554a3c2` + `ea79a2_9197df32320b` (02:39:33 — the Mnemosyne Health
Monitor pair from 10-26 / 10-31) and **`0e753e_e7f8194a7d06`** (10:44:12) — the Facebook birthday
repin row that had been open since **2026-10-01 09:00:55**, the oldest row on the board. That job's
`delivery_outcome: delivered` on the same run.
- **Opened on 11-01:** 5 rows; **4 resolved, 1 open**.
- **Now unclosed: exactly one** — `79f9bb_9c443b602eec` (`state: alerted`, `acked_at: None`).
### Deliberately not done
- No full vault mirror pass (drift counts recorded in `log.md`; a complete pass is bigger than an
audit's mandate).
- No edits to `### Open` beyond the `066bb5191280` note above: the 10-13 GitHub-token and Camofox
containment items were not re-probed this window, so their lines are left as last verified.
- Nothing removed or archived — no job retired, no system obsoleted, no decision reversed.
**Still open (carried):** the single-node no-failover topology behind every model-calling job; the
GitHub Trending Radar token; Camofox containment under multi-tab load; the Conduit health store with no
retention; **plus new:** the unreconciled admission files
`~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json` (both 11-01) and the unidentified `-15`
signal source.