Nightly audit 2026-11-03 (window 2026-11-02): new incident — Job Radar Stage 1 iteration-cap recurrence (9h 01m, suppressed, no incident raised); Weekly Review incident resolved (reconciled 01:27 AWST); scheduled-tasks + browser-backend re-verified; index/log updated

This commit is contained in:
Tony0410
2026-11-03 00:22:13 +08:00
parent fda56e53fe
commit ca014efd63
8 changed files with 516 additions and 20 deletions
+5 -4
View File
@@ -3,15 +3,15 @@ title: Anthony's Systems Wiki
type: collection
status: active
created: 2026-07-22
updated: 2026-11-02
verified_on: 2026-11-02
updated: 2026-11-03
verified_on: 2026-11-03
confidence: high
tags: [index, infrastructure, systems]
---
# Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-11-02 (nightly audit — the 11-01 fleet settled **167 / 1 / 2**, its first non-clean day in four: the Sunday **18:00 Weekly Review never ran and never delivered** (`79f9bb41ecf5`, *"Gateway connection failed; no local fallback"*, incident still open) and **three cron workers were `SIGTERM`ed (`-15`) mid-adoption** without writing a durable terminal state (`85edf6772bca` 07:30:46; `96eb95ce12f2` + `0e753e9720fd` 09:02:18) — for the first time with **no gateway restart anywhere in the window**, so the orphan class is now known to fire without a reload; a **fourth variant** is documented where **`ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning no execution row and no incident row**; delivery went **7 of 8**, the miss being `066bb5191280`'s `thread_id 34795` home-DM thread fallback (content delivered, run marked `failed`), while **Telegram flapped all day** (dual-stack `api.telegram.org` timeouts, disconnect 20:51:53, restarts 22:33:30 / 23:12:23) and the **relay adapter has now been absent ~29 h** (`Reconnecting relay` attempt 336); the longest-standing incident on the board — the 2026-10-01 Facebook-repin row — finally **closed at 10:44:12**)
> model routing and automation. Last updated: 2026-11-03 (nightly audit — the 11-02 fleet settled **177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`**, the **first fully clean ledger day in five**, and **both 11-01 `unknown` rows resolved in-window**: the Sunday 18:00 **Weekly Review reconciled at 01:27:22** after **7 h 26 m 43 s** and **delivered**, so *"the lost review" was never lost* — incident `79f9bb_9c443b602eec` auto-closed **01:27:21.606691** and its page is now **`resolved`** — while `ecc1fc13655a`'s 11-01 ghost `error` cleared on its 12:00:33 slot; **unclosed incident rows 1 → 0** (the first zero in the period), non-clean `last_status` **2 → 1**, deliveries **9 rows / 9 delivered**; ⚠ **but two ~9-hour runs failed invisibly.** The **Job Radar Stage 1** (`dca8482e4f76`, 07:00:33 → 16:01:52, **9 h 01 m 19 s**) hit the **agent iteration cap before writing its handoff**, so **both Stage 2 slots exited `[SILENT]` and nothing reached Notion** — and it **raised no `cron_incidents` row at all**, where the same class raised incidents on 10-27 and 10-28, because its terminal state flipped to `completed` with `error: NULL`; the **03:00 stocktake** (`1a8e90ea2271`) likewise ran **9 h 43 m 29 s**, **delivered no report** and left its output uncommitted in the working tree (committed by this audit).)
>
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
@@ -65,7 +65,8 @@ tags: [index, infrastructure, systems]
- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since *(superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → [[incidents/2026-10-29-conduit-voice-silence]])*. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]]
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]]
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery.
- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] — **The Sunday 18:00 Weekly Review did not run on 2026-11-01 and nothing was delivered:** its execution closed `unknown` at 18:05:13 with *"Cron execution unverified … Gateway connection failed; no local fallback"*, its incident row is **still `alerted`**, the job sits at `error` / `failure_streak: 1`, and `deliveries.db` shows **no row after 11:01:12** for the rest of the day. Its batch-mate `b63ed5817212` took the identical error and self-healed in 68 s. Same page records the **inverse silent failure**: `ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning **no execution row and no incident row at all**.
- [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] — **Third iteration-cap in seven days, and the first that raised no incident at all.** Job Radar Stage 1 (`dca8482e4f76`) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** on 2026-11-02, hit the **agent iteration cap before the handoff was written**, and closed **`status: completed` / `error: NULL` / `delivery_outcome: suppressed`**. Both Stage 2 slots declined correctly (`92665337…` 11:30 and the `05a3f00b…` 14:30 safety net both exited `[SILENT]`), so **nothing reached Notion, no cover letters were generated, and the day's 164 deduped candidates were never persisted** — a state a `status`-only health check reports as green. Contrast 10-27 (`dca848_e93a538ade6b`) and 10-28 (`dca848_0a2f03b5952b`), where the identical cap **did** open rows whose `error` text is the run's own INCOMPLETE report: on 11-02 the ledger **lost the red row it used to produce**. Discovery itself worked — SEEK **51**, LinkedIn **60 IDs**, Indeed live, WA Government **35** (197 raw / **164 new** after three-level dedup) — and a **URL-canonicalisation bug was found and fixed mid-run** (`au.seek.com` vs `au.seek.com.au` had been dropping SEEK *and* Indeed entries from the consolidated list). → [[systems/scheduled-tasks]]
- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] — **The Sunday 18:00 Weekly Review did not run on 2026-11-01 and nothing was delivered:** its execution closed `unknown` at 18:05:13 with *"Cron execution unverified … Gateway connection failed; no local fallback"*, its incident row is **still `alerted`**, the job sits at `error` / `failure_streak: 1`, and `deliveries.db` shows **no row after 11:01:12** for the rest of the day. Its batch-mate `b63ed5817212` took the identical error and self-healed in 68 s. Same page records the **inverse silent failure**: `ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning **no execution row and no incident row at all**. ✅ **RESOLVED 2026-11-02T01:27:21 — the review was never lost.** The *same* execution row (`be6ec11a8e68428084af9ab18fb148db`) closed **`completed` / `delivered` at 01:27:22** after **7 h 26 m 43 s**, the delivery lands at **01:27:21.802246** and incident `79f9bb_9c443b602eec` auto-closed at **01:27:21.606691** — so the 18:05:13 `unknown` was a **reconciler timing artefact on a still-running job**, not a verdict (page now `resolved`). Batch-mate `b63ed5` closed in 68 s and `79f9bb` in 7 h 26 m, both `completed`; **an `unknown` row on a long job is a timing artefact, not a failure**. `ecc1fc13655a`'s ghost `error` cleared too (12:00:33 → `completed` / `delivered` 12:29:26), taking **non-clean `last_status` 2 → 1**.
- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] — **Three restart-safe cron workers killed by `SIGTERM` (`-15`) after adopting their claim**, so the scheduler recorded *"whether side effects ran is unknown"*: `85edf6772bca` @07:30:46, `96eb95ce12f2` + `0e753e9720fd` @09:02:18. **A fourth variant of the orphan class** — and the first with **no gateway restart, reload or drain anywhere in the window** (PID `175269` up since 10-31 19:24:40; relay counter monotonic through both signals), so it cannot be attributed to reload behaviour. Likely trigger: a **30 s worker-ack window** smaller than this host's cold start (**83 plugins / 69 enabled** discovered in 70 s). All three rows now `resolved`; one execution left `unknown`.
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]]