From 987eba72e6e607d05a51366e3e19c68979d47e92 Mon Sep 17 00:00:00 2001 From: Tony0410 <47945770+Tony0410@users.noreply.github.com> Date: Fri, 30 Oct 2026 05:41:11 +0800 Subject: [PATCH] Stocktake 2026-10-30: 0 stale by age, but 5 pages drifted from live state Ran wiki-maintain.sh (skills-index regenerated: 302 skills / 69 categories, unchanged; freshness report 0 stale / 109 fresh / 3 no-date of 112 pages; Quartz rebuilt and deployed). Freshness is not accuracy. A live-vs-page diff found real drift on five pages. All corrections verified against live sources first. - infrastructure/tailscale: node table rebuilt from tailscale status --json. 50 non-funnel nodes (49 IPv4 peers + hermes), not 54. 2 IP changes, 4 peers expired out of the tailnet. Zero IP mismatches, zero missing/extra rows on re-verification. m93p records online (peer record chronically stale, third occurrence -- ping + SSH confirm). CT460 :443 route set (12 paths) and CT201 :443/:8443 tables re-probed and match. - systems/scheduled-tasks: fleet is 29 jobs (25 enabled / 4 disabled), not 28/25/3. Mnemosyne Health Monitor ea79a2107e36 was auto-paused 2026-10-30 01:04:22 after its 00:01:08 slot closed "unknown" on a gateway-connection failure. Non-clean last_status is 4, not 3. "Failing (delivery only)" is 2. Seven job rows carried stale error claims for jobs since cleared to ok -- each marked superseded with a dated correction, not rewritten (SCHEMA rule 9). - NEW incidents/2026-10-30-mnemosyne-health-monitor-paused: the job's own failure notice was never delivered (deliveries.db has zero rows for 2026-10-30), so the incident row is the only trace. Job not re-enabled and gateway not restarted by this run; root cause recorded as unverified. - infrastructure/hosts: CT201 LAN is 192.168.178.77 (was .166); CT450 Hermes is v0.22.5 / bf0203a794a0, same build as CT460 (page said v0.22.3 / 5b147ab036d6); cron count 29/25/4; orphaned "omni" peer has expired out of the tailnet entirely. Skills count of 302 left alone -- it is correct. - systems/homepage-dashboard: live app count is 123 (was 114); grid is a responsive repeat(auto-fill, minmax(230px, 1fr)), not a fixed 5-column grid; tab bar is built from the service groups (renderTabs iterates state.groups), so the settings.yaml layout block and its tab:/columns: keys are inert -- there is no "Overview" tab; saved theme is Boarding Pass / light, not dark amber; manager.py runs inside the homepage-manager container; and the "no Traefik container" evidence was wrong -- the container exists but is stopped (Exited (255) 2 months ago). The conclusion that Traefik does not occupy 443 stands. Same evidence error corrected in tailscale.md. Also corrected two of my own first-pass errors before committing: the homepage tab list (groups vs tabs) and the CT450 version (a stale root-level /usr/local/bin/hermes reports v0.19.0; the live agent is the hermes-user install at v0.22.5). No secrets written -- provider names, key names and HTTP status codes only. --- current-state.md | 22 ++- ...6-10-30-mnemosyne-health-monitor-paused.md | 124 +++++++++++++++++ index.md | 5 +- infrastructure/hosts.md | 12 +- infrastructure/tailscale.md | 49 ++++--- log.md | 62 +++++++++ raw/stocktake/freshness-20261030.txt | 125 ++++++++++++++++++ systems/homepage-dashboard.md | 87 +++++++++--- systems/scheduled-tasks.md | 31 ++--- systems/skills-index.md | 4 +- 10 files changed, 448 insertions(+), 73 deletions(-) create mode 100644 incidents/2026-10-30-mnemosyne-health-monitor-paused.md create mode 100644 raw/stocktake/freshness-20261030.txt diff --git a/current-state.md b/current-state.md index 97332d0..2bd52ff 100644 --- a/current-state.md +++ b/current-state.md @@ -39,11 +39,23 @@ Updated: 2026-10-30 then no-oped as designed (10 m 46 s / 1 m 51 s). → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], [[runbooks/job-radar]] -- **Cron: 2026-10-29 settled 176 `completed` / 0 `failed`** — the first zero-failure day since 2026-10-24 — with - **10 deliveries, all 10 `delivered`, 0 failures**. Non-clean `last_status` **5 → 3**; every `failure_streak` - reads **0**; unclosed incident rows **4 → 1**. Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**, pins - unchanged (19 of the 26 enabled jobs on `custom:omni-101`: 12 `smart-route`, 6 `free-stack`, 1 explicit; 7 - unpinned on the global default). +- **Cron: 2026-10-29 settled 177 `completed` / 0 `failed`** — the first zero-failure day since 2026-10-24 — with + **10 deliveries, all 10 `delivered`, 0 failures**. Non-clean `last_status` **5 → 3**; unclosed incident rows + **4 → 1** (the two radar rows closed `resolved` at 10:16:12). Fleet unchanged at **29 jobs (26 enabled / 3 + disabled)** at that audit. +- **Cron fleet changed shape overnight on 2026-10-30 — the Mnemosyne Health Monitor is paused and its failure + notice was never delivered.** `ea79a2107e36`'s 00:01:08 slot closed **`unknown`** at 01:04:22 on a + gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: + Gateway connection failed; no local fallback"*), opening incident `ea79a2_d93c3acdc4f2` (`detected`), and the + scheduler **auto-paused the job**. Fleet is now **29 jobs (25 enabled / 4 disabled)**; non-clean + `last_status` is **4** (2 `error` + 2 `delivery_failed`); `ea79a2107e36` carries `failure_streak: 1`. + ⚠ **The failure notice itself failed to send** — `hermes cron list` reports `send_path_degraded; live adapter + delivery to telegram:-1004321904721:102 failed`, and `deliveries.db` holds **zero rows for 2026-10-30**, so + the only trace is the incident row. **Unclosed incident rows 1 → 2.** Every other previously-red job + (`dca8482e4f76`, `1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, + `c0a9fa314542`) has **cleared to `ok` with streak 0** on its own slot. Pins: **18 of the 25 enabled jobs pin + `custom:omni-101`** (11 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`); 7 enabled rows are `no_agent` + jobs with no pin. → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] ## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 00:2x AWST) diff --git a/incidents/2026-10-30-mnemosyne-health-monitor-paused.md b/incidents/2026-10-30-mnemosyne-health-monitor-paused.md new file mode 100644 index 0000000..de3f221 --- /dev/null +++ b/incidents/2026-10-30-mnemosyne-health-monitor-paused.md @@ -0,0 +1,124 @@ +--- +title: Mnemosyne Health Monitor auto-paused after gateway-connection failure +type: incident +status: active +created: 2026-10-30 +updated: 2026-10-30 +verified_on: 2026-10-30 +confidence: high +tags: [cron, mnemosyne, gateway, incident, delivery] +sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents] +--- + +# Mnemosyne Health Monitor auto-paused after gateway-connection failure (2026-10-30) + +**Status: ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram. + +## Summary + +The Mnemosyne Health Monitor cron job `ea79a2107e36` (`0 */12 * * *`, delivery topic `102`) failed its +2026-10-30 00:01 slot with a **gateway-connection** fault. The run closed with status **`unknown`**, +the scheduler **auto-paused the job**, and the job's own failure notice **failed to send** — so the +only trace of the fault is the incident row in `cron_incidents`. + +## What happened + +| Field | Value | +|---|---| +| Job | `ea79a2107e36` — Mnemosyne Health Monitor | +| Schedule | `0 */12 * * *` (every 12 h) | +| Started | 2026-10-30 00:01:08.343024+08:00 | +| Finished | 2026-10-30 01:04:22.510091+08:00 (1 h 03 m 14 s) | +| Execution status | **`unknown`** | +| Job `last_status` | **`error`**, `failure_streak: 1` | +| Job `enabled` | **`false`** — auto-paused 2026-10-30 01:04:22 | +| Incident | `ea79a2_d93c3acdc4f2`, `failure_type: unknown`, state **`detected`** | +| Delivery | **failed — never sent** | + +Error text on the execution row: + +> Cron execution unverified; reconcile +> `/home/hermes/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json`: +> Gateway connection failed; no local fallback + +## The two traps in this one fault + +### 1. A failure notice that was itself never delivered + +`hermes cron list` reports for this job: + +> The result was not delivered: live adapter send failed: send_path_degraded; +> live adapter delivery to `telegram:-1004321904721:102` failed + +Confirmed at the artefact level: `deliveries.db` contains **zero rows created on 2026-10-30** +(2026-10-29 has 10 rows, all `delivered`). So the failure produced **no Telegram output at all** — +not even a notice about the failure. + +This is the same class as [[incidents/2026-08-08-gitea-config-backup-push-dead]]: a job whose green +row hides a broken destination. Here the destination was broken *and* the row went red, but the +**alert path** was the part that was dead. If Anthony had only watched `hermes cron list` for a +non-clean row, he would have seen the job had paused; if he had only watched Telegram, he would have +seen nothing at all. + +### 2. `unknown` is not `failed` + +The execution closed `unknown`, not `failed`. Per the wiki's own convention +([[systems/scheduled-tasks]]), a `status: unknown` row with **no matching `deliveries.db` row** is a +probable non-delivery that the dashboard hides behind the previous slot's error. The scheduler then +auto-paused the job — so the fleet lost a **monitoring** job, not a reporting job. + +## Root cause + +**Unverified.** The error names a gateway-connection failure with "no local fallback" during the +reconciliation of an admission record. The admission file referenced in the error is +`~/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json`. + +This is **not** the same failure as the 2026-10-26 planned-restart orphan class +([[incidents/2026-10-26-planned-restart-orphaned-cron-run]]), though the error string is similar +("Cron execution unverified; reconcile …"). The 10-26 rows were `alerted` and auto-closed; this row +is still `detected` and the job stayed paused. **No gateway restart was performed on 2026-10-30** +around that window, so the trigger is not established. + +## Current state (verified 2026-10-30 03:3x AWST) + +- Live `jobs.json`: `ea79a2107e36` → `enabled: false`, `last_status: error`, `failure_streak: 1`, + `paused_reason` = the gateway-connection text above. +- Live `cron_incidents` (table lives inside `executions.db`): `ea79a2_d93c3acdc4f2` is the **only** + new row since 2026-10-29; `closed_at` is **NULL**, `acked_at` is **NULL**, `alerted_at` is **NULL**. +- The other unclosed row is `0e753e_e7f8194a7d06` (Facebook monthly, `failure_type: agent`, `detected` + since 2026-10-01, clears on its 2026-11-01 slot) — **2 unclosed rows in total**. +- Fleet is now **29 jobs — 25 enabled / 4 disabled**; this job is the 4th disabled one. +- The rest of the fleet was healthy in the same window: 2026-10-29 settled **177 `completed` / 0 + `failed`**, and 2026-10-30 (to 05:4x) shows 37 `completed`, 1 `running` (this stocktake), 1 + `unknown` (this job), 0 `failed`. + +## Actions taken + +1. Recorded the incident and its two distinct failure modes here. +2. Updated [[systems/scheduled-tasks]] — the job's row, the Disabled Jobs table, the fleet counts + (29/25/4), the non-clean `last_status` count (3 → 4), the "Failing (last run failed outright)" + bullet (0 → 1), and the "Failing (delivery only)" bullet (1 → 2). +3. Updated [[infrastructure/hosts]] for the cron count. + +## Actions NOT taken — and why + +- **The job was not re-enabled.** Re-enabling a job whose gateway connection failed an hour ago + risks an immediate re-pause, and it would close the incident's purpose. The recovery belongs to + whoever owns the gateway fix. +- **No gateway restart was performed.** This was a stocktake run, and restarting the gateway is a + disruptive action with its own orphan-risk class ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]]). + The fault is recorded, not papered over. + +## Next step + +Re-enable `ea79a2107e36` **after** the gateway connection is confirmed healthy, then confirm the +run completes and — critically — that a delivery row appears in `deliveries.db`. The point of this +monitor is to report on exception to topic `102`; if the topic path is still degraded, re-enabling it +recreates a silent monitor. + +## Related + +- [[systems/scheduled-tasks]] — fleet ledger +- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — the earlier "execution unverified" class +- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — green row, dead destination +- [[systems/mnemosyne]] — the subsystem being monitored diff --git a/index.md b/index.md index 03154d0..d10574e 100644 --- a/index.md +++ b/index.md @@ -46,10 +46,11 @@ tags: [index, infrastructure, systems] - [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.69.2` (HEAD `942ddb8`, 2026-10-24 15:35 — *"Pulse: list candidates, decide in code, journal them (#342)"*; 24 commits on since the `0.64.0` baseline, **0 behind `origin/main`**)**, upstream `kaishi00/hermes-conduit-notifier`. **Gateway on `:9119` restarted 2026-10-25 18:42:33 — so the upstream commits are now LIVE** (the 10-24 "not live" note is superseded). One registered device (iPhone app **2.2.0**). ✅ **The first two real calls in the product's history are now on record — 2026-10-29 08:38:04 and 08:49:36** — after three faults in series were repaired (truncated speech-server URL → `…/v1/realtime`; 12 h dashboard session TTL → 30 d; dead `groq` STT → `local`, dead `openai` TTS → `ollama_local`) → [[incidents/2026-10-29-conduit-voice-silence]]. **So the memory fix is no longer evidence-free.** Still carries the uncommitted in-tree `dashboard/plugin_api.py` patch. The 10-24 refusal fault has not recurred → [[incidents/2026-10-24-conduit-call-request-refused]]. - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26** (peer offline **36 d**, `tx 312 / rx 0`; host answers neither `:8065` nor ICMP). **268** connect timeouts on 10-25 vs 264 on 10-24; the reconnect **attempt counter reset to 54** at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost. - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. -- [[systems/scheduled-tasks]] — **As of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 5 → 3 (all three are stale re-runs of old faults and every `failure_streak` reads 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all `delivered`. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with `missing_cover_letter_count 0` (Notion 455 → 475). Both stage-1 incident rows closed `resolved` at 10:16:12; the lone survivor is the Facebook monthly's `0e753e_e7f8194a7d06`.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** -- [[systems/skills-index]] — Installed skills inventory (**302 skills** (69 categories) as of the **2026-10-27** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at the 2026-10-27 run: **0 stale / 104 fresh / 4 no-date of 108 pages** (`freshness-20261027.txt`). The 2026-10-27 03:00 slot regenerated `systems/skills-index.md` but left it uncommitted in the authoring tree; the 2026-10-28 audit committed it. Note: `SKILL.md` files on disk still exceed the index (the generator de-dups one nested duplicate; see [[systems/scheduled-tasks]]). ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. +- [[systems/scheduled-tasks]] — **As of 2026-10-30 03:3x: 29 jobs (25 enabled, 4 disabled); non-clean `last_status` 3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries all `delivered`. ⚠ `ea79a2107e36` (Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closed `unknown` on a gateway-connection failure — and its failure notice was NEVER DELIVERED (`deliveries.db` has zero rows for 2026-10-30), so the incident row is the only trace → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. Every other previously-red job has cleared to `ok`/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 5 → 3 (all three are stale re-runs of old faults and every `failure_streak` read 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all `delivered`. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with `missing_cover_letter_count 0` (Notion 455 → 475). Both stage-1 incident rows closed `resolved` at 10:16:12; the lone survivor is the Facebook monthly's `0e753e_e7f8194a7d06`.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** +- [[systems/skills-index]] — Installed skills inventory (**302 skills** (69 categories) as of the **2026-10-30** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at the 2026-10-30 run: **0 stale / 109 fresh / 3 no-date of 112 pages** (`freshness-20261030.txt`). Note: `SKILL.md` files on disk still exceed the index by one (303 vs 302) — the generator de-dups one nested duplicate, so **302 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents +- [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — **The Mnemosyne Health Monitor cron job was auto-paused overnight and its failure notice never reached Telegram — the only trace is the incident row.** `ea79a2107e36`'s 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, opening incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, `detected`) — and the scheduler **paused the job at that moment**. Fleet is now **29 jobs (25 enabled / 4 disabled)**; live `last_status: error`, `failure_streak: 1`. ⚠ **Two traps in one fault:** the failure notice itself failed to send (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed` — `deliveries.db` has **zero rows for 2026-10-30**), and the run closed `unknown` rather than `failed`. The job was **not re-enabled** and the gateway was **not restarted** by the stocktake — root cause is **unverified**. → [[systems/scheduled-tasks]], [[systems/mnemosyne]] - [[incidents/2026-10-29-conduit-voice-silence]] — **Every Conduit voice call connected and then died silently; three faults sat in series, and the first two real calls in the product's history were placed on 2026-10-29 after all three were fixed.** (A) `voice.speech_server.url` was the truncated `wss://…:8443/v1/` — the plugin dials it **verbatim** and the bare path answers **403**, reported as *"the speech server refused the key."* (B) `dashboard.basic_auth.session_ttl_seconds` was the **12 h default**, so an expired phone session gets **WS 4401** with no Hermes turn and no error on the device (the phone re-authenticated at 08:00:59, minutes *after* a silent call). (C) Both host providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq` → **403/Cloudflare 1010** (blocked, not mis-keyed). Fixed 08:02–08:4x: URL → `…/v1/realtime`, TTL → **2 592 000 s (30 d)**, `stt.provider: local` (faster-whisper `base`), `tts.provider: ollama_local`; dashboard restarted 08:03:26. Verified by driving the whole call path for real, then by calls at **08:38:04** and **08:49:36**. ⚠ Two TTL keys exist — `dashboard.basic_auth.session_ttl_seconds` (raised) vs `dashboard.session_ttl_seconds` (**still 43 200**). `status: resolved`. - [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] — **The radar's two-stage split ran for real on 2026-10-28 and landed nothing: stage 1 stopped at the agent iteration cap before writing its handoff, so both stage-2 slots correctly no-oped (silent). Two consecutive stage-1 failures, two different ceilings (tool-call budget 10-27, iteration cap 10-28); no Notion writes for a second day.** **RESOLVED 2026-10-29** — the next stage-1 run (`dca8482e4f76`, 07:00:59 → 10:16:12, 3 h 15 m 13 s) closed `completed` *and* wrote `handoff/2026-10-29.json`, which stage 2 consumed at 10:40:59 and delivered as **20 pages + 2 cover letters** (`missing_cover_letter_count 0`). `status: resolved`, review 2026-11-13. - [[incidents/2026-10-27-job-radar-budget-exhausted]] — **The Daily Marketing Job Radar exhausted its tool-call budget for the second day running — and this time the run is recorded as `failed`, not `completed`.** On 2026-10-27 the 07:00 slot ran **07:01:08 → 11:03:54** (4 h 2 m 46 s) and its delivered report opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"*: discovery/verify/dedup/classify finished, **Notion writes, cover letters and the final audit never ran** (Notion DB held 451 entries, 0 added for the day). Incident row `dca848_e93a538ade6b` opened 11:03:53 (`alerted` 11:03:59, `failure_type: auth` — wrong for a budget fault) and was still open at the 2026-10-28 audit. **Response the same day:** the radar was split into stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) and new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030), created 17:34:15, two `completed` test fires that evening → [[runbooks/job-radar]]. **RESOLVED 2026-10-29:** both of this incident's ledger rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12** on the first successful post-split run — `dca8482e4f76` restored delivery to `local`, wrote the handoff inside budget, and no cover letter went missing (`missing_cover_letter_count 0`). `status: resolved`, review 2026-11-13. diff --git a/infrastructure/hosts.md b/infrastructure/hosts.md index f03580b..5b29268 100644 --- a/infrastructure/hosts.md +++ b/infrastructure/hosts.md @@ -3,8 +3,8 @@ title: Known Hosts and Paths type: infrastructure status: partial created: 2026-07-22 -updated: 2026-10-29 -verified_on: 2026-10-29 +updated: 2026-10-30 +verified_on: 2026-10-30 confidence: medium tags: [infrastructure, hosts, containers, paths] sources: [] @@ -20,11 +20,11 @@ sources: [] | litellm | LiteLLM proxy | `http://litellm:4000/v1` | Yes (from config) | | `custom:Omniroute:20128` | OmniRoute proxy (CT600 node `omniroute`) | `http://omniroute:20128/v1` → `100.88.81.19:20128` | Yes (live probe 2026-10-26: 6,480 models) | | `custom:omni-101` | OmniRoute proxy (CT101 node `omniroute-1`) — **the live default provider** | `http://100.78.63.20:20128` | Yes (live probe 2026-10-26) | -| CT201 | Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.166` (LAN) | Yes (live probe) | +| CT201 | Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.77` (LAN) | Yes (live probe 2026-10-30 — LAN address corrected from `192.168.178.166`) | | CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net`, `100.118.5.51` (Tailscale) | Yes | | PVE `pve` | Proxmox VE hypervisor hosting CT460 and the other `pve` guests | `100.96.100.82` (Tailscale), `pve.kangaroo-eel.ts.net` | Yes (live `pct list` 2026-10-26) | | PVE `m93p` | Second, unclustered Proxmox VE 9.2.10 hypervisor | `100.104.51.98` (Tailscale) | Yes (live `pct list` 2026-10-26) | -| CT450 (hermesagent) | Hermes Agent **v0.22.3** (2026.9.24), local `5b147ab036d6` — same build as CT460, hermes-webui, hermes-ui (Vite) | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes | +| CT450 (hermesagent) | Hermes Agent **v0.22.5** (2026.9.24), local `bf0203a794a0` — **same build as CT460**; hermes-webui, hermes-ui (Vite). ⚠ a stale **root-level** `/usr/local/bin/hermes` on CT450 still reports **v0.19.0** — the live agent is the `hermes`-user install under `~/.hermes/hermes-agent` | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes (live probe 2026-10-30 — version corrected from `v0.22.3`, build from `5b147ab036d6` to `bf0203a794a0`) | ## Project Paths @@ -35,7 +35,7 @@ sources: [] | `/home/hermes/.local/bin/hermes` | Hermes binary | | | `/home/hermes/.hermes/config.yaml` | Config | `_config_version: 51` | | `/home/hermes/.hermes/skills/` | Skills dir | 302 skills across 69 categories (`.archive/` excluded) | -| `/home/hermes/.hermes/cron/` | Cron jobs | 28 jobs — 25 enabled / 3 disabled | +| `/home/hermes/.hermes/cron/` | Cron jobs | 29 jobs — 25 enabled / 4 disabled (the 4th disabled job, `ea79a2107e36` Mnemosyne Health Monitor, was auto-paused 2026-10-30 01:04 after a gateway-connection failure) | | `/home/hermes/.hermes/mnemosyne/` | Mnemosyne DB | backups/, data/, logs/, models/ | | `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | cua-driver | v0.9.0 | | `~/.camofox/` | Camofox profiles | managed persistence | @@ -62,7 +62,7 @@ sources: [] - **PVE host `pve`:** `pve.kangaroo-eel.ts.net` (100.96.100.82) — **verified** via live `pct list` 2026-10-26 (17 LXCs) - **PVE host `m93p`:** 100.104.51.98, Proxmox VE 9.2.10 — **verified** via live `pct list` 2026-10-26 (19 LXCs). Guest IDs collide between the two hosts - **CT600 (`omniroute`):** OmniRoute server, `100.88.81.19` — **running and live**; `omniroute` is the hostname that resolves to it -- **CT601 (`omni`):** the old CT601 OmniRoute guest **no longer exists** on either hypervisor (`pct config 601` → no such config). The `omni` Tailscale peer (100.93.204.4) is its orphaned node, offline 69d +- **CT601 (`omni`):** the old CT601 OmniRoute guest **no longer exists** on either hypervisor (`pct config 601` → no such config on both `pve` and `m93p`). The orphaned `omni` Tailscale peer (`100.93.204.4`) has since **expired out of the tailnet entirely** — as of the 2026-10-30 live probe the tailnet contains **no** peer named `omni` and no peer at `100.93.204.4`, so the old "offline 69d" description no longer applies. The only OmniRoute nodes left are CT600 (`omniroute`) and CT101 (`omniroute-1`). - **CT101 (`omniroute`, m93p):** second OmniRoute node, `100.78.63.20` — running ## Unverified diff --git a/infrastructure/tailscale.md b/infrastructure/tailscale.md index 0a71a3f..4a76af4 100644 --- a/infrastructure/tailscale.md +++ b/infrastructure/tailscale.md @@ -3,8 +3,8 @@ title: Tailscale type: infrastructure status: active created: 2026-07-26 -updated: 2026-10-29 -verified_on: 2026-10-29 +updated: 2026-10-30 +verified_on: 2026-10-30 confidence: high tags: [tailscale, vpn, networking, infrastructure] --- @@ -19,12 +19,13 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access - **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`) - **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`) -## Tailscale Node Inventory (verified 2026-10-25) -Full node list from `tailscale status` — **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age. +## Tailscale Node Inventory (verified 2026-10-30) +Full node list from `tailscale status` — **50 non-funnel nodes** (49 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age. Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong). Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`. Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**. Re-verified 2026-10-21 03:0x: the 2026-10-19 table had drifted again — **`octop` (100.92.163.122), `springroll` (100.100.153.125), `openmausbot` and `openmausbot-vm` were missing from the table body** (only mentioned in prose), `omarchymac` had gone offline (online → idle/offline 3d), and every offline age was stale. Table rebuilt from `tailscale status --json`, keyed on DNS name rather than `HostName` (the JSON `HostName` for `freshrss` is `ct260`, and both `omniroute` nodes share the hostname `omniroute`). Container states cross-checked against live `pct list` on `pve` (100.96.100.82) and `m93p` (100.104.51.98): CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running. +Re-verified 2026-10-30 03:3x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 IP changes** (anthonys-mac-mini: 100.74.17.15 -> 100.76.60.37; omarchymac: 100.84.180.69 -> 100.126.173.115) and **4 nodes dropped out of the tailnet entirely** (anthonys-imac (100.112.123.124), cachyos (100.124.208.15), omarchymac-1 (100.126.173.115), omni (100.93.204.4)). `omarchymac` is now `100.126.173.115` (the peer formerly listed as `omarchymac-1`; the old `omarchymac` at `100.84.180.69` no longer matches any peer) and `anthonys-mac-mini` moved to `100.76.60.37` — both same-named nodes, new addresses. The four dropped peers are Tailscale expired/offline records (`anthonys-imac`, `cachyos`, `omni`, and the old `omarchymac` address), not newly-installed hardware. **`m93p` reads `Online: false` in the peer record for the third consecutive audit, yet the host is plainly up — `tailscale ping 100.104.51.98` → `pong from m93p via 192.168.178.118:41641 in 0s` and SSH returned `m93p / up 12 days, 11:02` — so the peer record is not authoritative for hypervisor reachability and the row records it as online. Node count corrected to **50 non-funnel nodes** (49 live IPv4 peers + `hermes` itself); `tailscale status --json` reports **72 peers total — 49 with a `100.` IPv4, 23 IPv6-only** (the `funnel-ingress-node` set). **Live IPv4 peers absent from the table: none.** Container identity re-confirmed from inside guests via `pct exec -- tailscale ip -4` on `pve` (100.96.100.82) and `m93p` (100.104.51.98); CT600 is confirmed **running** and CT601 confirmed **absent from both hypervisors**. Serve tables re-probed per host: CT460's `:443` route set matches the table below exactly (12 paths), and CT201 (`pct exec 201 -- tailscale serve status`) serves only `/` → `127.0.0.1:8085` and `/mgr` → `127.0.0.1:8084` on **both** `:443` and `:8443`, with no `/steel` or `/steel-api` routes remaining. Re-verified 2026-10-29 03:0x AWST (stocktake): State column rebuilt from `tailscale status --json` — **11 rows changed**. Two were **state transitions**, not age drift: **`sftpgo` (CT299, pve) came back online** (the 2026-10-28 audit had recorded it as `offline <1d`; live `tailscale ping 100.88.223.70` → `pong from sftpgo via 192.168.178.92:41641 in 0s`, `pct exec 299 -- systemctl is-active sftpgo` → `active`, `pct exec 299 -- tailscale ip -4` → `100.88.223.70`), and **`m93p` is online despite a stale peer record** — `tailscale status --json` still reports `Online: false` with `LastSeen 2026-10-28T18:40:00Z`, but `tailscale ping 100.104.51.98` → `pong from m93p via 192.168.178.118:41641 in 1ms` and SSH over the tailnet returned `m93p / up 11 days`. This is the **second** occurrence of the m93p stale-peer-record pattern (first seen 2026-10-26) — the peer record is not authoritative for hypervisor reachability. The remaining **9 rows were offline-age drift of exactly +1 day** (anthonys-imac 34d → 35d; cachyos 54d → 55d; cloudreve 95d → 96d — CT209 stopped; elementsynapse 98d → 99d — CT555 stopped; freshrss-new 71d → 72d; mattermost 38d → 39d — CT255 stopped; omarchybook 6d → 7d; omarchymac 10d → 11d; omni 71d → 72d — CT601 removed from both hypervisors; openmausbot 6d → 7d). Node count unchanged at **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself); `tailscale status --json` reports **76 peers total — 53 with a `100.` IPv4, 23 IPv6-only** (the `funnel-ingress-node` set). **Nodes in the table that are no longer in the tailnet: none. Live IPv4 peers absent from the table: none. IP mismatches: none.** Container identity re-confirmed from **inside** every LXC via `pct exec -- tailscale ip -4` across **both** hypervisors (`pve` 100.96.100.82, `m93p` 100.104.51.98) — 34 guests enumerated; every peer IP mapped to its CT as listed, and the three stopped guests (`CT209` cloudreve, `CT255` mattermost, `CT555` elementsynapse) and the three running-but-unconnected guests (`CT105` conduit, `CT1264` openmausbot, `CT100` obsidian-livesync) confirmed. CT601 remains absent from both hypervisors. Re-verified 2026-10-28 00:2x (nightly audit): State column rebuilt from `tailscale status --json` — **14 rows changed** (anthonys-imac: idle/offline 32d -> offline 34d; cachyos: idle/offline 52d -> offline 54d; cloudreve: idle/offline 93d — CT209 stopped -> offline 95d — CT209 stopped; conduit: idle/offline 49d -> offline 52d; elementsynapse: idle/offline 96d — CT555 stopped -> offline 98d — CT555 stopped; freshrss-new: idle/offline 69d -> offline 71d; m93p: online -> offline <1d; mattermost: idle/offline 36d — CT255 stopped -> offline 38d — CT255 stopped; omarchybook: idle/offline 4d -> offline 6d; omarchymac: idle/offline 8d -> offline 10d; omni: idle/offline 69d — CT601 removed from both hypervisors -> offline 71d — CT601 removed from both hypervisors; openmausbot: idle/offline 4d -> offline 6d; sftpgo: online — CT299 (pve) -> offline <1d — CT299 (pve); ubuntu-server-xfce: idle/offline 66d -> offline 68d). Header count corrected to **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself). Nodes in the table that are no longer in the tailnet: **none**. Live IPv4 peers absent from the table: **none**. **State transitions inside the audit window (not just age drift): `m93p` online → offline <1d, `sftpgo` (CT299) online → offline <1d.** @@ -34,29 +35,27 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st |---|---|---| | hermes | 100.118.5.51 | **self** — CT460 (pve) | | agentzero | 100.106.54.94 | online — CT475 (pve) | -| anthonys-imac | 100.112.123.124 | offline 35d | | anthonys-ipad | 100.111.198.120 | online | | anthonys-iphone | 100.100.116.67 | online | -| anthonys-mac-mini | 100.74.17.15 | online | -| cachyos | 100.124.208.15 | offline 55d | +| anthonys-mac-mini | 100.76.60.37 | online | | clawtest | 100.122.189.81 | online — CT333 (pve) | | cliproxyapi | 100.83.112.94 | online — CT103 (pve) | -| cloudreve | 100.111.187.7 | offline 96d — CT209 stopped | -| conduit | 100.81.79.87 | offline 52d | +| cloudreve | 100.111.187.7 | offline 97d — CT209 stopped | +| conduit | 100.81.79.87 | offline 53d | | docker | 100.96.150.69 | online — CT200 (pve) | | docker-01 | 100.121.12.31 | online — CT1001 (m93p) | -| elementsynapse | 100.75.250.37 | offline 99d — CT555 stopped | +| elementsynapse | 100.75.250.37 | offline 100d — CT555 stopped | | freellmapi | 100.127.208.75 | online — CT1006 (m93p) | | freshrss | 100.83.60.93 | online — CT1260 (m93p) | -| freshrss-new | 100.110.180.115 | offline 72d | +| freshrss-new | 100.110.180.115 | offline 73d | | gitea | 100.82.100.128 | online — CT221 (m93p) | | guest-vps | 100.117.14.84 | online | | hermesagent | 100.98.205.112 | online — CT450 (pve) | | homeassistant | 100.68.139.5 | online | | litellm | 100.79.29.64 | online — CT500 (m93p) | -| m93p | 100.104.51.98 | online — hypervisor (peer record stale: `tailscale ping` + SSH verified 2026-10-29) | +| m93p | 100.104.51.98 | online — hypervisor (peer record stale: `tailscale ping` + SSH verified 2026-10-30) | | mailflow | 100.115.60.92 | online — CT1262 (m93p) | -| mattermost | 100.98.116.14 | offline 39d — CT255 stopped | +| mattermost | 100.98.116.14 | offline 40d — CT255 stopped | | n8n | 100.84.124.30 | online | | nextcloud | 100.90.102.19 | online — CT270 (pve) | | ntfy | 100.67.153.116 | online — CT1241 (m93p) | @@ -64,13 +63,11 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | octop | 100.92.163.122 | online | | ollama | 100.108.44.119 | online — CT205 (pve) | | omarchy | 100.73.227.103 | online | -| omarchybook | 100.107.9.100 | offline 7d | -| omarchymac | 100.84.180.69 | offline 11d | -| omarchymac-1 | 100.126.173.115 | online | -| omni | 100.93.204.4 | offline 72d — CT601 removed from both hypervisors | +| omarchybook | 100.107.9.100 | offline 8d | +| omarchymac | 100.126.173.115 | online | | omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute | | omniroute-1 | 100.78.63.20 | online — CT101 (m93p) | -| openmausbot | 100.91.214.91 | offline 7d | +| openmausbot | 100.91.214.91 | offline 8d | | openmausbot-vm | 100.77.107.30 | online | | openweb | 100.69.230.13 | online — CT245 (m93p) | | paseo | 100.72.188.93 | online — CT1261 (m93p) | @@ -83,7 +80,7 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | social-dashboard | 100.70.183.37 | online — CT1263 (m93p) | | springroll | 100.100.153.125 | online | | springroll-browser | 100.109.221.91 | online | -| ubuntu-server-xfce | 100.101.26.104 | offline 68d | +| ubuntu-server-xfce | 100.101.26.104 | offline 70d | | vaultwarden | 100.123.225.90 | online — CT102 (m93p) | | webserver | 100.82.40.23 | online — CT207 (pve) | @@ -118,14 +115,14 @@ Re-verified against `tailscale serve status` on **2026-10-26** — the CT460 rou Re-verified 2026-10-26 (stocktake): the node table was rebuilt again from `tailscale status --json` — **`octoagent` (100.117.10.68) and `omarchymac-1` (100.126.173.115) were both missing entirely** and have been added; `m93p` (100.104.51.98) now reads **online** (it was "idle/offline 13m ago" — a stale Tailscale peer record, and the host was plainly reachable throughout); every offline age was refreshed (`mattermost` 35d → **36d**, `omni` 68d → **69d**, `elementsynapse` 95d → **96d**, `ubuntu-server-xfce` 64d → **66d**, `omarchymac` 7d → **8d**, `omarchybook` 3d → **4d**, `openmausbot` 3d → **4d**). The `omni` row now reads **"CT601 removed from both hypervisors"** rather than "CT601 stopped" — live `pct list` on both `pve` and `m93p` shows **no CT601 at all**, so the old guest is gone, not merely halted. Container identity re-confirmed from **inside** each LXC via `pct exec -- tailscale ip -4` on both hypervisors; every peer whose LXC has no Tailscale IP was checked against `pct list` (`CT105`/`CT1264` run but do not connect — `conduit` and `openmausbot` offline; `CT209`/`CT255`/`CT555` are `stopped`). Serve tables re-probed per host with `pct exec 201 -- tailscale serve status`. ## CT201 Serve Routes -Verified live on 2026-10-26 via `pct exec 201 -- tailscale serve status` on `pve` (100.96.100.82). The previous version of this table was wrong in two ways and has been corrected. +Re-verified live on 2026-10-30 via `pct exec 201 -- tailscale serve status` on `pve` (100.96.100.82). The previous version of this table was wrong in two ways and has been corrected. | Host / Port | Path | Target | Notes | |---|---|---|---| -| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Homepage asset proxy front door — **502** (re-probed 2026-10-26; upstream 8082 has no listener) | -| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` container — **200** | -| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Same asset proxy — **502** (re-probed 2026-10-26) | -| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` — **200** | +| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Homepage asset proxy front door — **502** (re-probed 2026-10-30; upstream 8082 has no listener — `curl http://127.0.0.1:8082/` → exit code 7 / `http=000`) | +| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` container — **200** (re-probed 2026-10-30) | +| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Same asset proxy — **502** (re-probed 2026-10-30) | +| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` — **200** (re-probed 2026-10-30) | **Steel routes removed 2026-10-29.** This table previously carried two more rows on `:443` — `/steel` → `http://127.0.0.1:5173` and `/steel-api` → `http://127.0.0.1:3000`. Both were deleted with @@ -134,7 +131,7 @@ are `/` → `127.0.0.1:8085` and `/mgr` → `http://127.0.0.1:8084`, on **both** → [[comparisons/steel-vs-camoufox]] Two corrections to the pre-2026-10-25 table: -1. **`:8443` is served by `tailscaled` on CT201, but so is `:443`.** `ss -ltnp` on CT201 shows `tailscaled` bound to `100.96.244.39:443`, and the serve table above lists routes on **both** ports. The old note claiming "Port 443 occupied by Traefik on CT201" is **false as of 2026-10-25** — no Traefik container or process exists on CT201 (`docker ps -a` and `ps aux` both came back empty for `traefik`), and Tailscale owns 443 there. +1. **`:8443` is served by `tailscaled` on CT201, but so is `:443`.** `ss -ltnp` on CT201 shows `tailscaled` bound to `100.96.244.39:443`, and the serve table above lists routes on **both** ports. The old note claiming "Port 443 occupied by Traefik on CT201" is **false as of 2026-10-25** — no Traefik **process** runs on CT201 (`ps aux` returns nothing for `traefik`), and Tailscale owns 443 there. *(Evidence corrected 2026-10-30: the original wording said `docker ps -a` "came back empty for `traefik`" — it does not. Live `docker ps -a` lists `runtipi-reverse-proxy | traefik:v3.2 | Exited (255) 2 months ago`. The container is **stopped**, not absent; the conclusion that Traefik does not occupy 443 stands.)* 2. The old table routed `:8443` straight to `http://localhost:8082`. That port has **no listener**; the live target is `8085` (the `asset_proxy.py` front door), which itself proxies to the dead 8082 and returns 502. → [[systems/homepage-dashboard]] ## Atomic Restore Script @@ -159,7 +156,7 @@ tailscale serve --set-path /jobs off ``` ## Notes -- **CT201 now serves Tailscale on BOTH `:443` and `:8443`** (verified 2026-10-25). The old "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there" claim was **wrong**: no Traefik container or process exists on CT201, and `tailscaled` owns 443 there (`ss -ltnp` shows it bound to `100.96.244.39:443`). The CT201 Serve Routes table above now records both ports. +- **CT201 now serves Tailscale on BOTH `:443` and `:8443`** (verified 2026-10-30). The old "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there" claim was **wrong**: no Traefik **process** runs on CT201, and `tailscaled` owns 443 there (`ss -ltnp` shows it bound to `100.96.244.39:443`). The CT201 Serve Routes table above now records both ports. *(Evidence corrected 2026-10-30: the original wording said "no Traefik container or process exists on CT201" — the container does exist, `runtipi-reverse-proxy | traefik:v3.2`, but it is **stopped** — `Exited (255) 2 months ago`. The conclusion that Tailscale owns 443 is unchanged.)* - CT460 has no conflict — Tailscale uses 443 directly, with the public webhook Funnel on `:8443`. - Routes must be re-added in order (restore script handles this). - After every rebuild, verify that `tailscale serve status` contains both the `:443` Serve routes and the `:8443/webhook` Funnel route. diff --git a/log.md b/log.md index 5ccc3a5..2c3bc7d 100644 --- a/log.md +++ b/log.md @@ -1584,3 +1584,65 @@ not pushed. `log.md`. The wiki-maintained pages under `infrastructure/` remain a separate, known gap. - **Removed/archived:** none. **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a key or token value. + +## 2026-10-30 — Stocktake (03:00 AWST): 0 stale by age, but 5 pages had drifted from live state + +- **Ran `wiki-maintain.sh`.** Regenerated `systems/skills-index.md` (**302 skills / 69 categories** — unchanged; the + generator de-dups one nested duplicate, so 302 is the correct unique-skill figure), wrote + `raw/stocktake/freshness-20261030.txt` (**0 stale / 109 fresh / 3 no-date of 112 pages**), rebuilt Quartz + (654 files) and deployed to `public/`. +- **Freshness is not accuracy.** The age scan found nothing, but a live-vs-page diff against the running estate found + real drift on five pages. All corrections below were verified against live sources before being written. +- **[[infrastructure/tailscale]] — node table rebuilt.** `tailscale status --json` now reports **50 non-funnel nodes** + (49 live IPv4 peers + `hermes`), not the 54 the page claimed. **2 IP changes** (`anthonys-mac-mini` 100.74.17.15 → + 100.76.60.37; `omarchymac` 100.84.180.69 → 100.126.173.115 — the peer formerly listed as `omarchymac-1`) and **4 nodes + dropped out of the tailnet entirely** (`anthonys-imac`, `cachyos`, `omarchymac-1`, `omni`). Every offline age refreshed. + `m93p` reads `Online: false` for the **third** consecutive audit yet is plainly up (`tailscale ping` → pong, SSH → `up 12 + days`), so the row records it online. Total peers **72** (49 IPv4 + 23 IPv6-only funnel). CT460's `:443` route set + matched the page exactly; CT201 serves only `/` → 8085 and `/mgr` → 8084 on both `:443` and `:8443`. +- **[[systems/scheduled-tasks]] — fleet ledger corrected.** Live `jobs.json` shows **29 jobs (25 enabled / 4 disabled)**, not + the 28/25/3 on the page: `ea79a2107e36` (Mnemosyne Health Monitor) was **auto-paused 2026-10-30 01:04:22** after its + 00:01:08 slot closed **`unknown`** on a gateway-connection failure. Non-clean `last_status` is **4** (not 3), and the + standing "every `failure_streak` reads 0" claim was false (`ea79a2107e36` = 1). "Failing (delivery only)" is **2**, not 1 + (`fad5a86979d4` joined `066bb5191280`). The retired DHL row was dropped from the job table (it lives in Removed Jobs). + Seven job rows carried **stale `error` claims** for jobs that have since cleared to `ok` — each was marked superseded + with a dated correction rather than rewritten (SCHEMA rule 9). The `executions.db` retention floor note was corrected to + the live **2026-10-24 09:22:36**. Window tallies: **2026-10-29 = 177 / 0** and **2026-10-28 = 169 / 2** (both +1 on the + page — the 23:59 Wiki Audit row had still been `running` at audit time). +- **New incident page: [[incidents/2026-10-30-mnemosyne-health-monitor-paused]].** The Mnemosyne Health Monitor's own + failure notice **failed to send** (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed`) — + `deliveries.db` holds **zero rows for 2026-10-30**, so the incident row is the only trace. The job was **not re-enabled** + and the gateway was **not restarted** by this run; root cause is **unverified** and recorded as such. +- **[[infrastructure/hosts]] — four corrections.** CT201's LAN address is `192.168.178.77`, not `192.168.178.166`; CT450's Hermes Agent is **v0.22.5** (2026.9.24), + local **`bf0203a794a0`** — the same build as CT460 — so the page's "v0.22.3 / 5b147ab036d6" was wrong on both + (a stale *root-level* `/usr/local/bin/hermes` on CT450 still reports v0.19.0 and is **not** the live agent — + the live one is the `hermes`-user install under `~/.hermes/hermes-agent`); the cron count is now 29/25/4; and + the orphaned `omni` + peer has **expired out of the tailnet entirely** (no peer named `omni`, no peer at 100.93.204.4), so the "offline 69d" + description no longer applied. The **skills count of 302 was left alone** — it is correct. +- **[[systems/homepage-dashboard]] — five corrections, and a second pass to fix my own first pass.** + "All Apps" is **114 → 123 live apps**, not 122 (the API dedupes by `(name, normalised href)` across groups; + the literal `All Apps` group in `services.yaml` holds 122 and is not what the tab bar counts). The grid is a + responsive `repeat(auto-fill, minmax(230px, 1fr))` (served `/dash.css` line 3640), not a fixed 5-column grid. + The tab list was wrong — but **my first correction was wrong too** and had to be redone: the live tab bar is + built from the service *groups* (`renderTabs()` iterates `state.groups`), **not** from the `layout:` block in + `settings.yaml`. That block and its `tab:`/`columns:` keys are **inert** — `grep -c "Overview" /dash.js` = 0 and + `grep -c "columns" /dash.js` = 0. There is **no "Overview" tab**; the live tabs are `☆ Pinned`, `All (123)` and + the **14 groups** (AI Quotas 3, Infrastructure 13, Everyday 8, Agents 20, Media 11, AI & LLM 17, Tools 20, + Storage 6, More 13, Resources 6, System Health 0, Subscriptions 1, Quick Launch 2, Uncategorised 3). The saved + theme is **`{"pair":"Boarding Pass","mode":"light"}`** — one of **59** theme pairs; its light-mode amber is + `#795000`, so the old "dark base with warm amber (#d4a04a)" was wrong on both counts. **`manager.py` runs inside + the `homepage-manager` container** (its container `Cmd` is the `pip install … && python3 /opt/homepage/manager.py` + line, confirmed via `docker inspect` and `docker top`); only **`asset_proxy.py`** is a host process + (`0.0.0.0:8085 users:(("python3",pid=4038))`). And the "no Traefik container exists" evidence was wrong — the + `runtipi-reverse-proxy` Traefik container exists but is **stopped** (`Exited (255) 2 months ago`); the *conclusion* + that Traefik does not occupy 443 stands. The same evidence error was corrected in [[infrastructure/tailscale]]. +- **[[current-state]]** and **[[index]]** updated with the new incident and the corrected fleet numbers. +- **Verified, not assumed:** CT600 **running** and CT601 **absent from both hypervisors** (`pct config 601` → no such + config on `pve` or `m93p`); the CT201 `:443`/`:8443` Serve tables; CT201 listeners 5173/3000/8082 (**none** — Steel's + removal confirmed); the homepage service groups and theme; and every cron counter, tally, incident row and delivery row + (read from `jobs.json`, `executions.db` — whose `cron_incidents` table lives inside it, the standalone + `cron_incidents.db`/`incidents.db` files are empty stubs — and `deliveries.db`). +- **Not changed:** `systems/skills-index.md` count (302 is correct), CT460's Serve route table (matched live exactly), and + the steel-removal records across the wiki (confirmed still true). +- **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a key or token value. diff --git a/raw/stocktake/freshness-20261030.txt b/raw/stocktake/freshness-20261030.txt new file mode 100644 index 0000000..178e96e --- /dev/null +++ b/raw/stocktake/freshness-20261030.txt @@ -0,0 +1,125 @@ +# Wiki freshness audit — 2026-10-30 +Threshold: >21 days since `updated` + +## STALE (0) + + +## FRESH (109) + +- [ 10d] CONTRIBUTING.md +- [ 4d] agents/hermes-chaos.md +- [ 4d] agents/hermes-production.md +- [ 10d] agents/index.md +- [ 1d] agents/nanobot.md +- [ 10d] agents/open-claw.md +- [ 19d] agents/qwenpaw.md +- [ 10d] agents/zeroclaw.md +- [ 1d] comparisons/camoufox-vs-chrome.md +- [ 10d] comparisons/hermes-vs-nanobot.md +- [ 10d] comparisons/memory-backends.md +- [ 10d] comparisons/model-routers.md +- [ 1d] comparisons/steel-vs-camoufox.md +- [ 10d] concepts/agent-memory.md +- [ 1d] concepts/browser-automation.md +- [ 10d] concepts/model-routing.md +- [ 10d] concepts/persistent-shells.md +- [ 10d] concepts/tool-calling.md +- [ 0d] current-state.md +- [ 1d] decisions/index.md +- [ 10d] decisions/keep-openrouter-as-fallback.md +- [ 10d] decisions/persistent-shell-enabled.md +- [ 1d] decisions/rollback-steel-to-camofox.md +- [ 10d] experiments/index.md +- [ 6d] incidents/2026-07-29-mnemosyne-relay.md +- [ 10d] incidents/2026-08-01-mnemosyne-capture-failure.md +- [ 6d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- [ 8d] incidents/2026-08-08-gitea-config-backup-push-dead.md +- [ 7d] incidents/2026-08-15-skills-index-drift.md +- [ 18d] incidents/2026-08-18-mnemosyne-capture-pipeline.md +- [ 5d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 10d] incidents/2026-08-29-gmail-read-state.md +- [ 9d] incidents/2026-09-02-sftpgo-outage.md +- [ 19d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 1d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 4d] incidents/2026-10-03-ct450-hermes-update.md +- [ 4d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 1d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 6d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 21d] incidents/2026-10-09-memory-recall-relevance.md +- [ 14d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 3d] incidents/2026-10-16-omniroute-503-resource-pressure.md +- [ 11d] incidents/2026-10-18-degenerate-stream-canned-substitute.md +- [ 3d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md +- [ 8d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md +- [ 5d] incidents/2026-10-22-conduit-memory-start-timeout.md +- [ 4d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md +- [ 4d] incidents/2026-10-24-conduit-call-request-refused.md +- [ 4d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md +- [ 2d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md +- [ 0d] incidents/2026-10-27-job-radar-budget-exhausted.md +- [ 0d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md +- [ 0d] incidents/2026-10-29-conduit-voice-silence.md +- [ 0d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md +- [ 0d] index.md +- [ 8d] infrastructure/backups.md +- [ 10d] infrastructure/credentials-map.md +- [ 1d] infrastructure/docker-services.md +- [ 10d] infrastructure/domains-and-tunnels.md +- [ 0d] infrastructure/hosts.md +- [ 16d] infrastructure/network-map.md +- [ 16d] infrastructure/proxmox.md +- [ 0d] infrastructure/tailscale.md +- [ 18d] queries/retained-research-answers.md +- [ 5d] reports/communities-wa-social-media-audit-2026.md +- [ 9d] reports/uber-eats/2026-07.md +- [ 9d] reports/uber-eats/2026-08.md +- [ 9d] reports/uber-eats/2026-09.md +- [ 9d] reports/uber-eats/index.md +- [ 10d] runbooks/backup-wiki.md +- [ 10d] runbooks/diagnose-docker.md +- [ 10d] runbooks/gateway-resource-resilience.md +- [ 0d] runbooks/job-radar.md +- [ 10d] runbooks/multiplexer-setup.md +- [ 4d] runbooks/provider-health.md +- [ 1d] runbooks/recover-docker-service.md +- [ 10d] runbooks/restart-browser.md +- [ 10d] runbooks/restart-hermes.md +- [ 10d] runbooks/restore-openrouter.md +- [ 10d] runbooks/test-openrouter.md +- [ 10d] runbooks/troubleshoot-tool-loop.md +- [ 4d] runbooks/update-hermes-safely.md +- [ 1d] systems/browser-backend.md +- [ 10d] systems/cloudflare-tunnel.md +- [ 0d] systems/conduit.md +- [ 4d] systems/current-profile.md +- [ 10d] systems/guanaco.md +- [ 10d] systems/headless-server.md +- [ 0d] systems/hermes-agent.md +- [ 10d] systems/hermes-ui.md +- [ 18d] systems/home-assistant.md +- [ 0d] systems/homepage-dashboard.md +- [ 19d] systems/ignis.md +- [ 4d] systems/memory-backend.md +- [ 4d] systems/messaging-integrations.md +- [ 6d] systems/mnemosyne.md +- [ 4d] systems/model-providers.md +- [ 18d] systems/nanobot.md +- [ 10d] systems/nextcloud.md +- [ 10d] systems/obsidian.md +- [ 8d] systems/omniroute.md +- [ 10d] systems/opencode-go.md +- [ 10d] systems/openrouter.md +- [ 19d] systems/paseo.md +- [ 11d] systems/push-notifications.md +- [ 0d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md +- [ 10d] systems/terminal-backend.md +- [ 10d] systems/tool-search.md + +## NO updated DATE (3) + +- SCHEMA.md +- experiments/active/example.md +- log.md + +TOTAL: 0 stale, 109 fresh, 3 no-date of 112 pages \ No newline at end of file diff --git a/systems/homepage-dashboard.md b/systems/homepage-dashboard.md index 22f0c12..0e122d3 100644 --- a/systems/homepage-dashboard.md +++ b/systems/homepage-dashboard.md @@ -3,8 +3,8 @@ title: Homepage Dashboard type: systems status: degraded created: 2026-07-26 -updated: 2026-10-29 -verified_on: 2026-10-29 +updated: 2026-10-30 +verified_on: 2026-10-30 confidence: high tags: [dashboard, homepage, homelab, proxmox, monitoring] --- @@ -32,30 +32,70 @@ The dashboard content itself is now **custom Python tooling** under `/opt/homepa - **Local:** `http://100.96.244.39:8082/` → **no listener** - **Manager UI:** `http://runtipi.kangaroo-eel.ts.net:8084/` → `homepage-manager`, up -## Architecture (verified 2026-10-20) +## Architecture (re-verified 2026-10-30) ``` CT201 (runtipi) ├── homarr container (port 8102) — live dashboard, HTTP 200 ├── homepage-manager container (8084) — web UI for adding/removing apps, up ├── homebox container (7745) — up -├── asset_proxy.py (python3, port 8085) — proxies 8085 → 8082, currently 502 -├── manager.py (python3) — dashboard management service, up +├── asset_proxy.py (python3, port 8085) — HOST process, proxies 8085 → 8082, currently 502 +├── manager.py (python3) — runs INSIDE the homepage-manager container (see below) └── Tailscale serve (ports 443 AND 8443) — / → 8085, /mgr → 8084 on both ports (the former /steel and /steel-api routes were removed 2026-10-29) ``` No container is listening on **8082** — `docker ps` on CT201 lists no `homepage` container at all. +> **Corrected 2026-10-30:** the old diagram implied `manager.py` also ran directly on the CT201 host. +> Live `docker inspect homepage-manager` shows its command is +> `["sh","-c","pip install --quiet --no-cache-dir PyYAML && python3 /opt/homepage/manager.py"]` +> with `/opt/homepage` bind-mounted — so **`manager.py` runs inside the `homepage-manager` container** +> (`MANAGER_PORT=8084`). Only **`asset_proxy.py` is a host process**: live `ss -ltnp` on CT201 shows +> `0.0.0.0:8085 users:(("python3",pid=4038))` — a bare `/usr/bin/python3 /opt/homepage/asset_proxy.py` +> with no container owning it. The host-side `:8084` listener is `docker-proxy`, not python. + ## Tabs -| Tab | Content | -|---|---| -| Overview | Infrastructure (2 cols) + Everyday (4 cols) + Resources | -| Media | Jellyfin, arr stack, RSS | -| AI | OmniRoute, LiteLLM, Lobe Chat, SearXNG | -| Agents | Hermes, OpenClaw, Nanobot, Camofox | -| Tools | IT tools, Cyber Chef, etc. + Storage | -| More | Home Assistant, N8n, Jupyter | -| All Apps | 114 apps in 5-column grid (alphabetical) | +> **Re-verified 2026-10-30 against the live manager API** (`GET http://127.0.0.1:8084/api/groups` on CT201), +> the served `/dash.js` and the served `/dash.css`. The old table below was wrong in several ways — the +> live **tab bar is built from the service *groups*** (`renderTabs()` in `dash.js` iterates `state.groups`), +> not from the `layout:` block in `settings.yaml`. That `layout:` block — and its `tab:`/`columns:` keys — +> is **inert**: `grep -c "Overview" /dash.js` returns **0**, and `grep -c "columns" /dash.js` returns **0**. +> There is **no "Overview" tab** and the `columns: N` values are never read. + +The live tab bar is: **`☆ Pinned`**, **`All (123)`**, then the **14 service groups** in order — each with a live +count badge: + +| Tab | Live count | Apps | +|---|---|---| +| ☆ Pinned | — | pinned apps only (19 pinned in `dash_prefs.json`) | +| All (123) | 123 | every app, deduped by `(name, normalised href)` — **corrected from 114** | +| AI Quotas | 3 | Claude, Codex, Ollama | +| Infrastructure | 13 | Gitea, Grafana, Guacamole, LAN watcher, PVE, Portainer docker, Portainer openweb, Runtipi, SFTPGO, Syncthing Runtipi, Theme Studio, cloudreve, filebrowser runtipi | +| Everyday | 8 | Homelab dashboard, Investbrain, Linkwarden, Mealie, NextCloud, cardholder, homebox, read later | +| Agents | 20 | AISBF Dashboard, Agent Dashboard, Agent zero, Camofox VNC chaos, Camofox VNC rhino, Chaos webUI, Herdr omarchy, Hermes Control, Hermes web UI (rhino), Hindsight chaos, Honcho memory dashboard, Mausbot ×2, Octoagent, PAN Hermes, Zeroclaw dash, herdr omarchybook, hermes desktop rhino, hermes workspace, nanobot dashboard | +| Media | 11 | Audio bookshelf, Flaresolverr, FreshRSS, Jellyfin, MeTube, NewsKu, Prowlarr, RSS-Bridge, Transmission, YouTube, qBitTorrent | +| AI & LLM | 17 | Agent estate wiki, Cliproxy api, FreeLLMAPI, Khoj ADMIN, Khoj SH, LiteLLM, Lobe chat (docker), Manifest router, Omniroute gateway, Opencode server, Openweb, Reasonix, Searxng, Simple chat ai, Vercel ai cloud, Voice Studio, World monitor | +| Tools | 20 | 8mb video compression, Cobalt, Convert to it, Cyber chef, Dockage, Element synapse, Excaldraw, IT tools, Matermost, Quiet thanks, Retroarch, Social dashboard (communities), Stirling PDF, Subscription tracker, T3 code frontend, Wordpress, code-server debian, compose craft, convert anything?, subtitle generator | +| Storage | 6 | Obsidian agent vault, filebrowser Debby, filebrowser DebianVM, filebrowser DebianVM NEW, filebrowser Docker, guacano | +| More | 13 | Cloudfare apps, GitHub, Kubuntu desktop, N8n, Next step, Omni tools, Reactive resume, Sunshine Ubuntu's-server, Yeah koko tts, hermes desktop, hermes desktop chaos, home assistant, jupyter server | +| Resources | 6 | Communities reporting - socials, Dashboard Manager, Helper Scripts, Homelab Docs, RADAR, Self-Hosted Directory | +| System Health | 0 | *empty* — the tab renders but holds no app cards (the health panel is separate, widget-driven) | +| Subscriptions | 1 | Renewals | +| Quick Launch | 2 | Paseo, Portainer | +| Uncategorised | 3 | Hermes desktop ui, Openchamber, Vaultwarden | + +**App-count reconciliation (two different numbers, both correct in context):** +- **123** — the live count the UI shows on the `All (123)` tab. `get_all_apps_flat()` walks every group except + `All Apps` and `AI Quotas & Rate Limits`, then **dedupes by `(name, normalised href)`** — so an app defined in + two groups collapses to one card (e.g. `Mausbot` appears in both `Agents` and `Tools`). +- **122** — the literal `All Apps` group in `/opt/homepage/config/services.yaml`, a generated roll-up that is + **not** what the tab bar counts. The old page's "114" was simply wrong. + +**Grid layout (corrected):** the served `/dash.css` (385 121 bytes, = `/opt/homepage/config/dashboard.css`) +sets `.app-grid` at **line 3640** to `grid-template-columns: repeat(auto-fill, minmax(230px, 1fr))` — a +**responsive auto-fill**, not a fixed 5-column grid. Narrower breakpoints override it to +`repeat(auto-fill, minmax(160px, 1fr))` (line 5001) and then `1fr 1fr` (line 5010). The `columns:` figures in +`settings.yaml` are dead config. ## Proxmox Integration - **Node widget:** Shows live VM/LXC counts, CPU, memory for PVE node @@ -112,15 +152,28 @@ Web form at port 8084 for adding/removing apps without SSH. Type anywhere on the dashboard to search services by name/description. Google search disabled — only searches your apps. ## Theme -- Dark base with warm amber (#d4a04a) accents +- **Corrected 2026-10-30:** the live saved theme is **`{"pair":"Boarding Pass","mode":"light"}`** — read from + `/opt/homepage/config/dash_prefs.json` key `dash.theme`. The previous "Dark base with warm amber + (#d4a04a) accents" claim is **wrong on both counts**: the mode is **light**, not dark, and + `#d4a04a` is not this theme's amber at all. **Boarding Pass** (`chassis: boarding`, "Boarding pass desk — + perforated stubs, gate codes, seat and route labels") is one of **59 theme pairs** in + `/opt/homepage/config/themes.json` under `dashboard.pairs`; in its **light** mode the amber accent is + **`#795000`** (dark mode would be `#f5c96a`). The dashboard also ships 26 legacy `homepage:` themes + (`obsidian-midnight`, `nord-frost`, `cyberpunk-terminal`, …) and three saved `custom:` themes in + `dash.customThemes` (E2E Test Theme, Whiteboard Ops 1, Evidence Wall 1), each with its own chassis and palette. - Bigger tiles (68px min-height, 40px icons) - Custom CSS in `custom.css` ## Tailscale Serve **CT201 serves Tailscale on BOTH `:443` and `:8443`** (verified live 2026-10-25 via `pct exec 201 -- tailscale serve status`). The earlier note on this page — "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 instead" — was **wrong**: -no Traefik container or process exists on CT201 (`docker ps -a` and `ps aux` both return nothing for `traefik`), and +no Traefik **process** runs on CT201 (`ps aux` returns nothing for `traefik`), and `ss -ltnp` shows `tailscaled` bound to `100.96.244.39:443`. Corrected on 2026-10-25. +> **Evidence corrected 2026-10-30:** the old wording said `docker ps -a` "returns nothing for +> `traefik`". That was **not accurate** — live `docker ps -a` on CT201 does list +> `runtipi-reverse-proxy | traefik:v3.2 | Exited (255) 2 months ago`. The **conclusion is unchanged** +> (Traefik is not running and does not occupy 443; `tailscaled` is bound to `100.96.244.39:443`), +> but the container is *stopped*, not absent. ## Maintenance - **Restart Manager:** `docker restart homepage-manager` @@ -139,7 +192,7 @@ no Traefik container or process exists on CT201 (`docker ps -a` and `ps aux` bot CT201's remaining routes are only `/` → 8085 and `/mgr` → 8084 on both `:443` and `:8443`. → [[infrastructure/tailscale]], [[comparisons/steel-vs-camoufox]] - **2026-10-25 (live probe, stocktake):** the "Port 443 occupied by Traefik on CT201" note was **false** — no Traefik - container or process exists on CT201, and `tailscaled` owns **both** `:443` and `:8443` there. Serve tables corrected: + **process** runs on CT201 (the `runtipi-reverse-proxy` Traefik container exists but is **stopped** — `Exited (255) 2 months ago` — re-verified 2026-10-30), and `tailscaled` owns **both** `:443` and `:8443` there. Serve tables corrected: `/` → 8085 and `/mgr` → 8084 on both ports *(the `/steel` and `/steel-api` routes listed at the time were deleted on 2026-10-29 — see above)*. Re-confirmed live: 8084 → **200**, 8085 → **502**, 8082 → **no listener**, `homepage-manager` up. → [[infrastructure/tailscale]] diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index f38a7d1..ee188b5 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count -- **Current as of 2026-10-30 00:0x AWST (live `jobs.json` + `executions.db`) — the radar's two-stage split worked end to end for the first time, and the fleet had its first clean day in a week.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3**, and every `failure_streak` reads **0** — the three survivors are all stale re-runs of long-standing rows (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`). Window **2026-10-29 settled 176 `completed` / 0 `failed`** (+ this audit `running`) — the first zero-failure day since 2026-10-24; 2026-10-28 stands at **168 / 2**. **Unclosed incident rows 4 → 1:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving only the Facebook monthly's `0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report). `dca8482e4f76` (stage 1) ran **07:00:59 → 10:16:12** (**3 h 15 m 13 s**), closed **`completed`**, and wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok` — opening **no** new incident row (contrast the two consecutive stage-1 failures on 10-27 and 10-28). `5fc292ff0f3b` (stage 2) fired **10:13:46**, consumed the handoff **10:40:59**, closed **10:44:48 `completed`** and delivered the same second: **20 pages / 20 `page_ids` / 2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went **455 → 475**. Both later slots no-oped as designed (**11:30 → 10 m 46 s**, **14:30 → 1 m 51 s**). Stage 1 correctly no longer delivers. See [[incidents/2026-10-27-job-radar-budget-exhausted]] and [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], now both **resolved**. **Pins unchanged** — 19 of the 26 enabled jobs pin `custom:omni-101` (12 `smart-route`, 6 `free-stack`, 1 explicit). +- **Current as of 2026-10-30 03:3x AWST (live `jobs.json` + `executions.db`) — the 10-29 clean day held, but the fleet changed shape overnight: the Mnemosyne Health Monitor was auto-paused and its failure notice could not be delivered.** Fleet now **29 jobs (25 enabled / 4 disabled)** — `ea79a2107e36` (Mnemosyne Health Monitor) was **auto-paused at 2026-10-30 01:04:22** after its 00:01:08 slot closed **`unknown`** on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, which opened incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`). Non-clean `last_status` is now **4** — the three long-standing survivors (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`) **plus `ea79a2107e36` (`error`, `failure_streak: 1`)** — so the standing "every `failure_streak` reads 0" claim is now false. Window **2026-10-29 settled 177 `completed` / 0 `failed`** (the page previously said 176 — the 23:59 Wiki Audit row `cbfe434a3b5e` had still been `running` at audit time and has since settled `completed`); **2026-10-28 stands at 169 / 2** (also +1, same cause). **Unclosed incident rows 1 → 2:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving the Facebook monthly's `0e753e_e7f8194a7d06` (`detected`, cannot clear before its 2026-11-01 slot) **plus the new `ea79a2_d93c3acdc4f2`**. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report); **zero delivery rows were created on 2026-10-30**, because the Mnemosyne Health Monitor's own failure notice **failed to send** (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed`) — a failure notice that was itself never delivered. The radar's two-stage split held its ground: `dca8482e4f76` (stage 1) and `5fc292ff0f3b` (stage 2) both read `ok` with streak 0, and every previously-red row (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) has since **cleared to `ok`** on its own scheduled slot — the page's older per-row notes still claimed `error` for several of them. **Pins: 18 of the 25 enabled jobs pin `custom:omni-101`** (11 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`); the other 7 enabled rows are `no_agent` jobs with no pin. +- **Prior window — as of 2026-10-30 00:0x AWST *(superseded by the 2026-10-30 03:3x entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split worked end to end for the first time, and the fleet had its first clean day in a week.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3**, and every `failure_streak` read **0** — the three survivors are all stale re-runs of long-standing rows (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`). Window **2026-10-29 settled 176 `completed` / 0 `failed`** (+ this audit `running`) — the first zero-failure day since 2026-10-24; 2026-10-28 stands at **168 / 2**. **Unclosed incident rows 4 → 1:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving only the Facebook monthly's `0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report). `dca8482e4f76` (stage 1) ran **07:00:59 → 10:16:12** (**3 h 15 m 13 s**), closed **`completed`**, and wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok` — opening **no** new incident row (contrast the two consecutive stage-1 failures on 10-27 and 10-28). `5fc292ff0f3b` (stage 2) fired **10:13:46**, consumed the handoff **10:40:59**, closed **10:44:48 `completed`** and delivered the same second: **20 pages / 20 `page_ids` / 2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went **455 → 475**. Both later slots no-oped as designed (**11:30 → 10 m 46 s**, **14:30 → 1 m 51 s**). Stage 1 correctly no longer delivers. See [[incidents/2026-10-27-job-radar-budget-exhausted]] and [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], now both **resolved**. **Pins unchanged** — 19 of the 26 enabled jobs pin `custom:omni-101` (12 `smart-route`, 6 `free-stack`, 1 explicit). - **Prior window — as of 2026-10-29 00:3x AWST *(superseded by the 2026-10-30 entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real, but stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.** - **Prior window — as of 2026-10-28 00:2x AWST** (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. *(superseded by the 2026-10-29 entry above)* - **Prior window — as of 2026-10-27 00:1x AWST *(superseded by the 2026-10-28 entry above)* (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`). @@ -28,15 +29,15 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] - **Current as of 2026-10-19 00:20 AWST *(superseded by the 2026-10-20 entry above)* (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]] - **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01; DB re-read 2026-10-20: **163 / 5** — one row was double-counted at the time) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] -- **Total jobs:** **28 — 25 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json` at 2026-10-19 00:20 AWST). The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. -- **Non-clean `last_status`:** **12 as of 2026-10-20 00:09** — 10 `error` + 2 `delivery_failed`. The burst re-dirtied four rows that were clean at the 10-19 audit (`dca8482e4f76`, `85edf6772bca`, `08eb86f02d37`, `c0a9fa314542`) and deepened two of the existing seven (`0e519874d280` and `ecc1fc13655a` both to `failure_streak: 2`); `fad5a86979d4` newly reads `delivery_failed` (bare-`telegram` target). `b63ed5817212` (dirtied twice) and `96eb95ce12f2` (four derivative reds) **cleared the same day** (18:01:56 / 12:46:11), and `ea79a2107e36` cleared on its 10-20 00:01 slot. Survivors, with next run: `ecc1fc13655a` (streak **2**, 10-20 12:00), `0e519874d280` (**2**, Thu 10-22 08:00), `1b23c042e17d` (**2**, Fri 10-23 08:00), `dca8482e4f76` (1, **10-20 07:00**), `85edf6772bca` (1, **10-20 07:00**), `08eb86f02d37` (1, 10-20 08:30), `c0a9fa314542` (1, **Mon 10-26 09:00**), `06a0a1c8704f` (1, Thu 10-22 10:00), `c8d52904e1d6` (1, Thu 10-22 18:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 0, 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → **12 now**. Three of these fire within eight hours of this audit, so the count will move again by morning. -- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]]. +- **Total jobs:** **29 — 25 enabled, 4 disabled** (verified from live `~/.hermes/cron/jobs.json` at 2026-10-30 03:3x AWST). The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. The **4th disabled job is `ea79a2107e36` (Mnemosyne Health Monitor)**, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; the other three are the long-standing `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher) and `0a7f933d06e2` (AgentMail Rhino Inbox Poll). +- **Non-clean `last_status`:** **4 as of 2026-10-30 03:3x AWST** — 2 `error` + 2 `delivery_failed`. Live `jobs.json` (authoritative for the per-job counters): `ea79a2107e36` (Mnemosyne Health Monitor — `error`, `failure_streak: 1`, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; incident `ea79a2_d93c3acdc4f2` open `detected`), `0e753e9720fd` (Facebook Birthday Monthly Scrape — `error`, streak 0, next slot 11-01 09:00), `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — `delivery_failed`, streak 0, most recent failure **2026-10-26 03:08:15**) and `066bb5191280` (Uber Eats Monthly Spend Report — `delivery_failed`, streak 0, next slot 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → 12 (10-20 00:09) → 3 (10-30 00:0x) → **4 now**, the increase being the Mnemosyne Health Monitor's overnight gateway fault. +- **Failing (last run failed outright):** **1** — `ea79a2107e36` (Mnemosyne Health Monitor), `failure_streak: 1`. Its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, and the job was **auto-paused** at that moment — it is now the 4th disabled job. ⚠ **Its failure notice was never delivered**: `hermes cron list` reports *"The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"*, and `deliveries.db` holds **zero rows for 2026-10-30**. A failure notice that fails to send is the same green-row/dead-job trap as [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the incident row `ea79a2_d93c3acdc4f2` is the only trace. The other previously-red jobs (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) have all **cleared to `ok` with streak 0** on their own scheduled slots. - Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04). - **Date-bucketing note (added 2026-10-25):** `executions.db` stores `started_at` with a `+08:00` offset, and SQLite's `date()` normalises to UTC — so `date(started_at)` moves every row that ran 00:00–07:59 AWST onto the **previous** day. Daily tallies on this page are bucketed with `substr(started_at,1,10)`, which preserves AWST calendar days. A recount via `date()` reads low by roughly a day's early-morning runs; this is the same class of drift that put 10-21 at 168 instead of 171. -- **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window. +- **Failing (delivery only):** **2** — `066bb5191280` (Uber Eats Monthly Spend Report: the report itself generated; the send failed against the DM thread — bare `telegram` target, 2026-10-01) and `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — the same dead-thread cause, most recently **2026-10-26 03:08:15**). Both carry `failure_streak: 0` because the *run* succeeded; only the delivery failed. - **Repin applied 2026-10-05:** `free-stack` ×6 (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`) and `smart-route` ×1 (`dca8482e4f76`), all on `custom:omni-101`. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. `free-stack` is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs. - **Verified live, not assumed:** `79f9bb41ecf5` succeeded on a manual run 2026-10-05 23:39 and `06a0a1c8704f` at 23:45 (both incidents auto-resolved); `dca8482e4f76` completed 23:56 and delivered to `telegram:-1004321904721:1030` (incident resolved 2026-10-06 00:05:07); and `1a8e90ea2271` completed automatically on **2026-10-06 03:00**. Four of the seven are therefore verified by live run. The remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are repinned but not yet re-run — they are due on their normal schedules. `1a8e90ea2271` has since run **twice more on schedule** (2026-10-06 03:00 → `33f4e95`; 2026-10-07 03:00 → `8c63ef2`), so the stocktake repin is verified across consecutive automatic runs. -- **Streak counting note:** the `N failures in a row` figure in `hermes cron list` comes from the `failure_streak` counter in `jobs.json`. `executions.db` retains only rows from **2026-09-30 onward**, so DB-based recounts undercount older streaks. Live counters as of 2026-10-06: **all 0**. +- **Streak counting note:** the `N failures in a row` figure in `hermes cron list` comes from the `failure_streak` counter in `jobs.json`. `executions.db` retains only a rolling window of rows — the live floor is now **2026-10-24 09:22:36** (1,001-row cap), so DB-based recounts undercount older streaks and cannot reproduce tallies for days before that floor. Quote the `jobs.json` counter as authoritative and state the floor whenever a DB recount is cited. Live counters as of 2026-10-30 03:3x AWST: **`ea79a2107e36` = 1; every other job = 0.** - **Re-verified 2026-10-06 AWST:** job count unchanged at 30 (27 enabled / 3 disabled). `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found`). - **Re-verified 2026-10-07 AWST:** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — all 30 `failure_streak` counters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176 `completed`, 0 `failed`, 1 `running` (the audit itself); 14 sessions, all `cron` — **no interactive (non-cron) sessions**. The stocktake job `1a8e90ea2271` ran on schedule at 03:00 and regenerated [[systems/skills-index]] (344 → 346 skills) plus `raw/stocktake/freshness-20261006.txt`, committing to Gitea as `33f4e95` — the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repin `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`). - **A stale notice, not a live outage, drove this review.** The interactive session on 2026-10-05 23:16 began because Anthony replied to a `85edf6772bca` failure notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check `executions.started_at` (and the `cron_incidents` state) before acting on a notice. @@ -52,27 +53,26 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) | | AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 | | Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | | -| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | `local` (failures → topic 1030) | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00. ✅ **First full split run 2026-10-29:** 07:00:59 → 10:16:12 (3 h 15 m 13 s), `completed`, wrote `handoff/2026-10-29.json` (170 discovered / 33 duplicates avoided / 37 verified); both of its ledger rows closed `resolved` at 10:16:12. **Delivery is now `local`** — stage 2 owns the report | +| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | `local` (failures → topic 1030) | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00. ✅ **First full split run 2026-10-29:** 07:00:59 → 10:16:12 (3 h 15 m 13 s), `completed`, wrote `handoff/2026-10-29.json` (170 discovered / 33 duplicates avoided / 37 verified); both of its ledger rows closed `resolved` at 10:16:12. **Delivery is now `local`** — stage 2 owns the report ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, `failure_streak: 1`" note above is **superseded**; the job cleared on its own 10-20 07:00 slot and has run clean since, including the successful 2026-10-29 split run. | | Daily Marketing Job Radar — Write and Letters (stage 2) | `5fc292ff0f3b` | `30 11,14 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Created 2026-10-27 17:34** — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). `custom:omni-101` / `smart-route`; first fires 2026-10-27 17:35 and 17:38 both `completed`. ✅ **First full split run 2026-10-29:** fired 10:13:46, consumed the handoff at 10:40:59, closed `completed` 10:44:48 and delivered the same second — **20 pages + 2 cover letters, `missing_cover_letter_count 0`**, Notion board 455 → 475; the 11:30 and 14:30 slots then no-oped correctly. See [[incidents/2026-10-27-job-radar-budget-exhausted]] | -| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant | -| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] | +| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "still carries `last_status: error` … now **streak 2**" note above is **superseded**; the job cleared on its Fri 2026-10-23 08:00 slot and has run clean since. | | Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent | | Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident `0e753e_e7f8194a7d06` (opened 2026-10-01 09:00:55) is **still open** in state `detected`, and `last_status` still reads `error`; **next slot 2026-11-01 09:00**, which is when the repin finally gets its live test. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic | | Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). ✅ **Verified live 2026-10-12 03:00** — completed and delivered (448 birthdays); the stale `last_status: error` cleared after two weeks and **both** `deepseek-v4-flash` incidents auto-closed at 03:05:39 (`fad5a8_a13bbc837e1c`, `fad5a8_c80b59a907f0`). ⚠ **Live status is no longer `ok`:** the 2026-10-19 03:04:47 run completed but recorded `delivery_failed` — `configured thread_id 34795 for telegram:1793951355:34795 was not found` (the CLI renders it as "ran, but the result was not delivered", then "delivered without thread_id"). The run is fine; only the send is. **No incident row was created for this.** ⚠ Bare `telegram` target should move to a group topic | -| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | ⚠ The **2026-10-16 12:00** slot failed on the `omni-101` 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed that incident. ⚠ **Then 2026-10-18 12:00 failed differently** — a model repetition loop aborted the run at 12:08:25 **after** it had delivered a canned "Your request was not processed…" line to the topic (12:08:11) → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. ⚠ **Then the 2026-10-19 12:01 slot failed again, on the `omni-101` 503** — `ecc1fc_c9976ea24553` (a **10-16** row, refreshed 10-19 12:02:58, `alerted`) plus the separate `ecc1fc_21a72aa11cc4` degenerate-stream row from 10-18, so **this one job holds two open incidents**. Live `last_status: error`, `failure_streak: 2`, next slot **2026-10-20 12:00** | -| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** ⚠ **The status reading changed again on 2026-10-19:** its scheduled Monday slot (`0 9 * * 1`) at **09:01 failed on the `omni-101` 503** (new incident `c0a9fa_764558b5ae83`, `alerted`), so **this week's roundup did not run at all** and `last_status` now reads `error` (streak 1). The token question is therefore **untested for a second consecutive Monday** — the job has not reached `digest.py` on a Monday since 2026-10-12. Next real test: **Mon 2026-10-26 09:00**. | -| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | ⚠ The **2026-10-16 08:30** slot failed on the `omni-101` 503 (streak 2 at the storm peak); and its **own 10-17 08:31** run restored a clean `last_status`. ⚠ **But the incident never closed: `08eb86_472c7b609c80` was refreshed on 2026-10-19 08:33 by the same 503, and is still `alerted`.** Live `last_status: error`, streak 1 | +| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | ⚠ The **2026-10-16 12:00** slot failed on the `omni-101` 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed that incident. ⚠ **Then 2026-10-18 12:00 failed differently** — a model repetition loop aborted the run at 12:08:25 **after** it had delivered a canned "Your request was not processed…" line to the topic (12:08:11) → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. ⚠ **Then the 2026-10-19 12:01 slot failed again, on the `omni-101` 503** — `ecc1fc_c9976ea24553` (a **10-16** row, refreshed 10-19 12:02:58, `alerted`) plus the separate `ecc1fc_21a72aa11cc4` degenerate-stream row from 10-18, so **this one job holds two open incidents**. Live `last_status: error`, `failure_streak: 2`, next slot **2026-10-20 12:00** ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, `failure_streak: 2`" note above is **superseded**; the job cleared on its own daily 12:00 slot after 2026-10-19. | +| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** ⚠ **The status reading changed again on 2026-10-19:** its scheduled Monday slot (`0 9 * * 1`) at **09:01 failed on the `omni-101` 503** (new incident `c0a9fa_764558b5ae83`, `alerted`), so **this week's roundup did not run at all** and `last_status` now reads `error` (streak 1). The token question is therefore **untested for a second consecutive Monday** — the job has not reached `digest.py` on a Monday since 2026-10-12. Next real test: **Mon 2026-10-26 09:00**. ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "reads `error` (streak 1)" note above is **superseded**; the job ran clean on its Mon 2026-10-26 09:00 slot, so the 10-19 outage has cleared. The **revoked-bot-token** finding above stands as a separate, still-unverified delivery-path question — that Monday run reached the model without raising a 503, which does not by itself prove the token works. | +| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | ⚠ The **2026-10-16 08:30** slot failed on the `omni-101` 503 (streak 2 at the storm peak); and its **own 10-17 08:31** run restored a clean `last_status`. ⚠ **But the incident never closed: `08eb86_472c7b609c80` was refreshed on 2026-10-19 08:33 by the same 503, and is still `alerted`.** Live `last_status: error`, streak 1 ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, streak 1" note above is **superseded**; the job cleared on its own daily 08:30 slot after 2026-10-19. | | Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) | | Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent | | Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent | | Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent | | Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95%. ⚠ **Both 503-era slots failed** (10-16 12:00, 10-17 00:00 — two of the four watchdog reds quote this job's error); cleared by its own **10-17 06:01** run. ⚠ **Both 2026-10-19 slots failed the same way** (06:01:05 and 12:01:16), taking its streak to **2** and re-triggering the watchdog; its **18:01 slot succeeded**, clearing the streak and auto-closing `b63ed5_bbe44c89aac1` at 18:01:56. Live `last_status: ok`. With `0 */6 * * *` this job is the fleet's best 503 sentinel — four probes a day | -| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. ⚠ Same two 503 slots in burst 1 (10-16 12:00, 10-17 00:00), then **again at 2026-10-19 12:01** — incident `ea79a2_747b27556fd0` was first opened 10-16 12:03, refreshed 10-19 12:02:55, and **auto-closed 2026-10-20 00:04:24** when the 00:01 slot succeeded. Live `last_status: ok`, streak 0 | +| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. ⚠ Same two 503 slots in burst 1 (10-16 12:00, 10-17 00:00), then **again at 2026-10-19 12:01** — incident `ea79a2_747b27556fd0` was first opened 10-16 12:03, refreshed 10-19 12:02:55, and **auto-closed 2026-10-20 00:04:24** when the 00:01 slot succeeded. Live `last_status: ok`, streak 0 ⚠ **Live status re-verified 2026-10-30 03:3x — the job is now DISABLED and failing:** its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, opening incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`), and the scheduler **auto-paused the job at that moment**. Live `last_status: error`, `failure_streak: 1`, `enabled: false`. The "Live `last_status: ok`, streak 0" note above is **superseded**. ⚠ **The failure notice was never delivered** — `hermes cron list` reports *"live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"* and `deliveries.db` has **zero rows for 2026-10-30**. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. | | Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30). ⚠ **Its reds are derivative — it double-counts other jobs' failures.** During the 503 burst it exited 1 on **4 slots** (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own error from the log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…`. One sentinel watching another job, not four independent faults. **The pattern repeated exactly in burst 2** — four more reds on 2026-10-19 (06:16, 06:31, 12:16, 12:31), all quoting the same coverage-monitor error, all clearing at **12:46:11** (`96eb95_565ec1deb1c7`, a 10-16 row refreshed through 10-19 12:31). **8 of the 26 failure rows across the two bursts were this one sentinel** | | Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified `delivered`, status `ok` | -| Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | ⚠ Two node faults, no code fault: 2026-10-15 08:06 `RuntimeError: Connection error.` (`0e5198_be2291809e56`) and **2026-10-19 08:01 `HTTP 503 … resource pressure`** (new incident `0e5198_ec2fac98a91d`, `alerted`). Both are open; `failure_streak: 2`. Next slot Thu 2026-10-22 08:00 | | +| Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | ⚠ Two node faults, no code fault: 2026-10-15 08:06 `RuntimeError: Connection error.` (`0e5198_be2291809e56`) and **2026-10-19 08:01 `HTTP 503 … resource pressure`** (new incident `0e5198_ec2fac98a91d`, `alerted`). Both are open; `failure_streak: 2`. Next slot Thu 2026-10-22 08:00 | ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "`failure_streak: 2`" note above is **superseded**; the job cleared on its Mon/Thu 08:00 slots after 2026-10-19. | | Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) | -| Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | ⚠ **2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `85edf6_256c5e150bc4`, `alerted`) — its first failure since its 10-15 connection-refused red cleared on 10-16. Live `last_status: error`, streak 1, next slot 2026-10-20 07:00 | | +| Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | ⚠ **2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `85edf6_256c5e150bc4`, `alerted`) — its first failure since its 10-15 connection-refused red cleared on 10-16. Live `last_status: error`, streak 1, next slot 2026-10-20 07:00 | ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, streak 1" note above is **superseded**; the job cleared on its own daily 07:00 slot after 2026-10-19. | | Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (`configured thread_id 34795 for telegram:1793951355 was not found`) — the 2026-09 report generated fine, only the send failed. Bare `telegram` → home DM; retarget to a group topic | | Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the `cliprox/terra` pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ **Repinned 2026-10-05 23:39** → `custom:omni-101` / `free-stack` and **verified by a live manual run**; the incident auto-resolved. Now `last_status: ok`, streak 0 | | weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 | @@ -95,6 +95,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] | Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. | | Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. | | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. | +| Mnemosyne Health Monitor | `ea79a2107e36` | **2026-10-30 01:04:22** | **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. | ## Known Issues diff --git a/systems/skills-index.md b/systems/skills-index.md index 69bc54c..5746d11 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-10-29 -verified_on: 2026-10-29 +updated: 2026-10-30 +verified_on: 2026-10-30 confidence: high tags: [skills, hermes] sources: []