diff --git a/incidents/2026-10-27-job-radar-budget-exhausted.md b/incidents/2026-10-27-job-radar-budget-exhausted.md index e1940d9..319a33f 100644 --- a/incidents/2026-10-27-job-radar-budget-exhausted.md +++ b/incidents/2026-10-27-job-radar-budget-exhausted.md @@ -3,9 +3,9 @@ title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion type: incident status: resolved created: 2026-10-28 -updated: 2026-10-30 -verified_on: 2026-10-30 -review_after: 2026-11-13 +updated: 2026-11-21 +verified_on: 2026-11-21 +review_after: 2026-12-21 confidence: high tags: [automation, cron, jobs, notion, silent-failure, scheduling] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api] @@ -121,6 +121,17 @@ handoff). Stage 1 itself no longer delivers — correct, because stage 2 owns th The questions this page left open are answered for this run: the handoff **was** written inside budget and consumed, and **no cover letter went missing on the first real run of the split**. +## Re-verification — 2026-11-21 (stocktake) + +Live probe: the split still holds and both stages are healthy — `dca8482e4f76` (Discovery) and +`5fc292ff0f3b` (Write and Letters) are both `enabled: true`, `last_status: ok`, `failure_streak: 0`, +pins unchanged (`custom:omni-101` / `smart-route`). Handoff files are being written every weekday — +latest `~/.hermes/radar-ref/handoff/2026-11-20.json` (7.9 KB), with files present for every weekday +back to 2026-11-13. The cap/budget fault has recurred since this incident (see +[[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]], +[[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]); both ledger rows this page opened +remain closed `resolved`. + ## Related - [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun diff --git a/incidents/2026-10-28-job-radar-stage1-iteration-cap.md b/incidents/2026-10-28-job-radar-stage1-iteration-cap.md index f87ff78..9835195 100644 --- a/incidents/2026-10-28-job-radar-stage1-iteration-cap.md +++ b/incidents/2026-10-28-job-radar-stage1-iteration-cap.md @@ -3,9 +3,9 @@ title: Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handof type: incident status: resolved created: 2026-10-29 -updated: 2026-10-30 -verified_on: 2026-10-30 -review_after: 2026-11-13 +updated: 2026-11-21 +verified_on: 2026-11-21 +review_after: 2026-12-21 confidence: high tags: [automation, cron, jobs, notion, silent-failure, scheduling] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir] @@ -91,6 +91,14 @@ the previous two slots in the split both died on it (2026-10-27 on the tool-call the iteration cap). Treat a `completed` run with **no handoff file** as the same fault, whatever the execution status says → [[incidents/2026-10-27-job-radar-budget-exhausted]]. +## Re-verification — 2026-11-21 (stocktake) + +Live probe: `dca8482e4f76` is `enabled: true`, `last_status: ok`, `failure_streak: 0`, and handoff +files exist for every weekday back to 2026-11-13 (latest `2026-11-20.json`). The iteration-cap fault +recurred on 2026-11-02 → [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] and again +on 2026-11-09 → [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]] — the ⚠ "flaky, not +solved" warning above stands. This page's ledger rows remain closed `resolved`. + ## Related - [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up diff --git a/incidents/2026-10-29-conduit-voice-silence.md b/incidents/2026-10-29-conduit-voice-silence.md index 3394e37..ad5efed 100644 --- a/incidents/2026-10-29-conduit-voice-silence.md +++ b/incidents/2026-10-29-conduit-voice-silence.md @@ -3,8 +3,8 @@ title: Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hou type: incident status: resolved created: 2026-10-30 -updated: 2026-10-30 -verified_on: 2026-10-30 +updated: 2026-11-21 +verified_on: 2026-11-21 confidence: high tags: [conduit, voice, tts, stt, config, session-ttl, incident] sources: @@ -98,6 +98,16 @@ Consequence: `voice.mode` was moved **`local` → `classic`** (request queued 08 - If an app update or the settings sync ever writes the bare `/v1/` back into `voice.speech_server.url`, every call silently 403s again. Re-check it after any Conduit update. - Watch for the Conduit settings tool re-flipping `tts.provider` should a `tts.provider` key ever appear in the app's schema. +## Re-verification — 2026-11-21 (stocktake) + +All three fixes still in place live in `~/.hermes/config.yaml`: `voice.speech_server.url` = +`wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime` (the `/realtime` suffix has not been +truncated back); `dashboard.basic_auth.session_ttl_seconds` = 2592000 while +`dashboard.session_ttl_seconds` is still 43200 (exactly the two-key warning above); +`tts.provider` = `ollama_local`; `stt.provider` = `local`. Classic mode remains selected in +`~/.hermes/conduit-settings.json`. A follow-on Conduit voice fault is recorded separately → +[[incidents/2026-10-31-conduit-voice-local-mode-not-applying]]. + ## Related - [Conduit](../systems/conduit.md) — service page (voice pipeline, call path, pairing) diff --git a/incidents/2026-10-30-ct460-host-outage-scheduler-gap.md b/incidents/2026-10-30-ct460-host-outage-scheduler-gap.md index 26f7ba6..9253207 100644 --- a/incidents/2026-10-30-ct460-host-outage-scheduler-gap.md +++ b/incidents/2026-10-30-ct460-host-outage-scheduler-gap.md @@ -3,8 +3,8 @@ title: CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026 type: incident status: resolved created: 2026-10-31 -updated: 2026-10-31 -verified_on: 2026-10-31 +updated: 2026-11-21 +verified_on: 2026-11-21 confidence: high tags: [cron, gateway, host, outage, pve, ct460, incident] sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-agent-log, live-deliveries-db, ssh-pve-uptime] @@ -16,6 +16,14 @@ sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-a closed clean (138 executions, 136 `completed`, **0 `failed`**). **Root cause of the host reboot is not established.** No monitor alerted; the gap was found by this audit. +## Re-verification — 2026-11-21 (stocktake) + +CT460's host has rebooted once more since this incident: current `uptime -s` reads +**2026-11-10 22:03:51**, matching the 2026-11-10 unclean gateway exit in `gateway.log` +(previous life `pid=3109529`, started 2026-11-07, SIGKILL/OOM-class exit, `suspected_oom=false`). +This page's 2026-10-30 window stands as recorded; later scheduler gaps were recorded as separate +incidents. + ## Summary The Hermes runtime went dark at **~10:58 AWST on 2026-10-30** and did not return until diff --git a/incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md b/incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md index 77e2c58..33a6f5a 100644 --- a/incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md +++ b/incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md @@ -3,9 +3,9 @@ title: m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start type: concept status: resolved created: 2026-10-30 -updated: 2026-10-30 -verified_on: 2026-10-30 -review_after: 2026-11-30 +updated: 2026-11-21 +verified_on: 2026-11-21 +review_after: 2026-12-21 confidence: high tags: [proxmox, m93p, nfs, lxc, outage, freshrss] sources: [live-probe, ssh-m93p, systemctl-status, pct-list, freshrss-script-output] @@ -75,6 +75,14 @@ If a service on m93p is unreachable and `pct list` shows it `stopped`: 3. `pct list` to confirm `running`; then check the service from the Hermes host 4. FreshRSS specifically: `curl -sS -m 15 -o /dev/null -w '%{http_code}' http://100.83.60.93/` +## Re-verification — 2026-11-21 (stocktake) + +`ssh root@100.104.51.98` probe: all **three** NFS mounts present (`ssd0-39`, `proxmox-31`, +`proxmox-54`), `m93p-nfs-recover.service` **enabled and active**. The guest fleet has grown since +this incident: **23 CTs, 21 running** — `nightbridge` (998) and `openmausbot` (1264) are stopped +(stop cause for 1264 not re-probed this run; it has a history of tailscale-binary absence). +No repeat of the boot-time mount loss has been recorded since the fix was installed. + ## Lessons - A Google Reader API `Auth failed ... timed out` error is an availability symptom first, an auth diff --git a/incidents/2026-10-30-mnemosyne-health-monitor-paused.md b/incidents/2026-10-30-mnemosyne-health-monitor-paused.md index f1b9b4c..9e39b80 100644 --- a/incidents/2026-10-30-mnemosyne-health-monitor-paused.md +++ b/incidents/2026-10-30-mnemosyne-health-monitor-paused.md @@ -3,8 +3,8 @@ title: Mnemosyne Health Monitor auto-paused after gateway-connection failure type: incident status: resolved created: 2026-10-30 -updated: 2026-10-31 -verified_on: 2026-10-31 +updated: 2026-11-21 +verified_on: 2026-11-21 confidence: high tags: [cron, mnemosyne, gateway, incident, delivery] sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents] @@ -24,6 +24,15 @@ host outage** and fired late in the 16:37:21 catch-up burst, still `running` at **Historical snapshot (2026-10-30 03:3x): ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram. +## Re-verification — 2026-11-21 (stocktake) + +`ea79a2107e36` is still `enabled: true` and firing every 12 h, but `last_status: error`, +`failure_streak: 4` — the monitor is (correctly) reporting the **Mnemosyne capture stall**: +its latest run found a user message with no `working_memory` row and `last capture +2026-11-18 11:25`. That is a live memory-pipeline fault the monitor is doing its job about, not a +delivery or gateway fault → [[incidents/2026-11-14-mnemosyne-capture-stall]]. The 2026-10-30 +resolution above (auto-close, deliveries resumed) stands. + ## Summary The Mnemosyne Health Monitor cron job `ea79a2107e36` (`0 */12 * * *`, delivery topic `102`) failed its diff --git a/log.md b/log.md index b37164f..316d7b7 100644 --- a/log.md +++ b/log.md @@ -1948,3 +1948,15 @@ Pages touched: new `incidents/2026-11-20-content-policy-blocked-digest-failures. `incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md` (115 → 210 slots); `incidents/2026-11-18-wiki-audit-credit-outage.md` (→ resolved); `systems/mnemosyne.md`; `systems/scheduled-tasks.md`; `current-state.md`; `index.md`. + +## 2026-11-21 03:xx — 03:00 stocktake (wiki-maintain.sh + re-verification) + +- `wiki-maintain.sh` ran clean: skills-index regenerated (299 skills / 67 categories), Quartz rebuilt (879 files), deployed, `hermes-wiki-serve` restarted. +- Freshness: 5 stale (22d, all updated 2026-10-30) → all 7 re-verified live this run (5 stale + 2 at exactly 21d, pattern 22): + - `incidents/2026-10-27-job-radar-budget-exhausted.md` — split holds, both stages ok/streak 0, handoffs through 2026-11-20; recurrences cross-ref'd. + - `incidents/2026-10-28-job-radar-stage1-iteration-cap.md` — same probe; ⚠ flaky-not-solved stands. + - `incidents/2026-10-29-conduit-voice-silence.md` — URL/TTL/STT/TTS fixes all confirmed live; classic mode still set. + - `incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md` — 3 mounts present, recover svc enabled+active; fleet now 23 CTs / 21 running. + - `systems/homepage-dashboard.md` — manager/homebox/homarr up; 8082 no listener, 8085→502, 8084→200, 8102→200; status stays degraded. + - `incidents/2026-10-30-ct460-host-outage-scheduler-gap.md` — host rebooted again 2026-11-10 22:03 (matches 11-10 unclean gateway exit); 10-30 record stands. + - `incidents/2026-10-30-mnemosyne-health-monitor-paused.md` — job enabled but `last_status: error`, streak 4 on the live capture-stall fault (last capture 2026-11-18) → 2026-11-14 stall incident. diff --git a/raw/stocktake/freshness-20261121.txt b/raw/stocktake/freshness-20261121.txt index 35bfcfd..ac17978 100644 --- a/raw/stocktake/freshness-20261121.txt +++ b/raw/stocktake/freshness-20261121.txt @@ -1,15 +1,10 @@ # Wiki freshness audit — 2026-11-21 Threshold: >21 days since `updated` -## STALE (5) +## STALE (0) -- [ 22d] incidents/2026-10-27-job-radar-budget-exhausted.md (updated 2026-10-30) -- [ 22d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md (updated 2026-10-30) -- [ 22d] incidents/2026-10-29-conduit-voice-silence.md (updated 2026-10-30) -- [ 22d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md (updated 2026-10-30) -- [ 22d] systems/homepage-dashboard.md (updated 2026-10-30) -## FRESH (132) +## FRESH (137) - [ 10d] CONTRIBUTING.md - [ 13d] agents/hermes-chaos.md @@ -61,8 +56,12 @@ Threshold: >21 days since `updated` - [ 2d] incidents/2026-10-24-conduit-call-request-refused.md - [ 3d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md - [ 2d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md -- [ 21d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md -- [ 21d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md +- [ 0d] incidents/2026-10-27-job-radar-budget-exhausted.md +- [ 0d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md +- [ 0d] incidents/2026-10-29-conduit-voice-silence.md +- [ 0d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md +- [ 0d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md +- [ 0d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md - [ 20d] incidents/2026-10-31-asana-access-failures.md - [ 20d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md - [ 20d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md @@ -125,6 +124,7 @@ Threshold: >21 days since `updated` - [ 8d] systems/hermes-hq-push.md - [ 10d] systems/hermes-ui.md - [ 17d] systems/home-assistant.md +- [ 0d] systems/homepage-dashboard.md - [ 17d] systems/ignis.md - [ 4d] systems/memory-backend.md - [ 19d] systems/messaging-integrations.md @@ -150,4 +150,4 @@ Threshold: >21 days since `updated` - experiments/active/example.md - log.md -TOTAL: 5 stale, 132 fresh, 3 no-date of 140 pages \ No newline at end of file +TOTAL: 0 stale, 137 fresh, 3 no-date of 140 pages \ No newline at end of file diff --git a/systems/homepage-dashboard.md b/systems/homepage-dashboard.md index 0e122d3..d3abcb9 100644 --- a/systems/homepage-dashboard.md +++ b/systems/homepage-dashboard.md @@ -3,8 +3,8 @@ title: Homepage Dashboard type: systems status: degraded created: 2026-07-26 -updated: 2026-10-30 -verified_on: 2026-10-30 +updated: 2026-11-21 +verified_on: 2026-11-21 confidence: high tags: [dashboard, homepage, homelab, proxmox, monitoring] --- @@ -97,6 +97,13 @@ sets `.app-grid` at **line 3640** to `grid-template-columns: repeat(auto-fill, m `repeat(auto-fill, minmax(160px, 1fr))` (line 5001) and then `1fr 1fr` (line 5010). The `columns:` figures in `settings.yaml` are dead config. +## Status re-verified 2026-11-21 (stocktake) + +CT201 live probe: `homepage-manager` up (restarted ~1 h before the probe), `homebox` and `homarr` +both up 10 days and healthy; **8082 still has no listener**, **8085 still 502** (upstream 8082 +dead), **8084 → 200**, **8102 → 200** (307 redirect on `/`, 200 after following). The degraded +status and the architecture claims above are unchanged; no content drift found. + ## Proxmox Integration - **Node widget:** Shows live VM/LXC counts, CPU, memory for PVE node - **Per-VM stats:** Click "View Proxmox stats" on mapped services