Stocktake 2026-11-21: re-verify 5 stale + 2 threshold pages against live state; freshness 0 stale; skills-index 299/67

This commit is contained in:
Tony0410
2026-11-21 03:16:30 +08:00
parent 375d52fa26
commit 95e2cd8357
9 changed files with 100 additions and 27 deletions
@@ -3,9 +3,9 @@ title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion
type: incident
status: resolved
created: 2026-10-28
updated: 2026-10-30
verified_on: 2026-10-30
review_after: 2026-11-13
updated: 2026-11-21
verified_on: 2026-11-21
review_after: 2026-12-21
confidence: high
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api]
@@ -121,6 +121,17 @@ handoff). Stage 1 itself no longer delivers — correct, because stage 2 owns th
The questions this page left open are answered for this run: the handoff **was** written inside budget and
consumed, and **no cover letter went missing on the first real run of the split**.
## Re-verification — 2026-11-21 (stocktake)
Live probe: the split still holds and both stages are healthy — `dca8482e4f76` (Discovery) and
`5fc292ff0f3b` (Write and Letters) are both `enabled: true`, `last_status: ok`, `failure_streak: 0`,
pins unchanged (`custom:omni-101` / `smart-route`). Handoff files are being written every weekday —
latest `~/.hermes/radar-ref/handoff/2026-11-20.json` (7.9 KB), with files present for every weekday
back to 2026-11-13. The cap/budget fault has recurred since this incident (see
[[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]],
[[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]); both ledger rows this page opened
remain closed `resolved`.
## Related
- [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun
@@ -3,9 +3,9 @@ title: Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handof
type: incident
status: resolved
created: 2026-10-29
updated: 2026-10-30
verified_on: 2026-10-30
review_after: 2026-11-13
updated: 2026-11-21
verified_on: 2026-11-21
review_after: 2026-12-21
confidence: high
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir]
@@ -91,6 +91,14 @@ the previous two slots in the split both died on it (2026-10-27 on the tool-call
the iteration cap). Treat a `completed` run with **no handoff file** as the same fault, whatever the
execution status says → [[incidents/2026-10-27-job-radar-budget-exhausted]].
## Re-verification — 2026-11-21 (stocktake)
Live probe: `dca8482e4f76` is `enabled: true`, `last_status: ok`, `failure_streak: 0`, and handoff
files exist for every weekday back to 2026-11-13 (latest `2026-11-20.json`). The iteration-cap fault
recurred on 2026-11-02 → [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] and again
on 2026-11-09 → [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]] — the ⚠ "flaky, not
solved" warning above stands. This page's ledger rows remain closed `resolved`.
## Related
- [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up
+12 -2
View File
@@ -3,8 +3,8 @@ title: Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hou
type: incident
status: resolved
created: 2026-10-30
updated: 2026-10-30
verified_on: 2026-10-30
updated: 2026-11-21
verified_on: 2026-11-21
confidence: high
tags: [conduit, voice, tts, stt, config, session-ttl, incident]
sources:
@@ -98,6 +98,16 @@ Consequence: `voice.mode` was moved **`local` → `classic`** (request queued 08
- If an app update or the settings sync ever writes the bare `/v1/` back into `voice.speech_server.url`, every call silently 403s again. Re-check it after any Conduit update.
- Watch for the Conduit settings tool re-flipping `tts.provider` should a `tts.provider` key ever appear in the app's schema.
## Re-verification — 2026-11-21 (stocktake)
All three fixes still in place live in `~/.hermes/config.yaml`: `voice.speech_server.url` =
`wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime` (the `/realtime` suffix has not been
truncated back); `dashboard.basic_auth.session_ttl_seconds` = 2592000 while
`dashboard.session_ttl_seconds` is still 43200 (exactly the two-key warning above);
`tts.provider` = `ollama_local`; `stt.provider` = `local`. Classic mode remains selected in
`~/.hermes/conduit-settings.json`. A follow-on Conduit voice fault is recorded separately →
[[incidents/2026-10-31-conduit-voice-local-mode-not-applying]].
## Related
- [Conduit](../systems/conduit.md) — service page (voice pipeline, call path, pairing)
@@ -3,8 +3,8 @@ title: CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026
type: incident
status: resolved
created: 2026-10-31
updated: 2026-10-31
verified_on: 2026-10-31
updated: 2026-11-21
verified_on: 2026-11-21
confidence: high
tags: [cron, gateway, host, outage, pve, ct460, incident]
sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-agent-log, live-deliveries-db, ssh-pve-uptime]
@@ -16,6 +16,14 @@ sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-a
closed clean (138 executions, 136 `completed`, **0 `failed`**). **Root cause of the host reboot is not
established.** No monitor alerted; the gap was found by this audit.
## Re-verification — 2026-11-21 (stocktake)
CT460's host has rebooted once more since this incident: current `uptime -s` reads
**2026-11-10 22:03:51**, matching the 2026-11-10 unclean gateway exit in `gateway.log`
(previous life `pid=3109529`, started 2026-11-07, SIGKILL/OOM-class exit, `suspected_oom=false`).
This page's 2026-10-30 window stands as recorded; later scheduler gaps were recorded as separate
incidents.
## Summary
The Hermes runtime went dark at **~10:58 AWST on 2026-10-30** and did not return until
@@ -3,9 +3,9 @@ title: m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start
type: concept
status: resolved
created: 2026-10-30
updated: 2026-10-30
verified_on: 2026-10-30
review_after: 2026-11-30
updated: 2026-11-21
verified_on: 2026-11-21
review_after: 2026-12-21
confidence: high
tags: [proxmox, m93p, nfs, lxc, outage, freshrss]
sources: [live-probe, ssh-m93p, systemctl-status, pct-list, freshrss-script-output]
@@ -75,6 +75,14 @@ If a service on m93p is unreachable and `pct list` shows it `stopped`:
3. `pct list` to confirm `running`; then check the service from the Hermes host
4. FreshRSS specifically: `curl -sS -m 15 -o /dev/null -w '%{http_code}' http://100.83.60.93/`
## Re-verification — 2026-11-21 (stocktake)
`ssh root@100.104.51.98` probe: all **three** NFS mounts present (`ssd0-39`, `proxmox-31`,
`proxmox-54`), `m93p-nfs-recover.service` **enabled and active**. The guest fleet has grown since
this incident: **23 CTs, 21 running** — `nightbridge` (998) and `openmausbot` (1264) are stopped
(stop cause for 1264 not re-probed this run; it has a history of tailscale-binary absence).
No repeat of the boot-time mount loss has been recorded since the fix was installed.
## Lessons
- A Google Reader API `Auth failed ... timed out` error is an availability symptom first, an auth
@@ -3,8 +3,8 @@ title: Mnemosyne Health Monitor auto-paused after gateway-connection failure
type: incident
status: resolved
created: 2026-10-30
updated: 2026-10-31
verified_on: 2026-10-31
updated: 2026-11-21
verified_on: 2026-11-21
confidence: high
tags: [cron, mnemosyne, gateway, incident, delivery]
sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents]
@@ -24,6 +24,15 @@ host outage** and fired late in the 16:37:21 catch-up burst, still `running` at
**Historical snapshot (2026-10-30 03:3x): ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
## Re-verification — 2026-11-21 (stocktake)
`ea79a2107e36` is still `enabled: true` and firing every 12 h, but `last_status: error`,
`failure_streak: 4` — the monitor is (correctly) reporting the **Mnemosyne capture stall**:
its latest run found a user message with no `working_memory` row and `last capture
2026-11-18 11:25`. That is a live memory-pipeline fault the monitor is doing its job about, not a
delivery or gateway fault → [[incidents/2026-11-14-mnemosyne-capture-stall]]. The 2026-10-30
resolution above (auto-close, deliveries resumed) stands.
## Summary
The Mnemosyne Health Monitor cron job `ea79a2107e36` (`0 */12 * * *`, delivery topic `102`) failed its
+12
View File
@@ -1948,3 +1948,15 @@ Pages touched: new `incidents/2026-11-20-content-policy-blocked-digest-failures.
`incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md` (115 → 210 slots);
`incidents/2026-11-18-wiki-audit-credit-outage.md` (→ resolved); `systems/mnemosyne.md`;
`systems/scheduled-tasks.md`; `current-state.md`; `index.md`.
## 2026-11-21 03:xx — 03:00 stocktake (wiki-maintain.sh + re-verification)
- `wiki-maintain.sh` ran clean: skills-index regenerated (299 skills / 67 categories), Quartz rebuilt (879 files), deployed, `hermes-wiki-serve` restarted.
- Freshness: 5 stale (22d, all updated 2026-10-30) → all 7 re-verified live this run (5 stale + 2 at exactly 21d, pattern 22):
- `incidents/2026-10-27-job-radar-budget-exhausted.md` — split holds, both stages ok/streak 0, handoffs through 2026-11-20; recurrences cross-ref'd.
- `incidents/2026-10-28-job-radar-stage1-iteration-cap.md` — same probe; ⚠ flaky-not-solved stands.
- `incidents/2026-10-29-conduit-voice-silence.md` — URL/TTL/STT/TTS fixes all confirmed live; classic mode still set.
- `incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md` — 3 mounts present, recover svc enabled+active; fleet now 23 CTs / 21 running.
- `systems/homepage-dashboard.md` — manager/homebox/homarr up; 8082 no listener, 8085→502, 8084→200, 8102→200; status stays degraded.
- `incidents/2026-10-30-ct460-host-outage-scheduler-gap.md` — host rebooted again 2026-11-10 22:03 (matches 11-10 unclean gateway exit); 10-30 record stands.
- `incidents/2026-10-30-mnemosyne-health-monitor-paused.md` — job enabled but `last_status: error`, streak 4 on the live capture-stall fault (last capture 2026-11-18) → 2026-11-14 stall incident.
+10 -10
View File
@@ -1,15 +1,10 @@
# Wiki freshness audit — 2026-11-21
Threshold: >21 days since `updated`
## STALE (5)
## STALE (0)
- [ 22d] incidents/2026-10-27-job-radar-budget-exhausted.md (updated 2026-10-30)
- [ 22d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md (updated 2026-10-30)
- [ 22d] incidents/2026-10-29-conduit-voice-silence.md (updated 2026-10-30)
- [ 22d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md (updated 2026-10-30)
- [ 22d] systems/homepage-dashboard.md (updated 2026-10-30)
## FRESH (132)
## FRESH (137)
- [ 10d] CONTRIBUTING.md
- [ 13d] agents/hermes-chaos.md
@@ -61,8 +56,12 @@ Threshold: >21 days since `updated`
- [ 2d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 3d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 2d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 21d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 21d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 0d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 0d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 0d] incidents/2026-10-29-conduit-voice-silence.md
- [ 0d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 0d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 0d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 20d] incidents/2026-10-31-asana-access-failures.md
- [ 20d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 20d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
@@ -125,6 +124,7 @@ Threshold: >21 days since `updated`
- [ 8d] systems/hermes-hq-push.md
- [ 10d] systems/hermes-ui.md
- [ 17d] systems/home-assistant.md
- [ 0d] systems/homepage-dashboard.md
- [ 17d] systems/ignis.md
- [ 4d] systems/memory-backend.md
- [ 19d] systems/messaging-integrations.md
@@ -150,4 +150,4 @@ Threshold: >21 days since `updated`
- experiments/active/example.md
- log.md
TOTAL: 5 stale, 132 fresh, 3 no-date of 140 pages
TOTAL: 0 stale, 137 fresh, 3 no-date of 140 pages
+9 -2
View File
@@ -3,8 +3,8 @@ title: Homepage Dashboard
type: systems
status: degraded
created: 2026-07-26
updated: 2026-10-30
verified_on: 2026-10-30
updated: 2026-11-21
verified_on: 2026-11-21
confidence: high
tags: [dashboard, homepage, homelab, proxmox, monitoring]
---
@@ -97,6 +97,13 @@ sets `.app-grid` at **line 3640** to `grid-template-columns: repeat(auto-fill, m
`repeat(auto-fill, minmax(160px, 1fr))` (line 5001) and then `1fr 1fr` (line 5010). The `columns:` figures in
`settings.yaml` are dead config.
## Status re-verified 2026-11-21 (stocktake)
CT201 live probe: `homepage-manager` up (restarted ~1 h before the probe), `homebox` and `homarr`
both up 10 days and healthy; **8082 still has no listener**, **8085 still 502** (upstream 8082
dead), **8084 → 200**, **8102 → 200** (307 redirect on `/`, 200 after following). The degraded
status and the architecture claims above are unchanged; no content drift found.
## Proxmox Integration
- **Node widget:** Shows live VM/LXC counts, CPU, memory for PVE node
- **Per-VM stats:** Click "View Proxmox stats" on mapped services