Stocktake 2026-11-21: re-verify 5 stale + 2 threshold pages against live state; freshness 0 stale; skills-index 299/67
This commit is contained in:
@@ -3,9 +3,9 @@ title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-28
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
review_after: 2026-11-13
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
review_after: 2026-12-21
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api]
|
||||
@@ -121,6 +121,17 @@ handoff). Stage 1 itself no longer delivers — correct, because stage 2 owns th
|
||||
The questions this page left open are answered for this run: the handoff **was** written inside budget and
|
||||
consumed, and **no cover letter went missing on the first real run of the split**.
|
||||
|
||||
## Re-verification — 2026-11-21 (stocktake)
|
||||
|
||||
Live probe: the split still holds and both stages are healthy — `dca8482e4f76` (Discovery) and
|
||||
`5fc292ff0f3b` (Write and Letters) are both `enabled: true`, `last_status: ok`, `failure_streak: 0`,
|
||||
pins unchanged (`custom:omni-101` / `smart-route`). Handoff files are being written every weekday —
|
||||
latest `~/.hermes/radar-ref/handoff/2026-11-20.json` (7.9 KB), with files present for every weekday
|
||||
back to 2026-11-13. The cap/budget fault has recurred since this incident (see
|
||||
[[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]],
|
||||
[[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]); both ledger rows this page opened
|
||||
remain closed `resolved`.
|
||||
|
||||
## Related
|
||||
|
||||
- [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handof
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-29
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
review_after: 2026-11-13
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
review_after: 2026-12-21
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir]
|
||||
@@ -91,6 +91,14 @@ the previous two slots in the split both died on it (2026-10-27 on the tool-call
|
||||
the iteration cap). Treat a `completed` run with **no handoff file** as the same fault, whatever the
|
||||
execution status says → [[incidents/2026-10-27-job-radar-budget-exhausted]].
|
||||
|
||||
## Re-verification — 2026-11-21 (stocktake)
|
||||
|
||||
Live probe: `dca8482e4f76` is `enabled: true`, `last_status: ok`, `failure_streak: 0`, and handoff
|
||||
files exist for every weekday back to 2026-11-13 (latest `2026-11-20.json`). The iteration-cap fault
|
||||
recurred on 2026-11-02 → [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] and again
|
||||
on 2026-11-09 → [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]] — the ⚠ "flaky, not
|
||||
solved" warning above stands. This page's ledger rows remain closed `resolved`.
|
||||
|
||||
## Related
|
||||
|
||||
- [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hou
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-30
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
confidence: high
|
||||
tags: [conduit, voice, tts, stt, config, session-ttl, incident]
|
||||
sources:
|
||||
@@ -98,6 +98,16 @@ Consequence: `voice.mode` was moved **`local` → `classic`** (request queued 08
|
||||
- If an app update or the settings sync ever writes the bare `/v1/` back into `voice.speech_server.url`, every call silently 403s again. Re-check it after any Conduit update.
|
||||
- Watch for the Conduit settings tool re-flipping `tts.provider` should a `tts.provider` key ever appear in the app's schema.
|
||||
|
||||
## Re-verification — 2026-11-21 (stocktake)
|
||||
|
||||
All three fixes still in place live in `~/.hermes/config.yaml`: `voice.speech_server.url` =
|
||||
`wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime` (the `/realtime` suffix has not been
|
||||
truncated back); `dashboard.basic_auth.session_ttl_seconds` = 2592000 while
|
||||
`dashboard.session_ttl_seconds` is still 43200 (exactly the two-key warning above);
|
||||
`tts.provider` = `ollama_local`; `stt.provider` = `local`. Classic mode remains selected in
|
||||
`~/.hermes/conduit-settings.json`. A follow-on Conduit voice fault is recorded separately →
|
||||
[[incidents/2026-10-31-conduit-voice-local-mode-not-applying]].
|
||||
|
||||
## Related
|
||||
|
||||
- [Conduit](../systems/conduit.md) — service page (voice pipeline, call path, pairing)
|
||||
|
||||
@@ -3,8 +3,8 @@ title: CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-31
|
||||
updated: 2026-10-31
|
||||
verified_on: 2026-10-31
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
confidence: high
|
||||
tags: [cron, gateway, host, outage, pve, ct460, incident]
|
||||
sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-agent-log, live-deliveries-db, ssh-pve-uptime]
|
||||
@@ -16,6 +16,14 @@ sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-a
|
||||
closed clean (138 executions, 136 `completed`, **0 `failed`**). **Root cause of the host reboot is not
|
||||
established.** No monitor alerted; the gap was found by this audit.
|
||||
|
||||
## Re-verification — 2026-11-21 (stocktake)
|
||||
|
||||
CT460's host has rebooted once more since this incident: current `uptime -s` reads
|
||||
**2026-11-10 22:03:51**, matching the 2026-11-10 unclean gateway exit in `gateway.log`
|
||||
(previous life `pid=3109529`, started 2026-11-07, SIGKILL/OOM-class exit, `suspected_oom=false`).
|
||||
This page's 2026-10-30 window stands as recorded; later scheduler gaps were recorded as separate
|
||||
incidents.
|
||||
|
||||
## Summary
|
||||
|
||||
The Hermes runtime went dark at **~10:58 AWST on 2026-10-30** and did not return until
|
||||
|
||||
@@ -3,9 +3,9 @@ title: m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start
|
||||
type: concept
|
||||
status: resolved
|
||||
created: 2026-10-30
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
review_after: 2026-11-30
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
review_after: 2026-12-21
|
||||
confidence: high
|
||||
tags: [proxmox, m93p, nfs, lxc, outage, freshrss]
|
||||
sources: [live-probe, ssh-m93p, systemctl-status, pct-list, freshrss-script-output]
|
||||
@@ -75,6 +75,14 @@ If a service on m93p is unreachable and `pct list` shows it `stopped`:
|
||||
3. `pct list` to confirm `running`; then check the service from the Hermes host
|
||||
4. FreshRSS specifically: `curl -sS -m 15 -o /dev/null -w '%{http_code}' http://100.83.60.93/`
|
||||
|
||||
## Re-verification — 2026-11-21 (stocktake)
|
||||
|
||||
`ssh root@100.104.51.98` probe: all **three** NFS mounts present (`ssd0-39`, `proxmox-31`,
|
||||
`proxmox-54`), `m93p-nfs-recover.service` **enabled and active**. The guest fleet has grown since
|
||||
this incident: **23 CTs, 21 running** — `nightbridge` (998) and `openmausbot` (1264) are stopped
|
||||
(stop cause for 1264 not re-probed this run; it has a history of tailscale-binary absence).
|
||||
No repeat of the boot-time mount loss has been recorded since the fix was installed.
|
||||
|
||||
## Lessons
|
||||
|
||||
- A Google Reader API `Auth failed ... timed out` error is an availability symptom first, an auth
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Mnemosyne Health Monitor auto-paused after gateway-connection failure
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-30
|
||||
updated: 2026-10-31
|
||||
verified_on: 2026-10-31
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
confidence: high
|
||||
tags: [cron, mnemosyne, gateway, incident, delivery]
|
||||
sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents]
|
||||
@@ -24,6 +24,15 @@ host outage** and fired late in the 16:37:21 catch-up burst, still `running` at
|
||||
|
||||
**Historical snapshot (2026-10-30 03:3x): ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
|
||||
|
||||
## Re-verification — 2026-11-21 (stocktake)
|
||||
|
||||
`ea79a2107e36` is still `enabled: true` and firing every 12 h, but `last_status: error`,
|
||||
`failure_streak: 4` — the monitor is (correctly) reporting the **Mnemosyne capture stall**:
|
||||
its latest run found a user message with no `working_memory` row and `last capture
|
||||
2026-11-18 11:25`. That is a live memory-pipeline fault the monitor is doing its job about, not a
|
||||
delivery or gateway fault → [[incidents/2026-11-14-mnemosyne-capture-stall]]. The 2026-10-30
|
||||
resolution above (auto-close, deliveries resumed) stands.
|
||||
|
||||
## Summary
|
||||
|
||||
The Mnemosyne Health Monitor cron job `ea79a2107e36` (`0 */12 * * *`, delivery topic `102`) failed its
|
||||
|
||||
@@ -1948,3 +1948,15 @@ Pages touched: new `incidents/2026-11-20-content-policy-blocked-digest-failures.
|
||||
`incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md` (115 → 210 slots);
|
||||
`incidents/2026-11-18-wiki-audit-credit-outage.md` (→ resolved); `systems/mnemosyne.md`;
|
||||
`systems/scheduled-tasks.md`; `current-state.md`; `index.md`.
|
||||
|
||||
## 2026-11-21 03:xx — 03:00 stocktake (wiki-maintain.sh + re-verification)
|
||||
|
||||
- `wiki-maintain.sh` ran clean: skills-index regenerated (299 skills / 67 categories), Quartz rebuilt (879 files), deployed, `hermes-wiki-serve` restarted.
|
||||
- Freshness: 5 stale (22d, all updated 2026-10-30) → all 7 re-verified live this run (5 stale + 2 at exactly 21d, pattern 22):
|
||||
- `incidents/2026-10-27-job-radar-budget-exhausted.md` — split holds, both stages ok/streak 0, handoffs through 2026-11-20; recurrences cross-ref'd.
|
||||
- `incidents/2026-10-28-job-radar-stage1-iteration-cap.md` — same probe; ⚠ flaky-not-solved stands.
|
||||
- `incidents/2026-10-29-conduit-voice-silence.md` — URL/TTL/STT/TTS fixes all confirmed live; classic mode still set.
|
||||
- `incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md` — 3 mounts present, recover svc enabled+active; fleet now 23 CTs / 21 running.
|
||||
- `systems/homepage-dashboard.md` — manager/homebox/homarr up; 8082 no listener, 8085→502, 8084→200, 8102→200; status stays degraded.
|
||||
- `incidents/2026-10-30-ct460-host-outage-scheduler-gap.md` — host rebooted again 2026-11-10 22:03 (matches 11-10 unclean gateway exit); 10-30 record stands.
|
||||
- `incidents/2026-10-30-mnemosyne-health-monitor-paused.md` — job enabled but `last_status: error`, streak 4 on the live capture-stall fault (last capture 2026-11-18) → 2026-11-14 stall incident.
|
||||
|
||||
@@ -1,15 +1,10 @@
|
||||
# Wiki freshness audit — 2026-11-21
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (5)
|
||||
## STALE (0)
|
||||
|
||||
- [ 22d] incidents/2026-10-27-job-radar-budget-exhausted.md (updated 2026-10-30)
|
||||
- [ 22d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md (updated 2026-10-30)
|
||||
- [ 22d] incidents/2026-10-29-conduit-voice-silence.md (updated 2026-10-30)
|
||||
- [ 22d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md (updated 2026-10-30)
|
||||
- [ 22d] systems/homepage-dashboard.md (updated 2026-10-30)
|
||||
|
||||
## FRESH (132)
|
||||
## FRESH (137)
|
||||
|
||||
- [ 10d] CONTRIBUTING.md
|
||||
- [ 13d] agents/hermes-chaos.md
|
||||
@@ -61,8 +56,12 @@ Threshold: >21 days since `updated`
|
||||
- [ 2d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 3d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 2d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 21d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 21d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 0d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 0d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 0d] incidents/2026-10-29-conduit-voice-silence.md
|
||||
- [ 0d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 0d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||
- [ 0d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 20d] incidents/2026-10-31-asana-access-failures.md
|
||||
- [ 20d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||
- [ 20d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||
@@ -125,6 +124,7 @@ Threshold: >21 days since `updated`
|
||||
- [ 8d] systems/hermes-hq-push.md
|
||||
- [ 10d] systems/hermes-ui.md
|
||||
- [ 17d] systems/home-assistant.md
|
||||
- [ 0d] systems/homepage-dashboard.md
|
||||
- [ 17d] systems/ignis.md
|
||||
- [ 4d] systems/memory-backend.md
|
||||
- [ 19d] systems/messaging-integrations.md
|
||||
@@ -150,4 +150,4 @@ Threshold: >21 days since `updated`
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 5 stale, 132 fresh, 3 no-date of 140 pages
|
||||
TOTAL: 0 stale, 137 fresh, 3 no-date of 140 pages
|
||||
@@ -3,8 +3,8 @@ title: Homepage Dashboard
|
||||
type: systems
|
||||
status: degraded
|
||||
created: 2026-07-26
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
confidence: high
|
||||
tags: [dashboard, homepage, homelab, proxmox, monitoring]
|
||||
---
|
||||
@@ -97,6 +97,13 @@ sets `.app-grid` at **line 3640** to `grid-template-columns: repeat(auto-fill, m
|
||||
`repeat(auto-fill, minmax(160px, 1fr))` (line 5001) and then `1fr 1fr` (line 5010). The `columns:` figures in
|
||||
`settings.yaml` are dead config.
|
||||
|
||||
## Status re-verified 2026-11-21 (stocktake)
|
||||
|
||||
CT201 live probe: `homepage-manager` up (restarted ~1 h before the probe), `homebox` and `homarr`
|
||||
both up 10 days and healthy; **8082 still has no listener**, **8085 still 502** (upstream 8082
|
||||
dead), **8084 → 200**, **8102 → 200** (307 redirect on `/`, 200 after following). The degraded
|
||||
status and the architecture claims above are unchanged; no content drift found.
|
||||
|
||||
## Proxmox Integration
|
||||
- **Node widget:** Shows live VM/LXC counts, CPU, memory for PVE node
|
||||
- **Per-VM stats:** Click "View Proxmox stats" on mapped services
|
||||
|
||||
Reference in New Issue
Block a user