Stocktake 2026-11-21: re-verify 5 stale + 2 threshold pages against live state; freshness 0 stale; skills-index 299/67
This commit is contained in:
@@ -3,9 +3,9 @@ title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion
|
|||||||
type: incident
|
type: incident
|
||||||
status: resolved
|
status: resolved
|
||||||
created: 2026-10-28
|
created: 2026-10-28
|
||||||
updated: 2026-10-30
|
updated: 2026-11-21
|
||||||
verified_on: 2026-10-30
|
verified_on: 2026-11-21
|
||||||
review_after: 2026-11-13
|
review_after: 2026-12-21
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
||||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api]
|
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api]
|
||||||
@@ -121,6 +121,17 @@ handoff). Stage 1 itself no longer delivers — correct, because stage 2 owns th
|
|||||||
The questions this page left open are answered for this run: the handoff **was** written inside budget and
|
The questions this page left open are answered for this run: the handoff **was** written inside budget and
|
||||||
consumed, and **no cover letter went missing on the first real run of the split**.
|
consumed, and **no cover letter went missing on the first real run of the split**.
|
||||||
|
|
||||||
|
## Re-verification — 2026-11-21 (stocktake)
|
||||||
|
|
||||||
|
Live probe: the split still holds and both stages are healthy — `dca8482e4f76` (Discovery) and
|
||||||
|
`5fc292ff0f3b` (Write and Letters) are both `enabled: true`, `last_status: ok`, `failure_streak: 0`,
|
||||||
|
pins unchanged (`custom:omni-101` / `smart-route`). Handoff files are being written every weekday —
|
||||||
|
latest `~/.hermes/radar-ref/handoff/2026-11-20.json` (7.9 KB), with files present for every weekday
|
||||||
|
back to 2026-11-13. The cap/budget fault has recurred since this incident (see
|
||||||
|
[[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]],
|
||||||
|
[[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]); both ledger rows this page opened
|
||||||
|
remain closed `resolved`.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun
|
- [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handof
|
|||||||
type: incident
|
type: incident
|
||||||
status: resolved
|
status: resolved
|
||||||
created: 2026-10-29
|
created: 2026-10-29
|
||||||
updated: 2026-10-30
|
updated: 2026-11-21
|
||||||
verified_on: 2026-10-30
|
verified_on: 2026-11-21
|
||||||
review_after: 2026-11-13
|
review_after: 2026-12-21
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
||||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir]
|
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir]
|
||||||
@@ -91,6 +91,14 @@ the previous two slots in the split both died on it (2026-10-27 on the tool-call
|
|||||||
the iteration cap). Treat a `completed` run with **no handoff file** as the same fault, whatever the
|
the iteration cap). Treat a `completed` run with **no handoff file** as the same fault, whatever the
|
||||||
execution status says → [[incidents/2026-10-27-job-radar-budget-exhausted]].
|
execution status says → [[incidents/2026-10-27-job-radar-budget-exhausted]].
|
||||||
|
|
||||||
|
## Re-verification — 2026-11-21 (stocktake)
|
||||||
|
|
||||||
|
Live probe: `dca8482e4f76` is `enabled: true`, `last_status: ok`, `failure_streak: 0`, and handoff
|
||||||
|
files exist for every weekday back to 2026-11-13 (latest `2026-11-20.json`). The iteration-cap fault
|
||||||
|
recurred on 2026-11-02 → [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] and again
|
||||||
|
on 2026-11-09 → [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]] — the ⚠ "flaky, not
|
||||||
|
solved" warning above stands. This page's ledger rows remain closed `resolved`.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up
|
- [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hou
|
|||||||
type: incident
|
type: incident
|
||||||
status: resolved
|
status: resolved
|
||||||
created: 2026-10-30
|
created: 2026-10-30
|
||||||
updated: 2026-10-30
|
updated: 2026-11-21
|
||||||
verified_on: 2026-10-30
|
verified_on: 2026-11-21
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [conduit, voice, tts, stt, config, session-ttl, incident]
|
tags: [conduit, voice, tts, stt, config, session-ttl, incident]
|
||||||
sources:
|
sources:
|
||||||
@@ -98,6 +98,16 @@ Consequence: `voice.mode` was moved **`local` → `classic`** (request queued 08
|
|||||||
- If an app update or the settings sync ever writes the bare `/v1/` back into `voice.speech_server.url`, every call silently 403s again. Re-check it after any Conduit update.
|
- If an app update or the settings sync ever writes the bare `/v1/` back into `voice.speech_server.url`, every call silently 403s again. Re-check it after any Conduit update.
|
||||||
- Watch for the Conduit settings tool re-flipping `tts.provider` should a `tts.provider` key ever appear in the app's schema.
|
- Watch for the Conduit settings tool re-flipping `tts.provider` should a `tts.provider` key ever appear in the app's schema.
|
||||||
|
|
||||||
|
## Re-verification — 2026-11-21 (stocktake)
|
||||||
|
|
||||||
|
All three fixes still in place live in `~/.hermes/config.yaml`: `voice.speech_server.url` =
|
||||||
|
`wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime` (the `/realtime` suffix has not been
|
||||||
|
truncated back); `dashboard.basic_auth.session_ttl_seconds` = 2592000 while
|
||||||
|
`dashboard.session_ttl_seconds` is still 43200 (exactly the two-key warning above);
|
||||||
|
`tts.provider` = `ollama_local`; `stt.provider` = `local`. Classic mode remains selected in
|
||||||
|
`~/.hermes/conduit-settings.json`. A follow-on Conduit voice fault is recorded separately →
|
||||||
|
[[incidents/2026-10-31-conduit-voice-local-mode-not-applying]].
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [Conduit](../systems/conduit.md) — service page (voice pipeline, call path, pairing)
|
- [Conduit](../systems/conduit.md) — service page (voice pipeline, call path, pairing)
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026
|
|||||||
type: incident
|
type: incident
|
||||||
status: resolved
|
status: resolved
|
||||||
created: 2026-10-31
|
created: 2026-10-31
|
||||||
updated: 2026-10-31
|
updated: 2026-11-21
|
||||||
verified_on: 2026-10-31
|
verified_on: 2026-11-21
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, gateway, host, outage, pve, ct460, incident]
|
tags: [cron, gateway, host, outage, pve, ct460, incident]
|
||||||
sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-agent-log, live-deliveries-db, ssh-pve-uptime]
|
sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-agent-log, live-deliveries-db, ssh-pve-uptime]
|
||||||
@@ -16,6 +16,14 @@ sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-a
|
|||||||
closed clean (138 executions, 136 `completed`, **0 `failed`**). **Root cause of the host reboot is not
|
closed clean (138 executions, 136 `completed`, **0 `failed`**). **Root cause of the host reboot is not
|
||||||
established.** No monitor alerted; the gap was found by this audit.
|
established.** No monitor alerted; the gap was found by this audit.
|
||||||
|
|
||||||
|
## Re-verification — 2026-11-21 (stocktake)
|
||||||
|
|
||||||
|
CT460's host has rebooted once more since this incident: current `uptime -s` reads
|
||||||
|
**2026-11-10 22:03:51**, matching the 2026-11-10 unclean gateway exit in `gateway.log`
|
||||||
|
(previous life `pid=3109529`, started 2026-11-07, SIGKILL/OOM-class exit, `suspected_oom=false`).
|
||||||
|
This page's 2026-10-30 window stands as recorded; later scheduler gaps were recorded as separate
|
||||||
|
incidents.
|
||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
The Hermes runtime went dark at **~10:58 AWST on 2026-10-30** and did not return until
|
The Hermes runtime went dark at **~10:58 AWST on 2026-10-30** and did not return until
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start
|
|||||||
type: concept
|
type: concept
|
||||||
status: resolved
|
status: resolved
|
||||||
created: 2026-10-30
|
created: 2026-10-30
|
||||||
updated: 2026-10-30
|
updated: 2026-11-21
|
||||||
verified_on: 2026-10-30
|
verified_on: 2026-11-21
|
||||||
review_after: 2026-11-30
|
review_after: 2026-12-21
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [proxmox, m93p, nfs, lxc, outage, freshrss]
|
tags: [proxmox, m93p, nfs, lxc, outage, freshrss]
|
||||||
sources: [live-probe, ssh-m93p, systemctl-status, pct-list, freshrss-script-output]
|
sources: [live-probe, ssh-m93p, systemctl-status, pct-list, freshrss-script-output]
|
||||||
@@ -75,6 +75,14 @@ If a service on m93p is unreachable and `pct list` shows it `stopped`:
|
|||||||
3. `pct list` to confirm `running`; then check the service from the Hermes host
|
3. `pct list` to confirm `running`; then check the service from the Hermes host
|
||||||
4. FreshRSS specifically: `curl -sS -m 15 -o /dev/null -w '%{http_code}' http://100.83.60.93/`
|
4. FreshRSS specifically: `curl -sS -m 15 -o /dev/null -w '%{http_code}' http://100.83.60.93/`
|
||||||
|
|
||||||
|
## Re-verification — 2026-11-21 (stocktake)
|
||||||
|
|
||||||
|
`ssh root@100.104.51.98` probe: all **three** NFS mounts present (`ssd0-39`, `proxmox-31`,
|
||||||
|
`proxmox-54`), `m93p-nfs-recover.service` **enabled and active**. The guest fleet has grown since
|
||||||
|
this incident: **23 CTs, 21 running** — `nightbridge` (998) and `openmausbot` (1264) are stopped
|
||||||
|
(stop cause for 1264 not re-probed this run; it has a history of tailscale-binary absence).
|
||||||
|
No repeat of the boot-time mount loss has been recorded since the fix was installed.
|
||||||
|
|
||||||
## Lessons
|
## Lessons
|
||||||
|
|
||||||
- A Google Reader API `Auth failed ... timed out` error is an availability symptom first, an auth
|
- A Google Reader API `Auth failed ... timed out` error is an availability symptom first, an auth
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Mnemosyne Health Monitor auto-paused after gateway-connection failure
|
|||||||
type: incident
|
type: incident
|
||||||
status: resolved
|
status: resolved
|
||||||
created: 2026-10-30
|
created: 2026-10-30
|
||||||
updated: 2026-10-31
|
updated: 2026-11-21
|
||||||
verified_on: 2026-10-31
|
verified_on: 2026-11-21
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, mnemosyne, gateway, incident, delivery]
|
tags: [cron, mnemosyne, gateway, incident, delivery]
|
||||||
sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents]
|
sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents]
|
||||||
@@ -24,6 +24,15 @@ host outage** and fired late in the 16:37:21 catch-up burst, still `running` at
|
|||||||
|
|
||||||
**Historical snapshot (2026-10-30 03:3x): ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
|
**Historical snapshot (2026-10-30 03:3x): ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
|
||||||
|
|
||||||
|
## Re-verification — 2026-11-21 (stocktake)
|
||||||
|
|
||||||
|
`ea79a2107e36` is still `enabled: true` and firing every 12 h, but `last_status: error`,
|
||||||
|
`failure_streak: 4` — the monitor is (correctly) reporting the **Mnemosyne capture stall**:
|
||||||
|
its latest run found a user message with no `working_memory` row and `last capture
|
||||||
|
2026-11-18 11:25`. That is a live memory-pipeline fault the monitor is doing its job about, not a
|
||||||
|
delivery or gateway fault → [[incidents/2026-11-14-mnemosyne-capture-stall]]. The 2026-10-30
|
||||||
|
resolution above (auto-close, deliveries resumed) stands.
|
||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
The Mnemosyne Health Monitor cron job `ea79a2107e36` (`0 */12 * * *`, delivery topic `102`) failed its
|
The Mnemosyne Health Monitor cron job `ea79a2107e36` (`0 */12 * * *`, delivery topic `102`) failed its
|
||||||
|
|||||||
@@ -1948,3 +1948,15 @@ Pages touched: new `incidents/2026-11-20-content-policy-blocked-digest-failures.
|
|||||||
`incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md` (115 → 210 slots);
|
`incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md` (115 → 210 slots);
|
||||||
`incidents/2026-11-18-wiki-audit-credit-outage.md` (→ resolved); `systems/mnemosyne.md`;
|
`incidents/2026-11-18-wiki-audit-credit-outage.md` (→ resolved); `systems/mnemosyne.md`;
|
||||||
`systems/scheduled-tasks.md`; `current-state.md`; `index.md`.
|
`systems/scheduled-tasks.md`; `current-state.md`; `index.md`.
|
||||||
|
|
||||||
|
## 2026-11-21 03:xx — 03:00 stocktake (wiki-maintain.sh + re-verification)
|
||||||
|
|
||||||
|
- `wiki-maintain.sh` ran clean: skills-index regenerated (299 skills / 67 categories), Quartz rebuilt (879 files), deployed, `hermes-wiki-serve` restarted.
|
||||||
|
- Freshness: 5 stale (22d, all updated 2026-10-30) → all 7 re-verified live this run (5 stale + 2 at exactly 21d, pattern 22):
|
||||||
|
- `incidents/2026-10-27-job-radar-budget-exhausted.md` — split holds, both stages ok/streak 0, handoffs through 2026-11-20; recurrences cross-ref'd.
|
||||||
|
- `incidents/2026-10-28-job-radar-stage1-iteration-cap.md` — same probe; ⚠ flaky-not-solved stands.
|
||||||
|
- `incidents/2026-10-29-conduit-voice-silence.md` — URL/TTL/STT/TTS fixes all confirmed live; classic mode still set.
|
||||||
|
- `incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md` — 3 mounts present, recover svc enabled+active; fleet now 23 CTs / 21 running.
|
||||||
|
- `systems/homepage-dashboard.md` — manager/homebox/homarr up; 8082 no listener, 8085→502, 8084→200, 8102→200; status stays degraded.
|
||||||
|
- `incidents/2026-10-30-ct460-host-outage-scheduler-gap.md` — host rebooted again 2026-11-10 22:03 (matches 11-10 unclean gateway exit); 10-30 record stands.
|
||||||
|
- `incidents/2026-10-30-mnemosyne-health-monitor-paused.md` — job enabled but `last_status: error`, streak 4 on the live capture-stall fault (last capture 2026-11-18) → 2026-11-14 stall incident.
|
||||||
|
|||||||
@@ -1,15 +1,10 @@
|
|||||||
# Wiki freshness audit — 2026-11-21
|
# Wiki freshness audit — 2026-11-21
|
||||||
Threshold: >21 days since `updated`
|
Threshold: >21 days since `updated`
|
||||||
|
|
||||||
## STALE (5)
|
## STALE (0)
|
||||||
|
|
||||||
- [ 22d] incidents/2026-10-27-job-radar-budget-exhausted.md (updated 2026-10-30)
|
|
||||||
- [ 22d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md (updated 2026-10-30)
|
|
||||||
- [ 22d] incidents/2026-10-29-conduit-voice-silence.md (updated 2026-10-30)
|
|
||||||
- [ 22d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md (updated 2026-10-30)
|
|
||||||
- [ 22d] systems/homepage-dashboard.md (updated 2026-10-30)
|
|
||||||
|
|
||||||
## FRESH (132)
|
## FRESH (137)
|
||||||
|
|
||||||
- [ 10d] CONTRIBUTING.md
|
- [ 10d] CONTRIBUTING.md
|
||||||
- [ 13d] agents/hermes-chaos.md
|
- [ 13d] agents/hermes-chaos.md
|
||||||
@@ -61,8 +56,12 @@ Threshold: >21 days since `updated`
|
|||||||
- [ 2d] incidents/2026-10-24-conduit-call-request-refused.md
|
- [ 2d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||||
- [ 3d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
- [ 3d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||||
- [ 2d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
- [ 2d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||||
- [ 21d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
- [ 0d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||||
- [ 21d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
- [ 0d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||||
|
- [ 0d] incidents/2026-10-29-conduit-voice-silence.md
|
||||||
|
- [ 0d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||||
|
- [ 0d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||||
|
- [ 0d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||||
- [ 20d] incidents/2026-10-31-asana-access-failures.md
|
- [ 20d] incidents/2026-10-31-asana-access-failures.md
|
||||||
- [ 20d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
- [ 20d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||||
- [ 20d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
- [ 20d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||||
@@ -125,6 +124,7 @@ Threshold: >21 days since `updated`
|
|||||||
- [ 8d] systems/hermes-hq-push.md
|
- [ 8d] systems/hermes-hq-push.md
|
||||||
- [ 10d] systems/hermes-ui.md
|
- [ 10d] systems/hermes-ui.md
|
||||||
- [ 17d] systems/home-assistant.md
|
- [ 17d] systems/home-assistant.md
|
||||||
|
- [ 0d] systems/homepage-dashboard.md
|
||||||
- [ 17d] systems/ignis.md
|
- [ 17d] systems/ignis.md
|
||||||
- [ 4d] systems/memory-backend.md
|
- [ 4d] systems/memory-backend.md
|
||||||
- [ 19d] systems/messaging-integrations.md
|
- [ 19d] systems/messaging-integrations.md
|
||||||
@@ -150,4 +150,4 @@ Threshold: >21 days since `updated`
|
|||||||
- experiments/active/example.md
|
- experiments/active/example.md
|
||||||
- log.md
|
- log.md
|
||||||
|
|
||||||
TOTAL: 5 stale, 132 fresh, 3 no-date of 140 pages
|
TOTAL: 0 stale, 137 fresh, 3 no-date of 140 pages
|
||||||
@@ -3,8 +3,8 @@ title: Homepage Dashboard
|
|||||||
type: systems
|
type: systems
|
||||||
status: degraded
|
status: degraded
|
||||||
created: 2026-07-26
|
created: 2026-07-26
|
||||||
updated: 2026-10-30
|
updated: 2026-11-21
|
||||||
verified_on: 2026-10-30
|
verified_on: 2026-11-21
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [dashboard, homepage, homelab, proxmox, monitoring]
|
tags: [dashboard, homepage, homelab, proxmox, monitoring]
|
||||||
---
|
---
|
||||||
@@ -97,6 +97,13 @@ sets `.app-grid` at **line 3640** to `grid-template-columns: repeat(auto-fill, m
|
|||||||
`repeat(auto-fill, minmax(160px, 1fr))` (line 5001) and then `1fr 1fr` (line 5010). The `columns:` figures in
|
`repeat(auto-fill, minmax(160px, 1fr))` (line 5001) and then `1fr 1fr` (line 5010). The `columns:` figures in
|
||||||
`settings.yaml` are dead config.
|
`settings.yaml` are dead config.
|
||||||
|
|
||||||
|
## Status re-verified 2026-11-21 (stocktake)
|
||||||
|
|
||||||
|
CT201 live probe: `homepage-manager` up (restarted ~1 h before the probe), `homebox` and `homarr`
|
||||||
|
both up 10 days and healthy; **8082 still has no listener**, **8085 still 502** (upstream 8082
|
||||||
|
dead), **8084 → 200**, **8102 → 200** (307 redirect on `/`, 200 after following). The degraded
|
||||||
|
status and the architecture claims above are unchanged; no content drift found.
|
||||||
|
|
||||||
## Proxmox Integration
|
## Proxmox Integration
|
||||||
- **Node widget:** Shows live VM/LXC counts, CPU, memory for PVE node
|
- **Node widget:** Shows live VM/LXC counts, CPU, memory for PVE node
|
||||||
- **Per-VM stats:** Click "View Proxmox stats" on mapped services
|
- **Per-VM stats:** Click "View Proxmox stats" on mapped services
|
||||||
|
|||||||
Reference in New Issue
Block a user