Compare commits

...
62 Commits
Author SHA1 Message Date
Tony0410 f603c29d17 stocktake 2026-11-20: re-verify 9-page 2026-10-29 stale cohort live; correct nanobot docker/cron drift, camofox host/port claims, dead-pin residual; freshness 0 stale/136 fresh 2026-11-20 03:11:58 +08:00
Tony0410 8bf0a37022 wiki audit 2026-11-20 (publish): skills-index regenerated (299 skills/67 categories), freshness report 20261120 (9 stale/127 fresh/3 no-date), log entry extended with freshness cohort + publish verification 2026-11-20 01:13:20 +08:00
Tony0410 226767f3ec wiki audit 2026-11-20: three-window catch-up (11-17/11-18/11-19, both prior audit slots died)
- new: incidents/2026-11-18-wiki-audit-credit-outage (audit failed 11-18 + 11-19, deliveries suppressed, incident row alerted_at=None)
- new: incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm (115 consecutive slots, 59 unresolved conflicts vs threshold 50)
- corrected: incidents/2026-11-14-mnemosyne-capture-stall + systems/mnemosyne — freeze ENDED 2026-11-18 19:25 AWST (105h52m, no intervention); 'capture dead' reading was conversational silence (every interactive session captured)
- systems/scheduled-tasks: new Count bullet 11-17 173(171/2) / 11-18 172(153/19) / 11-19 176(78/98); fleet 29 (26/3); non-clean last_status 1->4; unclosed incidents 5->4; deliveries 24 (23 delivered / 1 failed)
- index.md feed + incidents + systems lines; log.md dated entry; current-state.md new section; frontmatter dates -> 2026-11-20
2026-11-20 01:02:44 +08:00
Tony0410 16f2c878c0 wiki stocktake 2026-11-19: 3 stale pages re-verified (conduit-call open item re-probed, 10-26 orphaned-run incident resolved, WA communities report live-checked); freshness report; skills-index dates 2026-11-19 03:22:49 +08:00
Tony0410 98610466cc 2026-11-18 stocktake: re-verify 7 stale pages against live state, resolve gmail-digest refire incident 2026-11-18 03:19:54 +08:00
Tony0410 4574c4ebc3 wiki-maintain 2026-11-17: regenerate skills-index (299 skills/67 cats), freshness report (7 stale); verified live & updated current-profile (ops profile deleted, config v52, 6 MCP servers, 15 plugins, 21/26 pins), memory-backend (2901 memories, coverage working/episodic 100%), model-providers (pin split, pulse_memory aux slot on dead nous credential) 2026-11-17 03:11:54 +08:00
hermes e5612c8a8d audit 2026-11-16: memory stall 62h54m (counts unchanged, cron_incidents 3->5 unclosed, consolidator zero-item streak 120); NEW incident — 11-15 plugin load-timeout storm (52 failed loads, 519 dropped registrations incl. conduit_push 474, 11 pre_tool_call + 2 pre_llm_call hook timeouts, 10 refusals) never recorded by the 11-15 audit; hook page Recurrence 2 (load average removed as predictor); fleet 29 jobs, 11-16 ledger 175 rows 173/2, deliveries 11/11 2026-11-17 00:36:27 +08:00
Tony0410 4eee02b9a2 wiki-maintain 2026-11-16: regenerate skills-index (299 skills/67 cats), freshness report (1 stale: communities-wa audit, review_after 2026-11-26) 2026-11-16 03:03:34 +08:00
Tony0410 9f80926dda audit 2026-11-15: capture stall 38h46m (freeze total, counts unchanged); cron_incidents 1->3 unclosed (signature embeds elapsed-hours float); monitor's 4.64h decoded as newest-conversational-message age; watchdog lost an 11:45 slot to a 600s deps-guard timeout; ledger 168 rows 165/3 (11-14 settled 167/0) 2026-11-16 00:32:12 +08:00
Tony0410 402d8546e8 stocktake(2026-11-15): skills-index regenerated (299/67); storm incident re-verified (all 12 rows resolved, evidence-window note); conduit memory-timeout fix LOST in plugin 0.168.x restructure — page → degraded with regression section; 0 stale after fixes 2026-11-15 03:25:18 +08:00
Tony0410 646be98c31 wiki: correct the 11-14 log entry in place (withdraw the '12 h lead time' + '18:03 run' clauses) 2026-11-15 01:38:02 +08:00
Tony0410 82876abdfb wiki: 2026-11-14 audit corrections — withdraw the 'sign not magnitude' / '12 h of lead time' / '18:03 run' claims; verified escalation mechanism (activity-aware magnitude gate) + 29-cycle zero-item consolidation streak 2026-11-15 01:28:53 +08:00
Tony0410 2d439145bf audit(2026-11-14) follow-up: vault mirror sync into both conventions (14 files, sha256 verified; vault b7d7f18), first drift measurement (top-level: 68 match / 12 older / 0 orphan / 52 missing; wiki/: 68 / 56 / 1 / 7) and the vault-side Conduit incident orphan flagged for an owner decision 2026-11-15 00:47:58 +08:00
Tony0410 4a6e86948d wiki: 2026-11-14 nightly audit + corrections — Phonon TTS on CT200, Mnemosyne capture stall open
- new: systems/phonon-tts.md, incidents/2026-11-14-mnemosyne-capture-stall.md
- CT200 inventory re-verified: 64 non-seeder running / 72 total (delta = the TTS trio, no removals)
- fleet ledger: 11-14 = 167 rows / 166 completed / 0 failed; first red last_status since 11-13
  (Mnemosyne Health Monitor, error, streak 1); cron_incidents 0 -> 1 unclosed (138 rows)
- corrections pass: withdrew the 'unidentified extra container' claim; closed the alert-delivery
  question (delivered 00:03:39, for_failure=1)
- updated: index.md, current-state.md, log.md, systems/scheduled-tasks.md, systems/mnemosyne.md,
  infrastructure/docker-services.md
2026-11-15 00:42:01 +08:00
Tony0410 16767ef562 audit(2026-11-13) follow-up: regenerate systems/skills-index.md (300 skills / 67 categories) and the 2026-11-14 freshness report (0 stale / 131 fresh / 3 no-date of 134) 2026-11-14 00:29:06 +08:00
Tony0410 b0e4a54f8a audit(2026-11-13): third zero-failed day and a fully clean fleet; the Uber Eats fix clears the last non-clean cron row — and the pre_tool_call hook timeout recurred at load 12.08 (incident reopened resolved -> active, state.db 2.04 GB) 2026-11-14 00:27:05 +08:00
Tony0410 a0ec3a1ec5 reports: document the 2026-11-13 digest-script fix and re-verification 2026-11-13 19:39:50 +08:00
Tony0410 bc35c4bb27 reports(2026-11-13): regenerate the Uber Eats 2026-10 page after re-scrape verification (154 unique / 153 completed orders; Oct 9 orders $301.04 unchanged) — data re-verified against the live Uber API 2026-11-13 19:21:28 +08:00
Tony0410 e033a08f17 audit(2026-11-12): the estate's last incident closed itself; a past-0.9.42 hq-push install was refused by the plugin scan (fix parked in PR #2)
- 429 row cbfe43_ed01845310d8 closed resolved 2026-11-12T00:25:36 on the next clean slot; 0 of 137 unclosed
- 11-12: 176 rows / 175 completed / 0 failed / 1 running; 11-11: 172/172/0; deliveries 9/9 and 7/7 delivered
- NEW incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md (dangerous verdict, one critical in the
  plugin's own tile skill; --force cannot override; no per-verdict override exists)
- systems/hermes-hq-push.md corrected live (0.9.27 -> 0.9.42, tree mtime 2026-11-08 02:19, config line 13226)
- current-state / scheduled-tasks / index / log updated; skills-index regenerated; freshness 2026-11-13 (0 stale)
2026-11-13 00:41:15 +08:00
Hermes ab3652aa20 Stocktake 2026-11-12: tailscale live-vs-page drift corrected; sftpgo incident re-verified; pre-dispatch-filter signature found recurring on a second job/pin
- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json.
  mailflow/qwenpaw/springroll read Online:false in the peer record but all three ping
  (CT1262 + CT465 running; springroll via DERP(sin)) - stale peer records, rows stay online.
  omarchymac genuinely stale in the other direction (offline 2d/11-08 -> offline 4d, ping timeout).
  9 named offline ages + 6 codex ephemeral peers refreshed. New guest CT1007 homebridge
  (running, tailscaled inactive). DEAD Serve targets 8769/9123 re-confirmed dead; CT201 table
  unchanged; container counts 17 LXC+4 VM (pve) / 20 LXC+2 VM (m93p) confirmed.
- incidents/2026-09-02-sftpgo-outage.md: 22d stale, re-probed live (pct list running, peer
  Online:true, pong, healthz 401; swap now 512 not 43Gi). Cleared.
- incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md: signature recurred 2026-11-10 on
  cbfe434a3b5e (custom:omni-101/free-stack, HTTP 429 credits 100%); incident row resolved
  2026-11-12T00:25. Original subject clean 4 slots. Retention-floor caveat recorded.
- Threshold re-verifications (19d/18d): mnemosyne-relay (tools:None gone, 0 log hits),
  work-outlook (himalaya still login not supported), omniroute-free-deepseek
  (good-free/free-stack both HTTP 200, non-deepseek resolution).
- Correction recorded: previous entry mis-described the 2026-11-12 stale page as log.md.
- wiki-maintain.sh re-run after all edits: freshness 0 stale / 130 fresh / 3 no-date of 133.
2026-11-12 03:19:12 +08:00
Hermes Wiki Audit b94de554f2 audit(2026-11-11): log the vault sync, the re-measured drift and the 2026-11-12 freshness report 2026-11-12 00:24:31 +08:00
Hermes Wiki Audit 1b0cf1fa71 audit(2026-11-11) follow-up: settle the incident-ledger row count and note acked_at is NULL 2026-11-12 00:18:26 +08:00
Hermes Wiki Audit 729bb597cf audit(2026-11-11): the audit's own 429 killed the 11-10 window; stocktake + radar recover with output
Window 2026-11-10 23:59 -> 2026-11-11 23:59 (two windows: the 11-10 audit failed).

NEW incident: incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
 - cbfe434a3b5e ran 23:59:54.486572 -> 2026-11-11 00:12:29.646987 = 12m35s, failed, delivery suppressed
 - RuntimeError: HTTP 429 ... all targets skipped by pre-dispatch filters (credits 100% conn:3021bab6)
   reset after 6h24m31s; 6 candidates, attempted 0; tenant tn-5ud77abtl1 quota exceeded
 - incident row cbfe43_ed01845310d8 (rate_limit) alerted_at NULL, closed_at NULL = only open row
 - consequence: the 2026-11-10 window was never audited; non-clean last_status 1 -> 2

RESOLVED with output (dated Recovery sections added):
 - stocktake 1a8e90ea2271 03:00:56.081504 -> 04:11:38.724852 = 1h10m42s, 36 stale re-verified -> 0
 - job radar stage1 07:01:08 -> 07:37:34 = 36m26s, first handoff since 2026-11-06 (22,682 B)
 - radar stage2 recovery fire 07:35:56 -> 08:03:47 closed row 5fc292_773822455856 resolved

UPDATED pages: current-state.md, systems/scheduled-tasks.md, systems/opencode-go.md
(opencode-zen 401 api_key_expired now observed 7x, paired with nous 404 credits 14x),
incidents/2026-10-22-nous-portal-refresh-token-invalid.md, index.md, log.md
2026-11-12 00:16:39 +08:00
Hermes c108eb1012 stocktake 2026-11-11: freshness report regenerated after the log/index addendum (index.md 1d -> 0d); published build now matches HEAD 2026-11-11 03:59:11 +08:00
Hermes 341dda95dd stocktake 2026-11-11: 36 stale pages re-verified (0 stale); fallback chain rewritten 4/6 slots; CT333 'clawtest' naming claim corrected; nous 404 credits signature replaces refresh-token 401; estate shrank (VM 310 + CT601 gone)
- freshness: 36 stale / 93 fresh / 3 no-date -> 0 stale / 129 fresh / 3 no-date of 132 (same-day report regenerated after the corrections, per pattern 13)
- skills: 299 skills / 67 categories (count unchanged; generator arithmetic re-verified by hand against the live tree: 300 SKILL.md files - 1 nested duplicate = 299)
- runbooks/test-openrouter: live fallback chain rewritten (primary back to smart-route/custom:omni-101; mimo-v2.5, stepfun, qwen3.5/ollama-cloud out; guacano + kilo-reflection-beam-free in); ollama.com dropped from the URL test list; guacano/litellm noted as internal + key-gated
- comparisons/hermes-vs-nanobot: the 2026-10-20 'CT333 is zeroclaw in Proxmox naming' claim was itself wrong - pct list and hostname both return clawtest
- incidents/2026-10-22-nous: refresh-token failure gone from logs; new signature 404 insufficient_credits_for_paid_model from auxiliary.pulse_memory (ninth slot, back on nous, paid model) - open item revised
- incidents/2026-08-29-gmail: email crons now ok but root cause unfixed 74 days (himalaya message read still in fetch_newsletters.py; goimapnotify back up PID 441; monitor still paused)
- 30 further pages re-verified live and de-flagged with dated probe notes (ct207 ingress, CT205 guanaco v0.8.15, CT270 nextcloud 33.0.2.2, headless stack, hermes-ui both hosts, systemd units + cgroup limits, terminal/tool_search/mnemosyne/custom_providers config blocks, ops profile still deleted, hosts/pve guest lists)
- log.md addendum + index.md date bump; no page deleted, no secrets written
2026-11-11 03:52:48 +08:00
Tony0410 7ceaec1a2f audit(2026-11-10): re-verify 2 stale pages (push bridge live/embedded, freshrss midday 0-failed in 22d); freshness 0 stale; log addendum 2026-11-10 01:10:17 +08:00
Tony0410 6de35a37ff audit(2026-11-10): post-audit freshness/maintenance pass 2026-11-10 01:06:19 +08:00
Hermes 262bcc74d5 audit(2026-11-10): 11-09 first failed-free cron day since 11-05; stocktake 'completed' with zero edits; Job Radar Stage 1 stall (9h09m, no handoff) documented; tailscale 58 nodes; omniroute 55 combos 2026-11-10 00:58:22 +08:00
Tony0410 3abd9901d3 freshness report regenerated after the nightly audit page-date corrections (0 stale / 127 fresh / 3 no-date of 130) 2026-11-09 00:22:47 +08:00
Tony0410 e2f1ae7a0b nightly audit 2026-11-09: pre_tool_call hook-timeout cascade (new incident page), hook budget 30->120s, state.db 1.89GB + corruption remnant, no-agent health monitor verified 73s, third silent stocktake incident, skills 299/67, freshness 0 stale/127 fresh of 130 2026-11-09 00:21:01 +08:00
Tony0410 4fd138bb18 audit 2026-11-08 (window 11-07): cron claim loss in gateway restart drain (2 runs orphaned, 2 incident rows unalerted); Hermes 0.22.5->0.23.0 (snapshot skipped), camofox plugin 1.1.0, conduit 0.134.4, hq-push 0.9.42, opencode-go key expired; skills 302->299 (4 archived); freshness 0/126/3 of 129 2026-11-08 00:41:58 +08:00
Tony0410 0b4ff25fd2 resolve: Gitea config backup push dead since 2026-08-08 — 92-commit backlog drained, script hardened to fail loudly
- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo
- Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved)
- Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB
- Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification,
  decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree,
  honest AgentMail/Telegram delivery reporting
- Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed
- Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated
- Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with
  'lost its durable fire claim ownership' and left the tree dirty)
2026-11-07 05:28:39 +08:00
hermes b820993ecc wiki audit 2026-11-07: Job Radar concurrent-fire claim discard (new incident page), scheduled-tasks re-verified section, index+log+runbook updates, freshness 2026-11-07 2026-11-07 00:48:09 +08:00
Tony0410 2d5219b068 stocktake 2026-11-06 (pass 2): re-verify the one page sitting on the 21-day threshold
incidents/2026-10-14-omniroute-node-connection-refused-storm.md was at exactly 21d and
would have tripped the threshold on the next run. Its residual claim ("omniroute-1
answers again") was re-probed live and still holds: TCP connect to 100.78.63.20:20128
succeeds (no Errno 111), the Tailscale peer reads Online:true, and a keyed
POST /v1/chat/completions returns HTTP 200. Page re-verified (updated/verified_on ->
2026-11-06, review_after -> 2026-12-06) with the probe recorded on the page.

New trap recorded on the page and in systems/scheduled-tasks.md: on omniroute-1 the
RAW model id deepseek/deepseek-v4-flash returns HTTP 401 "No active credentials for
provider: deepseek", while the fleet's deepseek-paid-smart alias returns HTTP 200 and
resolves to that same model. The 401 is an alias-routing artefact, not an outage - probe
the alias the fleet pins, never the raw model id. (A first probe using the wrong
key_env also produced a 401; omni-101's key env is HERMES_CUSTOM_100_78_63_20_20128_API_KEY.)

Freshness re-run after the edits: 0 stale / 124 fresh / 3 no-date of 127 pages.
Quartz rebuilt and redeployed. Vault mirror synced to both conventions (sha-verified).
2026-11-06 04:07:12 +08:00
Tony0410 88baf4fb5a stocktake 2026-11-06: 0 stale / 124 fresh / 3 no-date; correct Tailscale state + Docker seeder false negative
Ran wiki-maintain.sh (repo script). skills-index regenerated -> 302 skills / 68 categories.
Freshness audit at the 21-day threshold -> 0 stale / 124 fresh / 3 no-date of 127 pages.
Quartz rebuilt (761 files) and deployed to public/; hermes-wiki-serve.service restarted.

Live-vs-page drift diff (age scan alone missed all of this):

- infrastructure/tailscale.md: 1 state transition + 8 offline-age drifts + 2 dead Serve
  routes. anthonys-mac-mini came back online (peer record Online:true, tailscale ping ->
  pong via DERP(syd)). Offline ages refreshed +1d each (cloudreve 104d, elementsynapse
  107d, freshrss-new 80d, octoagent 4d, octop 2d, omarchybook 15d, omarchymac 4d,
  openmausbot 15d). Every peer IP still matches the live JSON; no peer missing, no
  orphan row. /agent-estate->8769 and /synergy-prep->9123 are DEAD (no listener) while
  still present in `tailscale serve status` - both now marked DEAD, matching
  network-map.md. All other Serve targets have live listeners.

- infrastructure/docker-services.md: the 2026-11-05 "the seeder does not exist on any
  host" finding was FALSE - a point-in-time sampling artefact. The worldmonitor-seeder-*
  containers are transient and systemd-timer driven on CT200 (worldmonitor-seed.timer
  4h -> seeder-scheduled; worldmonitor-seed-fast.timer 15min -> seeder-fast), visible
  only while a run is in flight. Both present at 2026-11-06 03:1x; neither at
  2026-11-05 05:1x. The "no cron job" claim was also wrong (systemd timers, not cron) and
  the "rename" narrative was wrong (one script, two container names). Corrected on the
  page with strikethrough + dated correction. CT200's 61-container running set still
  matches live set-wise. CT333 container renamed again: flamboyant_mahavira ->
  nostalgic_keldysh.

- systems/scheduled-tasks.md: no drift. Live jobs.json reads 29 jobs / 26 enabled /
  3 disabled, 1 non-clean last_status - exactly as recorded.

- current-state.md / index.md: false seeder claim and "every state matching" claim
  marked superseded with dated corrections (SCHEMA rule 9), not silently rewritten.

- infrastructure/network-map.md: dates bumped, dead routes re-confirmed, peer recheck
  note refreshed.

- log.md: appended (append-only) with the full stocktake record.

No secrets, tokens or keys written or quoted.
2026-11-06 03:50:34 +08:00
Tony0410 8abbc6bb60 audit 2026-11-06 follow-up: index.md frontmatter gains sources, de-duplicate header summary line 2026-11-06 00:32:23 +08:00
Tony0410 eba8062735 audit 2026-11-06 (window 2026-11-05): incident ledger emptied 3 -> 0, first zero-failed day since 11-02, both 11-04 casualties recovered; NEW OPEN incident - live .env values verbatim in 17 mnemosyne rows, the 1.7 GB session store and two enabled Facebook Birthday prompts; freshness 0 stale / 124 fresh / 3 no-date of 127 2026-11-06 00:29:52 +08:00
Tony0410 2f3fc9952b stocktake 2026-11-05: fix skills-index generator divergence at the root; correct Docker + Tailscale drift against live probes
ROOT CAUSE FIXED - systems/skills-index.md had TWO WRITERS:
the repo wiki-maintain.sh (302 skills / 68 categories) and the stale
~/.hermes/scripts/wiki-maintain.sh (mtime 2026-08-17, 349 skills / 26
categories - it used Path.rglob() and counted the parked .archive/ tree) were
different scripts with different generators, and wiki-maintenance/SKILL.md
line 569 pointed the 23:59 nightly audit at the stale path while the 03:00
stocktake ran the repo path. Each job overwrote the other's page; neither
raised an error. The stale copy is now a delegating wrapper with no generator
logic, and the repo script gained --no-build so the wrapper honours the old
interface without a second generator. Verified by execution.

Live-probe corrections:
- infrastructure/docker-services.md: CT200 61 running / 68 total (was 62);
  worldmonitor-seeder-scheduled does not exist on ANY host (zero 'seeder'
  matches across CT200/201/333/460); CT333 first container renamed again
  (nostalgic_leavitt -> flamboyant_mahavira); CT201 20/26, CT460 1/1,
  CT1001 1/9.
- infrastructure/tailscale.md: node table verified programmatically - 52 rows
  vs 52 live IPv4 peers, no live peer missing, no orphan row, every
  online/offline state matching. conduit 59d->60d, ubuntu-server-xfce
  75d->76d. Serve table still matches tailscale serve status exactly.
- infrastructure/hosts.md: skills figure corrected to 302 / 68 categories.
- index.md: skills-index bullet refreshed to the 2026-11-05 figures.

New incident: incidents/2026-11-05-skills-index-generator-divergence.md
(closes the still-open item on the 2026-08-15 skills-index drift incident).

Freshness: 0 stale / 123 fresh / 3 no-date of 126 pages.
2026-11-05 05:55:45 +08:00
Tony0410 20aa11b0c4 audit corrections 2026-11-05: fleet count 29/26/3 (was 158/144/14), 11-04 closes 167 completed/2 failed/1 running, 11-03 re-settled 171/1, deepseek-paid-smart 503 on omniroute:20128 vs 200 on omniroute-1 2026-11-05 01:38:41 +08:00
hermes 3b5daccb4d wiki: nightly audit 2026-11-05 window — network-map re-verified by live probe (9377=Camofox not Stratos, 8790/5174 gone, +9120/9222/7777/6767/8085/9900, full Tailscale Serve table, two dead routes 8769/9123); commit pending 11-04 stocktake re-verifications; log + skills-index 2026-11-05 01:22:45 +08:00
hermes e4086ec18e wiki: nightly audit 2026-11-04 window — new incident (Job Radar + Newsletter Digest both killed by upstream stream failure; Stage 2 rebuilt handoff and delivered 5 roles + 1 letter); CT299 second freeze documented; CL_MODEL 402 caveat in job-radar runbook; scheduled-tasks/current-state/index/log updated 2026-11-05 01:04:31 +08:00
Hermes Wiki Audit 785fbc40b2 wiki: nightly audit 2026-11-03 window — new Job Radar stream-timeout incident, hermes-hq-push system page; update 11-02 recurrence, scheduled-tasks, index, log 2026-11-04 00:16:23 +08:00
Tony0410 ca014efd63 Nightly audit 2026-11-03 (window 2026-11-02): new incident — Job Radar Stage 1 iteration-cap recurrence (9h 01m, suppressed, no incident raised); Weekly Review incident resolved (reconciled 01:27 AWST); scheduled-tasks + browser-backend re-verified; index/log updated 2026-11-03 00:22:13 +08:00
Tony0410 fda56e53fe audit 2026-11-02: Weekly Review lost to gateway handoff failure; three cron workers SIGTERM'd
Window 2026-11-01 00:00 -> 2026-11-02 00:4x AWST. 167 completed / 1 running / 2 unknown.

NEW incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (active)
NEW incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md (resolved)
UPDATED systems/scheduled-tasks.md (Count -> 2026-11-02 window, Re-verified section)
UPDATED systems/messaging-integrations.md (Telegram flapping, relay adapter absent ~29h)
UPDATED index.md, log.md
2026-11-02 00:27:09 +08:00
Tony0410 8bb3d196b1 reports(uber-eats): add October 2026 report, correct historical timestamps to Perth time
- Add 2026-10 report: 9 orders, $301.04 total, $33.45 avg
- Regenerate 2026-07/08/09 with correct Australia/Perth timestamps
  (Camofox browser TZ was hardcoded to America/Los_Angeles, making
  Uber's DOM render receipt times 15h behind with AM/PM stripped)
- Totals unchanged for historical months; only timestamps corrected
- Preserve original 'created' frontmatter date on regeneration
- Fix cron job 066bb5191280 delivery target + port reference
2026-11-01 05:56:18 +08:00
Tony0410 a0b3734c95 stocktake(2026-11-01): fix the Quartz build (YAML title), close the "unfixed memory ranking" incident, and correct live drift in tailscale/docker/proxmox
BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic.
Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an
unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a
mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)".
A scan of all 117 pages found exactly one such line; it is now properly quoted.
Quartz now builds cleanly: 690 files emitted and deployed.

RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page).
The page claimed the recall-ranking fix "was not present in live config" and that
conflict 152 was "the only open conflict". Both claims were false:
- The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5,
  fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 —
  not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the
  2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks.
- The original regression test now passes: "mnemosyne recall 'Are you up to date'"
  returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale
  Uber Eats episode no longer outranks anything.
- Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise.
All three open items closed; page moved to status: resolved.

LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01):
- infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping
  times out); m93p peer record reads false for the 4th consecutive time while the
  host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages
  refreshed; no IP changes, no missing peers.
- infrastructure/docker-services.md: CT201's Tailscale IP was wrong
  (100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale
  worldmonitor-seeder-fast entry removed from the CT200 list, which is now
  regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to
  thirsty_payne.
- infrastructure/proxmox.md: storage percentages refreshed on both hosts from live
  pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an
  elevated load average recorded.
- systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md:
  re-probed and verified_on bumped.
- systems/skills-index.md: regenerated — 301 unique skills in 68 categories,
  46 archived. (301 is the unique (category, skill_name) count; find returns 302
  files due to the known nested obsidian-git-sync alias duplicate.)

Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages.
2026-11-01 04:48:30 +08:00
Tony0410 c69681dc16 audit(2026-11-01): replace HTML entities in the mirror note 2026-11-01 00:33:17 +08:00
Tony0410 97660d3dcb audit(2026-11-01): record the whole-tree mirror catch-up in the log 2026-11-01 00:33:09 +08:00
Tony0410 0c826f2b5f audit(2026-11-01): Conduit updated to 0.122.2 (memory-timeout patch now upstream, call store re-shaped); document the local-voice-mode fault, the force-drained gateway reload that orphaned a cron run, and both dead Asana paths; 2026-10-31 settled 160/4 2026-11-01 00:30:42 +08:00
hermes 74a5b2d117 audit(2026-10-31): document CT460 5h30m host outage; correct the 'paused' health monitor (it was running, closed completed 10:02:48); reconcile fleet counts to 26/3 2026-10-31 00:39:11 +08:00
Tony0410 db6af66878 incident: m93p NFS mount loss took down all NFS-backed LXC containers (2026-10-30) 2026-10-30 17:41:34 +08:00
Tony0410 987eba72e6 Stocktake 2026-10-30: 0 stale by age, but 5 pages drifted from live state
Ran wiki-maintain.sh (skills-index regenerated: 302 skills / 69 categories,
unchanged; freshness report 0 stale / 109 fresh / 3 no-date of 112 pages;
Quartz rebuilt and deployed).

Freshness is not accuracy. A live-vs-page diff found real drift on five pages.
All corrections verified against live sources first.

- infrastructure/tailscale: node table rebuilt from tailscale status --json.
  50 non-funnel nodes (49 IPv4 peers + hermes), not 54. 2 IP changes, 4 peers
  expired out of the tailnet. Zero IP mismatches, zero missing/extra rows on
  re-verification. m93p records online (peer record chronically stale, third
  occurrence -- ping + SSH confirm). CT460 :443 route set (12 paths) and
  CT201 :443/:8443 tables re-probed and match.
- systems/scheduled-tasks: fleet is 29 jobs (25 enabled / 4 disabled), not
  28/25/3. Mnemosyne Health Monitor ea79a2107e36 was auto-paused 2026-10-30
  01:04:22 after its 00:01:08 slot closed "unknown" on a gateway-connection
  failure. Non-clean last_status is 4, not 3. "Failing (delivery only)" is 2.
  Seven job rows carried stale error claims for jobs since cleared to ok --
  each marked superseded with a dated correction, not rewritten (SCHEMA rule 9).
- NEW incidents/2026-10-30-mnemosyne-health-monitor-paused: the job's own
  failure notice was never delivered (deliveries.db has zero rows for
  2026-10-30), so the incident row is the only trace. Job not re-enabled and
  gateway not restarted by this run; root cause recorded as unverified.
- infrastructure/hosts: CT201 LAN is 192.168.178.77 (was .166); CT450 Hermes
  is v0.22.5 / bf0203a794a0, same build as CT460 (page said v0.22.3 /
  5b147ab036d6); cron count 29/25/4; orphaned "omni" peer has expired out of
  the tailnet entirely. Skills count of 302 left alone -- it is correct.
- systems/homepage-dashboard: live app count is 123 (was 114); grid is a
  responsive repeat(auto-fill, minmax(230px, 1fr)), not a fixed 5-column grid;
  tab bar is built from the service groups (renderTabs iterates state.groups),
  so the settings.yaml layout block and its tab:/columns: keys are inert --
  there is no "Overview" tab; saved theme is Boarding Pass / light, not dark
  amber; manager.py runs inside the homepage-manager container; and the "no
  Traefik container" evidence was wrong -- the container exists but is
  stopped (Exited (255) 2 months ago). The conclusion that Traefik does not
  occupy 443 stands. Same evidence error corrected in tailscale.md.

Also corrected two of my own first-pass errors before committing: the
homepage tab list (groups vs tabs) and the CT450 version (a stale root-level
/usr/local/bin/hermes reports v0.19.0; the live agent is the hermes-user
install at v0.22.5).

No secrets written -- provider names, key names and HTTP status codes only.
2026-10-30 05:41:11 +08:00
Tony0410 1d335ca8e6 wiki(audit): 2026-10-29 window — Conduit voice fault documented and repaired; radar two-stage split resolved both incidents (20 pages, 2 letters, Notion 455->475); cron 176/0 2026-10-30 00:21:21 +08:00
Tony0410 cc95dcc54d stocktake: re-verify Tailscale table, correct falsified cliprox/terra claim, regenerate skills-index
Wiki maintenance (wiki-maintain.sh, exit 0) + live-vs-page drift audit.

Script:
- systems/skills-index.md regenerated from the live skill tree: 302 skills in 69
  categories; 45 archived skills under ~/.hermes/skills/.archive/ now excluded and
  noted. (303 SKILL.md files exist outside .archive/, but
  productivity/obsidian-git-sync/obsidian-git-sync is a nested alias duplicate,
  so 302 unique is correct.)
- Freshness report raw/stocktake/freshness-20261029.txt: 2 stale / 105 fresh /
  3 no-date. After the corrections below and a second script run: 0 stale /
  107 fresh / 3 no-date.
- Quartz rebuilt (110 pages -> 644 files) and deployed to public/.

Drift corrections (live-verified):
- infrastructure/tailscale.md: State column rebuilt from tailscale status --json,
  11 rows changed. Two were state transitions, not age drift:
  * sftpgo (CT299, pve) is back online - tailscale ping pong in 0s,
    systemctl is-active sftpgo -> active, tailscale ip -4 inside the guest.
  * m93p is online despite a stale peer record (Online: false in the JSON) -
    tailscale ping pong in 1ms, SSH over the tailnet returned up 11 days.
    Second occurrence of this pattern after 2026-10-26.
  The other 9 rows were offline-age drift of exactly +1 day. Node count unchanged
  at 54 (53 live IPv4 peers + hermes); 76 peers total, 23 IPv6-only funnel
  ingress. No missing peers, no IP mismatches. Container identity re-confirmed
  from inside all 34 guests via pct exec tailscale ip -4 on both hypervisors;
  CT601 still absent from both.

- incidents/2026-10-01-cliproxy-connection-auth-expired.md: the residual claim
  "the connection credential itself was never re-authenticated" is falsified.
  Live POST /v1/chat/completions for cliprox/terra against custom:Omniroute:20128
  (CT600) returns HTTP 200 - the connection has since been re-authenticated. The
  same pin on custom:omni-101 (CT101) still returns HTTP 401, so the nodes carry
  separate credentials. Claim marked superseded, not deleted (SCHEMA rule 9); the
  remediation was never reversed and remains correct.

- incidents/2026-10-05-dead-model-pin-repin.md: every claim re-probed. All 7 jobs
  still on custom:omni-101 with routing aliases; both aliases HTTP 200;
  ollama-cloud/qwen3.5:397b still HTTP 410; nvidia/deepseek-v4-flash still
  HTTP 400; command-code/gpt-5.6-terra now HTTP 401 rather than the original 403
  - still dead, different failure signature.

- systems/scheduled-tasks.md: the Known-Issues entry repeating the
  "never re-authenticated" claim now carries the correction. Fleet figures
  re-checked against live jobs.json and remain accurate (29 jobs / 26 enabled /
  3 disabled, 5 non-clean last_status).

- log.md / index.md: stocktake entry appended and index last-updated refreshed.

Verified correct, no change: systems/homepage-dashboard (CT201 has 8084 and 8085
listening, no 8082, Homarr on 8102); CT201 Serve routes (/ -> 8085, /mgr -> 8084
on both :443 and :8443, Steel routes still gone); CT460 Serve routes (12 paths
on :443, plus :8443/webhook Funnel and :9443).

Noted for a human, not performed: vault mirror at /home/hermes/wiki/agent-estate-wiki
has drifted on both conventions (top-level 35 match / 3 stale / 72 missing;
wiki/ 41 match / 53 stale / 16 missing). A full mirror pass is larger than a
stocktake's mandate.
2026-10-29 03:42:54 +08:00
Tony0410andHermes 677bb0c959 docs(wiki): correct Steel Browser references across 15 pages — Steel fully removed, Camofox is sole backend
Root cause of the long-standing cloud_provider: steel residue: the 2026-07-30
rollback disabled the browser-steel plugin and set browser.engine: camofox but
left browser.cloud_provider: steel in config.yaml. The split config was
inert-but-wrong, so it survived ~3 months of audits unnoticed.

Corrects pages that still described Steel as present, running or configured:
- systems/browser-backend.md (incl. gateway-restart env-var note)
- comparisons/steel-vs-camoufox.md, comparisons/camoufox-vs-chrome.md
- decisions/rollback-steel-to-camofox.md, decisions/index.md
- infrastructure/docker-services.md, infrastructure/tailscale.md, infrastructure/hosts.md
- systems/homepage-dashboard.md, systems/scheduled-tasks.md
- concepts/browser-automation.md, runbooks/recover-docker-service.md
- current-state.md, index.md, agents/nanobot.md
- log.md (dated entry; supersedes the 00:3x audit's 'endpoint dark' wording)

Also restores the Ignis compose row in infrastructure/docker-services.md, which
the 2026-10-11 revision dropped (verified: /home/hermes/docker/ignis/docker-compose.yml
exists and the ignis container is Up on CT460).

Co-Authored-By: Hermes <hermes@local>
2026-10-29 02:02:49 +08:00
hermes ec0c30fb30 wiki(audit): 2026-10-28 window — radar stage-1 iteration cap (no handoff, stage-2 no-oped), stocktake 400 never alerted, Steel removed from the stack 2026-10-29 00:41:30 +08:00
Tony0410 2cbf0b65c1 audit: correct the 2026-10-28 log entry heading to this window and this story 2026-10-28 00:21:33 +08:00
Tony0410 3ae05871c4 audit: align verified_on for the 2026-10-26 orphan incident with the 2026-10-28 recurrence check 2026-10-28 00:21:24 +08:00
Tony0410 f880ba981d audit: 2026-10-28 nightly — radar budget overrun now a recorded failure (new incident page); radar split into two cron jobs; tailscale peer table re-verified; TTS provider corrected to the Mac mini speech server
Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76
(Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed
failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before
Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and
cover letters never ran (Notion DB held 451 entries, 0 added). Incident row
dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified).

Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5,
deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5,
topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3).

Also: the two incident rows the 2026-10-26 planned restart raised auto-closed
00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean
last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never
committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of
108) — committed here and index.md corrected; tailscale.md peer table rebuilt from
tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo
CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to
provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained
the voice pipeline section.

New: incidents/2026-10-27-job-radar-budget-exhausted.md
Nothing removed or archived. No secrets.
2026-10-28 00:20:59 +08:00
Tony0410 a2dd0bd650 audit: 2026-10-26 nightly — planned restart orphaned a live cron run (new incident page); 503 burst fully closed; radar cover-letter PDF pipeline moved to Chromium-over-HTML 2026-10-27 00:11:50 +08:00
Tony0410 3f796dc3b9 stocktake(2026-10-26): strike superseded follow-ups per SCHEMA rule 9
Mark-don't-delete: the CT450 '3 commits behind' follow-up and the two
mnemosyne-vector-recall follow-ups are struck through with their verified
2026-10-26 resolution rather than removed.

- CT450 tracks origin/main (there is no origin/master on that remote) and is
  0 commits behind.
- Coverage monitor b63ed5817212 is enabled, last_status ok, working coverage
  96.8% — correctly has nothing to fire on.
- fastembed 0.8.1 now imports in both managed runtimes plus the legacy venv.
2026-10-26 04:35:33 +08:00
Tony0410 7fe109b914 stocktake(2026-10-26): record measured vault-mirror drift counts in log.md
Vault at /home/hermes/wiki tracks 125 files under two conventions. This run
forward-synced all 22 touched-page copies (sha256-verified) and committed the
vault locally; it is local-only and was not pushed. Remaining drift measured by
hash sweep and recorded rather than silently mirrored:
  top-level: match=25 stale=2 missing=80
  wiki/     : match=34 stale=58 missing=15
2026-10-26 04:30:35 +08:00
168 changed files with 12836 additions and 1511 deletions
+1
View File
@@ -3,3 +3,4 @@ public/
package-lock.json package-lock.json
package.json package.json
*.log *.log
.audit-report-*.txt
+12 -3
View File
@@ -3,8 +3,8 @@ title: Contributing to the Agent Estate Wiki
type: guide type: guide
status: active status: active
created: 2026-08-16 created: 2026-08-16
updated: 2026-10-20 updated: 2026-11-11
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: high confidence: high
tags: [wiki, contributing, governance, multi-agent] tags: [wiki, contributing, governance, multi-agent]
sources: [] sources: []
@@ -47,7 +47,7 @@ Each agent owns **its own section only**. Do not edit another agent's pages with
| `agents/nanobot/` | Nanobot (CT333) | | `agents/nanobot/` | Nanobot (CT333) |
| `agents/zeroclaw/` | ZeroClaw (CT333) | | `agents/zeroclaw/` | ZeroClaw (CT333) |
| `agents/qwenpaw/` | QwenPaw (CT465) | | `agents/qwenpaw/` | QwenPaw (CT465) |
| `agents/open-claw/` | Open-Claw (VM 403) | | ~~`agents/open-claw/`~~ | ~~Open-Claw (VM 403)~~ — **removed 2026-10-20; re-confirmed absent 2026-11-11** |
| `shared/`, `infrastructure/`, `runbooks/`, `decisions/`, `incidents/` | Anyone, but **facts must be verified against live state** and linked to a source | | `shared/`, `infrastructure/`, `runbooks/`, `decisions/`, `incidents/` | Anyone, but **facts must be verified against live state** and linked to a source |
## What to save (and what NOT to save) ## What to save (and what NOT to save)
@@ -100,6 +100,15 @@ After pushing, the Quartz web view updates on the next daily maintenance run (03
4. Make your first commit: your role, host, config pointers, and known limitations. 4. Make your first commit: your role, host, config pointers, and known limitations.
5. Keep `updated:` honest on every edit. 5. Keep `updated:` honest on every edit.
> **Access paths re-verified 2026-11-11.** `http://100.118.5.51:9120/` returns **200** (as does a
> real page such as `/systems/skills-index`), the unit behind it is
> `hermes-wiki-serve.service` — **active**, listening on `0.0.0.0:9120` (a **user** unit — check it
> with `systemctl --user`, never bare `systemctl`, or it reads as dead). The external route
> `https://hermes.kangaroo-eel.ts.net/wiki/` returns **200** via the Tailscale Serve path.
> `git remote -v` confirms the remote is the documented
> `http://…@gitea:3000/Anthony/agent-estate-wiki.git` — note the URL carries a credential; it is
> deliberately **not** reproduced here, use `git remote -v` to read it.
## Related ## Related
- [[agents/index]] — estate map - [[agents/index]] — estate map
+4 -4
View File
@@ -3,8 +3,8 @@ title: Hermes Chaos
type: system type: system
status: active status: active
created: 2026-08-15 created: 2026-08-15
updated: 2026-10-26 updated: 2026-11-08
verified_on: 2026-10-26 verified_on: 2026-11-08
confidence: high confidence: high
tags: [hermes, chaos, ct450, experimental, agent] tags: [hermes, chaos, ct450, experimental, agent]
sources: [live-probe, config-inspection] sources: [live-probe, config-inspection]
@@ -28,10 +28,10 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod
## Current State ## Current State
- **Status:** Active (experimental) - **Status:** Active (experimental)
- **Hermes version:** `v0.22.3` (2026.9.24), local `5b147ab036d6` — **the same build as CT460** (updated 2026-10-21). **0 commits behind `origin/main`** (verified 2026-10-26; the old "3 commits behind" claim is stale), though `hermes update --check` reports stable **v0.22.5** available. Python 3.14.7. See [[incidents/2026-10-03-ct450-hermes-update]]. - **Hermes version:** ⚠ **now `v0.23.0`** — the live install stamp `~/.hermes/hermes-agent/install-stamp.json` inside CT450 reads `baseVersion: 0.23.0`, `displayVersion: 0.23.0`, `distance: 0`, `payload: bootstrap` (probed 2026-11-08 00:4x AWST by the nightly audit; `hermes --version` not re-run in that probe). The rest of this bullet is the 2026-11-07 correction, written when CT450 was still `v0.22.5` (2026.9.24), local `bf0203a794a0` — **the same build as CT460** (updated 2026-10-21). ⚠ **Corrected 2026-11-07:** this line still read *"`v0.22.3` … local `5b147ab036d6`"* while asserting it was *"the same build as CT460"* — it was not. Live `hermes --version` on CT450 then returned **`v0.22.5` … local `bf0203a794a0`**, byte-identical to CT460's own `hermes --version`, so the *same build* claim finally became true. `hermes update --check` then reported stable **v0.23.0** (`2bd378ed1b8d`) available; CT460 took it on 2026-11-07 and CT450's stamp now reads the same version. Python 3.14.9. See [[incidents/2026-10-03-ct450-hermes-update]].
- **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]]. - **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]].
- **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01) - **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01)
- **Cron jobs:** **25 — 13 enabled / 12 disabled** (verified live in CT450's `jobs.json` 2026-10-26). **9 job names are shared with CT460**; 8 of those are disabled here and enabled on CT460, while `Important-mail monitor` is the reverse (enabled on CT450, disabled on CT460) - **Cron jobs:** **25 — 13 enabled / 12 disabled** (verified live in CT450's `jobs.json` **2026-11-07**; unchanged). **9 job names are shared with CT460**; 8 of those are disabled here and enabled on CT460, while `Important-mail monitor` is the reverse (enabled on CT450, disabled on CT460) — both confirmed by a live set-diff of the two `jobs.json` files on 2026-11-07
- **Production counterparts:** CT460 has **28 jobs / 25 enabled** — the two fleets no longer share a job count, and the CT450 duplicates are paused - **Production counterparts:** CT460 has **28 jobs / 25 enabled** — the two fleets no longer share a job count, and the CT450 duplicates are paused
## Providers & Routing ## Providers & Routing
+6 -5
View File
@@ -3,8 +3,8 @@ title: Hermes Production
type: system type: system
status: active status: active
created: 2026-08-15 created: 2026-08-15
updated: 2026-10-26 updated: 2026-11-08
verified_on: 2026-10-26 verified_on: 2026-11-08
confidence: high confidence: high
tags: [hermes, production, ct460, agent] tags: [hermes, production, ct460, agent]
sources: [live-probe, config-inspection] sources: [live-probe, config-inspection]
@@ -27,7 +27,8 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
## Current State ## Current State
- **Status:** Active - **Status:** Active
- **Hermes version:** `v0.22.3` (2026.9.24), local `5b147ab036d6` — updated 2026-10-21 22:47→22:57 AWST; **verified live 2026-10-26**. (`hermes update --check` reports stable **v0.22.5** available) - **Hermes version:** ⚠ **`v0.23.0`** — updated **2026-11-07 05:05:13 → 05:43:10 AWST** by `hermes update` (method `git`, outcome **`success`**, 37 m 57 s), `bf0203a794a07aac3ff764972ab9087baa4e4f4e` → **`2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`**; local version `0.22.5` → `0.23.0`. ⚠ The **`snapshot` stage was skipped again** (`disabled by updates.pre_update_backup (mode: off)`) → **no rollback point**. Restarts: `hermes-gateway` (pre-update `pid 2981474`) and `hermes-dashboard` (`Stopping` 05:43:03 → `Started` 05:43:04, `:9119`); the gateway went round **again 11:11:44 → 11:11:55** (live process `pid 3109529`). **Live-probed 2026-11-08 00:4x AWST.** Previous: `v0.22.3` (2026.9.24), local `5b147ab036d6` — updated 2026-10-21 22:47→22:57 AWST.
- **Cron ledger 2026-11-07:** 163 rows — **159 `completed` / 3 `failed` / 1 `running`** (that *running* row is this nightly audit). Two of the three failures are one event: the **05:01:00 gateway restart drain** orphaned *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42 → 05:02:22) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31 → 05:02:22) — both closed `failed` on `lost its durable fire claim ownership`. The third (`96eb95ce12f2`, 05:02:41 → 05:02:59) is the agent-log ERROR watchdog firing 19 s later on the first failure's traceback. → [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15) - **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
- **Install directory:** `/home/hermes/.hermes/hermes-agent` - **Install directory:** `/home/hermes/.hermes/hermes-agent`
- **Config:** `~/.hermes/config.yaml` - **Config:** `~/.hermes/config.yaml`
@@ -43,7 +44,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
## Providers & Routing ## Providers & Routing
- **Default provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`, CT101/m93p) — verified live 2026-10-26. The `custom:omni` provider key does not exist in `custom_providers`, but **2 cron jobs still pin it** (see [[systems/scheduled-tasks]]) - **Default provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`, CT101/m93p) — verified live 2026-10-26, **re-verified 2026-11-07** (live `config.yaml` `model.default.provider: custom:omni-101`; a keyed `deepseek-paid-smart` chat probe against `100.78.63.20:20128` returns **HTTP 200**). The `custom:omni` provider key does not exist in `custom_providers`. ⚠ **Corrected 2026-11-07:** the old claim that *"2 cron jobs still pin `custom:omni`"* is **no longer true** — live `jobs.json` shows **no job pinned to the bare `custom:omni`**; **all 21 provider-pinned jobs use `custom:omni-101`** (see [[systems/scheduled-tasks]])
- **Default model:** `smart-route` (previously `freellmapi-auto`) - **Default model:** `smart-route` (previously `freellmapi-auto`)
- **Fallback providers** (live chain, re-verified 2026-10-26): `nous` / `meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` → `custom:guacano` / `deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` → `kilocode` / `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6_1t`. ⚠ The **first** entry is the dead `nous` credential — a fallover tries it first - **Fallback providers** (live chain, re-verified 2026-10-26): `nous` / `meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` → `custom:guacano` / `deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` → `kilocode` / `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6_1t`. ⚠ The **first** entry is the dead `nous` credential — a fallover tries it first
- **Auxiliary models:** all eight `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) are **`no-think/freeapi/auto` on `custom:omni-101`** with a `freeapi/deepseek-v4-flash` → `kg/kilo-auto/free` fallback chain (moved 2026-10-25 19:13) - **Auxiliary models:** all eight `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) are **`no-think/freeapi/auto` on `custom:omni-101`** with a `freeapi/deepseek-v4-flash` → `kg/kilo-auto/free` fallback chain (moved 2026-10-25 19:13)
@@ -74,7 +75,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
| Aspect | Production (CT460) | Chaos (CT450) | | Aspect | Production (CT460) | Chaos (CT450) |
|--------|-------------------|---------------| |--------|-------------------|---------------|
| Purpose | Stable daily operations | Experimental/testing | | Purpose | Stable daily operations | Experimental/testing |
| Cron jobs | 28 (25 enabled / 3 disabled) | 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460 | | Cron jobs | **29 (26 enabled / 3 disabled)** (re-verified 2026-11-07) | 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460 (re-verified 2026-11-07) |
| Config changes | Requires verification | Safe to experiment | | Config changes | Requires verification | Safe to experiment |
| Memory | Production Mnemosyne | Separate state | | Memory | Production Mnemosyne | Separate state |
+10 -2
View File
@@ -3,8 +3,8 @@ title: Agent Estate
type: system type: system
status: active status: active
created: 2026-08-15 created: 2026-08-15
updated: 2026-10-20 updated: 2026-11-11
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: high confidence: high
tags: [agents, estate, overview] tags: [agents, estate, overview]
--- ---
@@ -40,6 +40,14 @@ also has **no VM 403** (its LXC guests are 100, 101, 102, 104, 105, 221, 245, 50
**ZeroClaw, by contrast, was re-verified live on 2026-10-20** and **is running** on CT333 **ZeroClaw, by contrast, was re-verified live on 2026-10-20** and **is running** on CT333
(`/usr/local/bin/zeroclaw daemon --allow-degraded-security`, PID 136) — see [[agents/zeroclaw]]. (`/usr/local/bin/zeroclaw daemon --allow-degraded-security`, PID 136) — see [[agents/zeroclaw]].
> **Guest lists re-probed 2026-11-11 — VM 403 and VM 401 still absent, and the estate shrank a
> little more.** `qm list` on `pve` (`100.96.100.82`) now shows only VMs **100, 101, 110, 282**
> (VM **310** has gone since the 2026-10-19 probe listed it), and `pct list` shows CTs 103, 200,
> 201, 205, 207, 209 (stopped), 215, 255, 270, 299 (stopped), 333, 450, 460, 465, 475, 555
> (stopped), 600 — CT **601** is gone. Live CT names: `205 ollama`, `333 clawtest`,
> `450 hermesagent`, `600 omniroute`. `m93p` was not re-probed this pass; its 2026-10-20 list
> stands. ZeroClaw is running on CT333 (PID 142, `--allow-degraded-security`).
## Shared Infrastructure ## Shared Infrastructure
All agents share: All agents share:
+44 -17
View File
@@ -3,8 +3,8 @@ title: Nanobot
type: agent type: agent
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-11 updated: 2026-11-20
verified_on: 2026-10-11 verified_on: 2026-11-20
confidence: high confidence: high
tags: [agent, nanobot, homelab] tags: [agent, nanobot, homelab]
sources: [live-probe, runtime-config] sources: [live-probe, runtime-config]
@@ -20,7 +20,7 @@ Primary homelab AI agent running on `clawtest` (CT333). Handles agent orchestrat
- **Workspace:** `/home/nano/.nanobot/workspace` - **Workspace:** `/home/nano/.nanobot/workspace`
## Current state ## Current state
- **Status: ACTIVE.** Verified 2026-10-11: gateway process `nanobot-gateway` (PID 2259048) running and listening. - **Status: ACTIVE.** Re-verified 2026-11-20: `nanobot-gateway` PID 1256104 (since 2026-11-19), `/health` → `status ok, ready true`, v0.4.1. (Prior check 2026-10-11: PID 2259048.)
- **Gateway:** port `8765` (HTTP API + chat completions) — `LISTEN 0.0.0.0:8765` - **Gateway:** port `8765` (HTTP API + chat completions) — `LISTEN 0.0.0.0:8765`
- **WebSocket bridge:** port `8766` — `LISTEN 0.0.0.0:8766` - **WebSocket bridge:** port `8766` — `LISTEN 0.0.0.0:8766`
- **Serve API:** port `8900` was recorded as the Serve API; **not listening** in the 2026-10-11 `ss -ltn` check. Treat as unverified/possibly removed. - **Serve API:** port `8900` was recorded as the Serve API; **not listening** in the 2026-10-11 `ss -ltn` check. Treat as unverified/possibly removed.
@@ -35,23 +35,34 @@ Primary homelab AI agent running on `clawtest` (CT333). Handles agent orchestrat
| `agentmail-daemon.py` | nano | — | started Sep 23 | | `agentmail-daemon.py` | nano | — | started Sep 23 |
| `readlater` Next dev server | nano | 3210 | started Sep 23 | | `readlater` Next dev server | nano | 3210 | started Sep 23 |
Other listeners on CT333 include 8787, 8880, 8899, 8999, 3000, 5050, 8080 and the Tailscale 443. Other listeners on CT333 include 8787, 8880, 8899, 8999, 5050, 8080 and the Tailscale 443. *(Port 3000 was published by the `steel-browser` container, which was removed 2026-10-29.)*
## Docker on CT333 (verified 2026-10-11) ## Docker on CT333 (verified 2026-10-11; Steel container removed 2026-10-29)
`steel-browser`, `gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`, `edge-tts`, `onecli-postgres-1`, `tender_brattain`, plus healthy `community-spark` and `quirky_darwin`. ~~`tender_brattain`, plus healthy `community-spark` and `quirky_darwin`~~ → **2026-11-20 live list:** `gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`, `edge-tts`, `onecli-postgres-1`, `ecstatic_swartz` (random-name successor of `tender_brattain`), healthy `community-spark`; `quirky_darwin` removed; stopped `qdrant` present (Exited 5 months ago).
**Steel is gone from CT333 too.** The `steel-browser` container (image `ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image were removed on **2026-10-29**. It was already orphaned — its last CDP navigation was 2026-10-20, it crashed on 2026-10-22, it received no requests after 2026-10-21, and it had no Tailscale Serve route, no entry in `~/.zeroclaw/config.toml` (which uses `backend = "agent_browser"` on native_webdriver at `127.0.0.1:9515`), and no reference from Nanobot's browser harness. → [[comparisons/steel-vs-camoufox]]
## Cron jobs ## Cron jobs
Unverified against live Nanobot scheduler on 2026-10-11. Previously recorded: **Verified live 2026-11-20** from `/home/nano/.nanobot/workspace/cron/jobs.json` — **11 jobs, 8 enabled**
- `auto-upgrade-freellmapi-safeguard` — daily at 03:30 AWST (all schedules Australia/Perth):
- `morning-quota-report` — daily at 07:30 AWST
- `daily-quota-730am` — daily at 07:30 AWST | Job | Schedule | Enabled |
- `vault-daily-seed` — daily at 08:00 AWST |---|---|---|
- `daily-agent-briefing` — daily at 08:15 AWST | `auto-upgrade-freellmapi-safeguard` | daily 03:30 | ✅ |
- `rss-check-proxmox-selfhst` — Mon/Thu at 09:00 AWST | `nightly-skills-export-sync` | daily 04:00 | ✅ |
- `daily-quota-7pm` — daily at 19:00 AWST | `morning-news-podcast` | daily 07:00 | ✅ |
- `evening-quota-report` — daily at 21:00 AWST | `daily-quota-730am` | daily 07:30 | ✅ |
- `heartbeat` — every 30m | `morning-quota-report` | daily 07:30 | ❌ disabled |
- `dream` — every 2h (memory consolidation) | `vault-daily-seed` | daily 08:00 | ✅ |
| `daily-agent-briefing` | daily 08:15 | ❌ disabled (2026-10-30) |
| `daily-quota-7pm` | daily 19:00 | ✅ |
| `evening-quota-report` | daily 21:00 | ❌ disabled |
| `heartbeat` | every 30m | ✅ |
| `dream` | every 2h (memory consolidation) | ✅ |
~~`rss-check-proxmox-selfhst` — Mon/Thu at 09:00 AWST~~ **No longer in the scheduler** (removed at
some point before 2026-11-20; the enabled quota-notification pair `daily-quota-730am` /
`daily-quota-7pm` remains enabled, per Anthony's standing preference).
## Skills ## Skills
Active skills include: `agent-reach`, `verify-nanobot-config-against-docs`, `cli-app-exa`, `playwright`, `system-prompts-collection`, `agent-skill-retrieval`, `enhanced-search`, `api-gateway`, `debugging-and-error-recovery`, `mnemosyne-memory-hooks`, `proxmox-shell-fallback`, `agentmail`, `langfuse`, `install-and-verify-agent-skills`, `verify-system-claims`, `skill-interceptor`, `tool-call-failure-recovery`, `using-agent-skills`, `api-vault`, `repair-memory`, `patch-config-relative-paths`, `long-task-management`, `nanobot-full-restore-backup`, `context-diet`, `frontend-design`. Active skills include: `agent-reach`, `verify-nanobot-config-against-docs`, `cli-app-exa`, `playwright`, `system-prompts-collection`, `agent-skill-retrieval`, `enhanced-search`, `api-gateway`, `debugging-and-error-recovery`, `mnemosyne-memory-hooks`, `proxmox-shell-fallback`, `agentmail`, `langfuse`, `install-and-verify-agent-skills`, `verify-system-claims`, `skill-interceptor`, `tool-call-failure-recovery`, `using-agent-skills`, `api-vault`, `repair-memory`, `patch-config-relative-paths`, `long-task-management`, `nanobot-full-restore-backup`, `context-diet`, `frontend-design`.
@@ -71,5 +82,21 @@ Active skills include: `agent-reach`, `verify-nanobot-config-against-docs`, `cli
- [[runbooks/restart-hermes]] - [[runbooks/restart-hermes]]
- [[runbooks/nanobot-restart]] - [[runbooks/nanobot-restart]]
> **Re-verified 2026-11-20 (stocktake, live probes on CT333).** Gateway healthy:
> `GET :8765/health` → `{"status":"ok","ready":true,"websocket":"running","version":"0.4.1"}`;
> `nanobot-gateway` PID **1256104** (started 2026-11-19), ports 8765 + 8766 listening, 8900 still
> **not listening**. Tailscale IP confirmed `100.122.189.81`. The mnemosyne MCP, chroma_query_api,
> agentmail-daemon and readlater (:3210) processes all still run.
>
> **Docker drift corrected:** live `docker ps -a` shows `ecstatic_swartz` (Up 19h) where this page
> listed `tender_brattain` — Docker regenerates random names for compose-less containers, so this is
> a rename-class drift, not a new service. `quirky_darwin` is **absent even with `-a`** (removed, not
> stopped). `community-spark`, `gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`,
> `edge-tts`, `onecli-postgres-1` all Up ~9 days and healthy where applicable; a stopped `qdrant`
> (Exited 5 months ago) also exists. No steel container or image remains; nothing on :3000.
>
> **Cron verified live 2026-11-20** from `/home/nano/.nanobot/workspace/cron/jobs.json` (the
> `~/.nanobot/cron/` dir is empty — the scheduler file lives under `workspace/cron/`):
> **11 jobs, 8 enabled** — see the rewritten Cron jobs section above.
## Related ## Related
- [[comparisons/hermes-vs-nanobot]] - [[comparisons/hermes-vs-nanobot]]
+8 -2
View File
@@ -3,8 +3,8 @@ title: Open-Claw
type: system type: system
status: active status: active
created: 2026-08-15 created: 2026-08-15
updated: 2026-10-20 updated: 2026-11-11
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: high confidence: high
tags: [openclaw, open-claw, vm403, gateway, agent] tags: [openclaw, open-claw, vm403, gateway, agent]
sources: [skill-inspection, config-reference] sources: [skill-inspection, config-reference]
@@ -36,6 +36,12 @@ Autonomous AI agent gateway running on a separate QEMU VM. Docker-based with Tai
1263 `social-dashboard`, 1264 `openmausbot`, 998 `nightbridge` (stopped) — and **no VM 403 there 1263 `social-dashboard`, 1264 `openmausbot`, 998 `nightbridge` (stopped) — and **no VM 403 there
either**. The Tailscale peer list likewise contains **no `open-claw` node**. So Open-Claw is not either**. The Tailscale peer list likewise contains **no `open-claw` node**. So Open-Claw is not
merely moved between hosts: it is gone from both. merely moved between hosts: it is gone from both.
- **Guest lists re-probed 2026-11-11.** `qm list` on `pve` (`100.96.100.82`) now shows only VMs
**100, 101, 110, 282** (VM **310** is gone too — the page previously listed it), and `pct list`
shows CTs 103, 200, 201, 205, 207, 209 (stopped), 215, 255, 270, 299 (stopped), 333, 450, 460,
465, 475, 555 (stopped), **600** — CT **601 is gone** as well. Live CT names read
`205 ollama`, `333 clawtest`, `450 hermesagent`, `600 omniroute`. **No VM 403 on either hypervisor,
and no `open-claw` Tailscale peer** — the "NOT PRESENT" verdict holds and is now ~3 weeks stable.
- **Consequence for the rest of this page:** the Tailscale IP `100.64.172.34`, the LAN IP - **Consequence for the rest of this page:** the Tailscale IP `100.64.172.34`, the LAN IP
`192.168.178.193` and the control UI `https://open-claw.kangaroo-eel.ts.net` are all historical. `192.168.178.193` and the control UI `https://open-claw.kangaroo-eel.ts.net` are all historical.
Treat the whole page as a design record, not a description of a running system. Treat the whole page as a design record, not a description of a running system.
+9 -7
View File
@@ -3,8 +3,8 @@ title: QwenPaw
type: system type: system
status: active status: active
created: 2026-08-15 created: 2026-08-15
updated: 2026-10-11 updated: 2026-11-04
verified_on: 2026-10-11 verified_on: 2026-11-04
confidence: high confidence: high
tags: [qwenpaw, ct465, agent, paseo] tags: [qwenpaw, ct465, agent, paseo]
sources: [live-probe, config-reference] sources: [live-probe, config-reference]
@@ -24,11 +24,13 @@ Multi-agent gateway platform. Runs on CT465 with its own workspace, agent profil
## Current State ## Current State
- **Status:** Active — `paseo.service` active on CT465, verified 2026-10-11 - **Status:** Active — `paseo.service` active on CT465, **re-verified 2026-11-04**
- **Default chat model:** OmniRoute Smart Route (`auto/smart`) - **Default chat model:** OmniRoute Smart Route (`auto/smart`)
- **Config format:** JSON (`agent.json` per workspace) - **Config format:** JSON (`agent.json` per workspace)
- **Daemon:** `paseo.service` - **Daemon:** `paseo.service` — **system** unit on CT465 (distinct from the **user** unit `paseo-daemon.service` that runs on CT460; see [[systems/paseo]])
- **Also listening on CT465:** PVE exporter (`100.96.150.69:9221`), a Python service on `:9876`, and local SMTP `:25` - **CT465 listening ports (live 2026-11-04):** `0.0.0.0:6767` (Paseo), `*:13500`, `127.0.0.1:25463`, `127.0.0.1:11434` (Ollama), `127.0.0.1:38585`, `100.110.72.126:42510`, `:22`
- ⚠ **Corrected 2026-11-04.** The old claim that "PVE exporter (`100.96.150.69:9221`), a Python service on `:9876`, and local SMTP `:25`" were listening **on CT465** was **wrong on all three counts**. Live `ss -ltn` inside CT465 shows **none** of them. `:9221` and `:9876` listen on the **`pve` hypervisor host itself** (`100.96.100.82`), not in the guest, and `100.96.150.69` is **CT200** (the Docker host), not the PVE exporter. `:25` is the hypervisor's own loopback SMTP, also not in CT465.
- **Port 6767 has two live implementations — do not conflate them:** CT465's own `paseo.service` listens on `0.0.0.0:6767` and answers **HTTP 404** on `/` (the daemon is alive; `/` is not a route). The tailnet node `paseo` (`100.72.188.93`) is a **different guest** — CT1261 on `m93p` — also running `paseo.service` on `:6767`, and it answers **HTTP 200** on `/`. CT460's own user-unit daemon is a third instance (also `:6767`, **HTTP 404** on `/`; `/api/health` → 200).
- **CT400/CT465 identity:** confirmed CT465 = `qwenpaw`, running - **CT400/CT465 identity:** confirmed CT465 = `qwenpaw`, running
## Operating Layers ## Operating Layers
@@ -60,8 +62,8 @@ qwenpaw daemon status
When adding an OpenAI-compatible provider from Proxmox: When adding an OpenAI-compatible provider from Proxmox:
- Verify target by VMID and hostname before writing config - Verify target by VMID and hostname before writing config
- CT600/`omniroute` is the **live** OmniRoute service at `100.88.81.19:20128` — verified serving 2026-10-11 - CT600/`omniroute` is the **live** OmniRoute service at `100.88.81.19:20128` — verified serving 2026-10-11, still live 2026-11-04
- CT601/`omni` is a separate guest at `100.93.204.4` — **stopped**, do not use - ⚠ **Corrected 2026-11-04:** CT601/`omni` is **no longer present on either hypervisor** — `pct list` on both `pve` and `m93p` shows **no CT601 at all**, and the `omni` tailnet peer (`100.93.204.4`) has dropped out of the tailnet entirely. The guest was **removed**, not merely stopped. Use CT600.
- A third instance `omniroute-1` at `100.78.63.20:20128` is also live; Hermes registers it as provider `omni-101` - A third instance `omniroute-1` at `100.78.63.20:20128` is also live; Hermes registers it as provider `omni-101`
- Query `/v1/models` using the exact API key stored for QwenPaw, not a management key - Query `/v1/models` using the exact API key stored for QwenPaw, not a management key
- See [[systems/omniroute]] for the full live node table - See [[systems/omniroute]] for the full live node table
+9 -2
View File
@@ -3,8 +3,8 @@ title: ZeroClaw
type: system type: system
status: active status: active
created: 2026-08-15 created: 2026-08-15
updated: 2026-10-20 updated: 2026-11-11
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: high confidence: high
tags: [zeroclaw, ct333, daemon, agent] tags: [zeroclaw, ct333, daemon, agent]
sources: [skill-inspection, config-reference] sources: [skill-inspection, config-reference]
@@ -40,6 +40,13 @@ Autonomous AI daemon running on CT333. Handles independent agent operations with
> the live command line. > the live command line.
> `/root/.zeroclaw/` contains `config.toml` plus ~15 dated `config.toml.bak*` snapshots. > `/root/.zeroclaw/` contains `config.toml` plus ~15 dated `config.toml.bak*` snapshots.
> **Re-verified 2026-11-11 — running as documented.** `pct exec 333 -- pgrep -af zeroclaw` returns
> `142 /usr/local/bin/zeroclaw daemon --allow-degraded-security` (PID has rotated from 136, up
> since boot). `/usr/local/bin/zeroclaw` is the live path; `/root/.local/bin/zeroclaw` also exists.
> CT333 is named **`clawtest`** in `pct list` and `pct exec 333 -- hostname` returns `clawtest` —
> so the `clawtest` label on this page is correct, and the contrary claim on
> [[comparisons/hermes-vs-nanobot]] has been corrected.
## Key Paths ## Key Paths
| Item | Path | | Item | Path |
+16 -3
View File
@@ -3,8 +3,8 @@ title: Camoufox vs Chrome
type: comparison type: comparison
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-11-20
verified_on: 2026-10-20 verified_on: 2026-11-20
confidence: medium confidence: medium
tags: [comparison, browser] tags: [comparison, browser]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -35,7 +35,10 @@ Compare the two browser-automation options for Hermes's browser tool.
`browser.camofox:` → `adopt_existing_tab: true`, `managed_persistence: true`, `browser.camofox:` → `adopt_existing_tab: true`, `managed_persistence: true`,
`rewrite_loopback_urls: true`, `user_id: anthony`. `rewrite_loopback_urls: true`, `user_id: anthony`.
**Note `inactivity_timeout` is 450s, not the 120s recorded here** — that value changed. **Note `inactivity_timeout` is 450s, not the 120s recorded here** — that value changed.
`cloud_provider` is `steel` with `engine: camofox`; the two keys describe different layers. ~~`cloud_provider` is `steel` with `engine: camofox`; the two keys describe different layers.~~
**Superseded 2026-10-29:** `cloud_provider` now reads **`camofox`** (with `camera_provider: camofox`,
`engine: auto`, `cdp_url: ""`), and Steel is **fully removed** from the estate — the `browser-steel` plugin
is deleted and there is no Steel fallback path. → [[systems/browser-backend]]
- **Live processes (2026-10-20):** `cua-driver serve` (PID 3312393), `camoufox-ws-server.mjs`, - **Live processes (2026-10-20):** `cua-driver serve` (PID 3312393), `camoufox-ws-server.mjs`,
and a live `camoufox-bin` (FirefoxDev build) tree under `/home/hermes/.cache/camoufox/`. and a live `camoufox-bin` (FirefoxDev build) tree under `/home/hermes/.cache/camoufox/`.
Camofox's REST port **9377 is not listening** on CT460 — the Camofox browser service runs on Camofox's REST port **9377 is not listening** on CT460 — the Camofox browser service runs on
@@ -43,6 +46,16 @@ Compare the two browser-automation options for Hermes's browser tool.
- Runs headed under Xvfb rather than a true headless mode, per [[systems/headless-server]] — fluxbox provides the window manager, x11vnc/noVNC expose it for human viewing. - Runs headed under Xvfb rather than a true headless mode, per [[systems/headless-server]] — fluxbox provides the window manager, x11vnc/noVNC expose it for human viewing.
- `cua-driver` bridges Hermes's computer-use tool calls to the browser. - `cua-driver` bridges Hermes's computer-use tool calls to the browser.
> **Re-verified 2026-11-20 (stocktake).** Live config unchanged: `cloud_provider: camofox`,
> `camera_provider: camofox`, `engine: auto`, `headed: true`, `inactivity_timeout: 450`,
> `use_gateway: false`, `cdp_url: ""`, `camofox.managed_persistence: true`, `user_id: anthony`.
> `cua-driver` still **v0.28.2**; `cua-driver serve` + `camoufox-ws-server.mjs` + a `camoufox-bin`
> tree all running on CT460 (under Xvfb `:99` 1600×1200 + fluxbox).
> **Correction:** ~~Camofox's REST port 9377 is not listening on CT460 — the service runs on CT450~~ —
> `:9377` **is listening on CT460** (`node`, `/health` → `ok`, engine `camoufox`); the service is the
> local `systemd --user` unit `camofox-browser.service` (active), per [[systems/browser-backend]]'s
> 2026-10-13 correction. The CT450 attribution is dead. The ws-server also listens on `:9222` (CDP).
> No Chrome/Chromium process found in this probe either.
### Chrome (not found in use) ### Chrome (not found in use)
- No `google-chrome`, `chromium`, or CDP-only launch flags found in the sanitized config or running process list during this audit. - No `google-chrome`, `chromium`, or CDP-only launch flags found in the sanitized config or running process list during this audit.
- Included in the comparison because the page existed as a placeholder — if Chrome/CDP was evaluated and rejected, that reasoning isn't captured anywhere; worth a [[decisions/index]] entry if anyone remembers why Camoufox was chosen over stock Chrome. - Included in the comparison because the page existed as a placeholder — if Chrome/CDP was evaluated and rejected, that reasoning isn't captured anywhere; worth a [[decisions/index]] entry if anyone remembers why Camoufox was chosen over stock Chrome.
+10 -4
View File
@@ -3,8 +3,8 @@ title: Hermes vs Nanobot
type: comparison type: comparison
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-11-11
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: high confidence: high
tags: [comparison, hermes, nanobot] tags: [comparison, hermes, nanobot]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -29,8 +29,14 @@ Compare Hermes (this agent) with Nanobot, another agent process in the homelab,
> >
> **One caveat worth adding:** CT333 now runs **both** Nanobot *and* ZeroClaw (the latter verified > **One caveat worth adding:** CT333 now runs **both** Nanobot *and* ZeroClaw (the latter verified
> live on 2026-10-20 — see [[agents/zeroclaw]]). The page's framing of CT333 as the Nanobot host > live on 2026-10-20 — see [[agents/zeroclaw]]). The page's framing of CT333 as the Nanobot host
> is not wrong, but the host is no longer single-purpose. Note also that **CT333 is `zeroclaw` in > is not wrong, but the host is no longer single-purpose.
> the current Proxmox naming** — the wiki uses the older "clawtest" label for the same LXC. >
> **Correction 2026-11-11: the "CT333 is `zeroclaw` in Proxmox naming" claim was itself wrong.**
> Live `pct list` on the PVE host (`100.96.100.82`) still reports CT333's name as **`clawtest`**
> and `pct exec 333 -- hostname` returns **`clawtest`** too — the wiki's original label is correct
> on both counts. `zeroclaw` is only the **process/daemon** that runs inside CT333, alongside
> `nanobot`. Re-verified live the same day: the peer is on Tailscale at `100.122.189.81 clawtest`
> and `http://nanobot.martinwa.org` still answers **301**.
## At a glance ## At a glance
| | Hermes (ct460) | Nanobot (ct333 "clawtest") | | | Hermes (ct460) | Nanobot (ct333 "clawtest") |
+10 -2
View File
@@ -3,8 +3,8 @@ title: Memory Backends
type: comparison type: comparison
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-11-11
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: high confidence: high
tags: [comparison, memory] tags: [comparison, memory]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -43,6 +43,14 @@ The three archived scripts (`hindsight-watchdog.py.archived`, `hindsight_migrati
> in cron contexts, which is why scheduled jobs produce no working memories. See > in cron contexts, which is why scheduled jobs produce no working memories. See
> [[concepts/agent-memory]] and [[systems/mnemosyne]]. > [[concepts/agent-memory]] and [[systems/mnemosyne]].
> **Re-verified 2026-11-11 — still Mnemosyne only.** `memory.provider: mnemosyne`,
> `memory_enabled: true`, `flush_min_turns: 6`, `memory_char_limit: 5000`,
> `user_char_limit: 3000`, `nudge_interval: 10`, `user_profile_enabled: true`; Mnemosyne sub-block
> `auto_sleep: true`, `default_scope: session`, `vector_type: int8`, `sleep_threshold: 50`,
> `profile_isolation: true`, `shared_surface_path: data/shared/mnemosyne.db`,
> `shared_surface_read: true`. No `openviking` key anywhere in `config.yaml`; no migration
> performed. The OpenViking NOT-adopted verdict stands.
## Related ## Related
- [[systems/mnemosyne]] - [[systems/mnemosyne]]
- [[systems/memory-backend]] - [[systems/memory-backend]]
+12 -2
View File
@@ -3,8 +3,8 @@ title: Model Routers
type: comparison type: comparison
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-11-11
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: high confidence: high
tags: [comparison, routing] tags: [comparison, routing]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -51,6 +51,16 @@ Compare LiteLLM, OmniRoute, and OpenRouter as model-routing layers for Hermes.
> Any runbook that probes these endpoints with a bare `curl` will see 401 and may wrongly conclude > Any runbook that probes these endpoints with a bare `curl` will see 401 and may wrongly conclude
> the provider is down — it is not. > the provider is down — it is not.
> **Re-verified 2026-11-11 — the corrections recorded here all still hold.** The live `model:` block
> is OmniRoute-primary (`custom:omni-101` / `smart-route` / `http://100.78.63.20:20128`), and
> LiteLLM is the **tail** of the six-entry fallback chain (`or-inclusionai_ling-2_6-1t` via
> `custom` → `http://litellm:4000/v1`). The five `custom_providers` and their base URLs are exactly
> as listed: `litellm` `http://litellm:4000/v1`, `guacano` `http://ollama:11000/v1`,
> `Omniroute:20128` `http://omniroute:20128/v1`, `command-code`
> `https://api.commandcode.ai/provider/v1`, `omni-101` `http://100.78.63.20:20128`. OpenRouter is
> still absent from `custom_providers`. **Not re-probed this pass:** the live `/v1/models` counts
> (1869 / 6457) and their 2026-10-20 timestamps stand as recorded — treat them as unverified.
## Detail ## Detail
### LiteLLM — fallback tail + a `custom_providers` entry ### LiteLLM — fallback tail + a `custom_providers` entry
+32 -16
View File
@@ -3,8 +3,8 @@ title: Steel Browser vs Camoufox
type: comparison type: comparison
status: active status: active
created: 2026-07-29 created: 2026-07-29
updated: 2026-10-13 updated: 2026-11-20
verified_on: 2026-10-13 verified_on: 2026-11-20
confidence: high confidence: high
tags: [comparison, browser, steel, camofox, decision] tags: [comparison, browser, steel, camofox, decision]
sources: [live-probe, session-history] sources: [live-probe, session-history]
@@ -13,20 +13,20 @@ sources: [live-probe, session-history]
# Steel Browser vs Camoufox # Steel Browser vs Camoufox
## Purpose ## Purpose
Compare the two browser-automation backends available to Hermes, including the Jul 29–30 experiment and the decision to revert. Compare the two browser-automation backends Hermes has used, including the Jul 29–30 experiment and the decision to revert. **Camofox is now the only backend available — Steel was fully removed on 2026-10-29.**
## At a glance ## At a glance
| | Steel Browser (CT201) | Camofox (systemd --user on CT460) | | | Steel Browser (CT201) | Camofox (systemd --user on CT460) |
|---|---|---| |---|---|---|
| **In use** | Disabled since 2026-07-30 | **Yes** — primary backend | | **In use** | **Removed 2026-10-29** (primary 2026-07-29 → 07-30) | **Yes** — the only backend |
| **Engine** | Chromium 140 (Playwright) | Camoufox (Firefox-based) | | **Engine** | Chromium 140 (Playwright) | Camoufox (Firefox-based) |
| **Connection** | CDP (`ws://` via port 9223) | REST API + CDP WebSocket, both `localhost:9377` | | **Connection** | CDP (`ws://` via port 9223) | REST API + CDP WebSocket, both `localhost:9377` |
| **API** | REST at `:3000` — create/destroy/query sessions | Camoufox REST + CDP WebSocket | | **API** | REST at `:3000` — create/destroy/query sessions | Camoufox REST + CDP WebSocket |
| **Headless** | Native headless | Xvfb + VNC (headed, anti-fingerprint) | | **Headless** | Native headless | Xvfb + VNC (headed, anti-fingerprint) |
| **Container** | Docker on CT201 | `systemd --user` unit on the Hermes host (CT460) | | **Container** | Docker on CT201 — **removed 2026-10-29** | `systemd --user` unit on the Hermes host (CT460) |
| **Stealth** | Standard Chromium | Anti-detection fingerprinting | | **Stealth** | Standard Chromium | Anti-detection fingerprinting |
| **Session viewer** | Built-in UI at `/steel` | VNC at `/vnc-camofox` | | **Session viewer** | Built-in UI at `/steel` — **route deleted** | VNC at `/vnc-camofox` |
| **Plugin** | `browser-steel` plugin (disabled) | Native Hermes `browser.cloud_provider: camofox` | | **Plugin** | `browser-steel` — **deleted 2026-10-29** | Native Hermes `browser.cloud_provider: camofox` |
## The Steel Experiment (Jul 29–30) ## The Steel Experiment (Jul 29–30)
@@ -39,7 +39,7 @@ Compare the two browser-automation backends available to Hermes, including the J
1. **Jul 29:** Steel deployed on CT201 via Docker Compose, Chromium 140. Hermes configured with `browser.cloud_provider: steel`, `browser-steel` plugin enabled. 1. **Jul 29:** Steel deployed on CT201 via Docker Compose, Chromium 140. Hermes configured with `browser.cloud_provider: steel`, `browser-steel` plugin enabled.
2. **Jul 29 (later):** CDP connectivity issues surfaced — Hermes was reaching `0.0.0.0` from the plugin's internal CDP URL. Required `_normalize_cdp_url()` patch in the Steel plugin to remap to `100.96.244.39`. 2. **Jul 29 (later):** CDP connectivity issues surfaced — Hermes was reaching `0.0.0.0` from the plugin's internal CDP URL. Required `_normalize_cdp_url()` patch in the Steel plugin to remap to `100.96.244.39`.
3. **Jul 30:** DeepSeek V4 Flash 0731 went live and rejected tool names that didn't match `^[a-zA-Z0-9_-]+$`. The Steel plugin's tools (`steel_scrape`, `steel_session_options`) were fine, but the route through OmniRoute → Steel → CDP was unreliable with empty-stream errors. 3. **Jul 30:** DeepSeek V4 Flash 0731 went live and rejected tool names that didn't match `^[a-zA-Z0-9_-]+$`. The Steel plugin's tools (`steel_scrape`, `steel_session_options`) were fine, but the route through OmniRoute → Steel → CDP was unreliable with empty-stream errors.
4. **Jul 30 (evening):** Decision to revert — Camoufox re-enabled, browser-steel plugin disabled, `CAMOFOX_URL` and `CAMOFOX_API_KEY` uncommented in `.env`. Steel left running on CT201 but Hermes no longer routes through it. 4. **Jul 30 (evening):** Decision to revert — Camoufox re-enabled, browser-steel plugin disabled, `CAMOFOX_URL` and `CAMOFOX_API_KEY` uncommented in `.env`. Steel left running on CT201 but Hermes no longer routes through it. *(Superseded 2026-10-29 — Steel was removed entirely.)*
### What broke ### What broke
- CDP URL normalization required a plugin patch to handle Docker's internal networking - CDP URL normalization required a plugin patch to handle Docker's internal networking
@@ -48,7 +48,8 @@ Compare the two browser-automation backends available to Hermes, including the J
### Lesson learned ### Lesson learned
Steel is a capable backend but adds a network hop and another plugin layer. Camofox runs locally on the Steel is a capable backend but adds a network hop and another plugin layer. Camofox runs locally on the
Hermes host (CT460) with fewer failure modes. The `browser-steel` plugin still exists (disabled) if needed in future. Hermes host (CT460) with fewer failure modes. Steel was **fully removed on 2026-10-29** — plugin deleted, and
the CT201 and CT333 deployments torn down — so returning to Steel means redeploying it, not flipping a flag.
## Detail ## Detail
@@ -60,14 +61,29 @@ Hermes host (CT460) with fewer failure modes. The `browser-steel` plugin still e
- **Auth:** Already authenticated into services (Uber Eats, LinkedIn, etc.) — no cookie files needed - **Auth:** Already authenticated into services (Uber Eats, LinkedIn, etc.) — no cookie files needed
- **Limitation:** Headed via Xvfb, consumes display resources - **Limitation:** Headed via Xvfb, consumes display resources
### Steel Browser (disabled but available) ### Steel Browser (removed 2026-10-29 — no longer available)
- **Deployed:** 2026-07-29 on CT201, Docker, custom `steel-browser-api:fixed` image - **Deployed:** 2026-07-29 on CT201, Docker, custom `steel-browser-api:fixed` image *(historical)*
- **Status:** Still running on CT201, Hermes not connected - **Status:** **FULLY REMOVED on 2026-10-29** — not merely offline. On CT201 the containers
- **CDP endpoint:** `http://100.96.244.39:9223` `steel-browser-api-1` / `steel-browser-ui-1`, the `steel-network` Docker network, the images
- **REST API:** `http://100.96.244.39:3000` (`steel-browser-api:fixed`, `ghcr.io/steel-dev/steel-browser-api:latest`,
- **UI:** `https://runtipi.kangaroo-eel.ts.net/steel` `ghcr.io/steel-dev/steel-browser-ui:latest`) and the project dir `/root/steel-browser` were all deleted
- **Plugin:** `browser-steel` at `/home/hermes/.hermes/plugins/browser-steel/` — disabled, `plugin_steel` toolset flagged `requires_health=steel` (~6 GB reclaimed). The Tailscale Serve routes `/steel` and `/steel-api` were removed, so the former CDP
`:9223`, REST `:3000` and UI `/steel` addresses are all retired — nothing listens on 3000, 5173 or 9223.
- **CT333 copy:** a second, separate Steel install (`steel-browser`, `ghcr.io/steel-dev/steel-browser:latest`)
was also removed with its image. It had been orphaned — no requests since 2026-10-21, and no reference from
Zeroclaw, Nanobot or any Serve route.
- **Plugin:** **deleted 2026-10-29** — `~/.hermes/plugins/browser-steel/` is gone and no `browser-steel` key
remains in `plugins.disabled` / `plugins.entries`, so the `plugin_steel` toolset cannot be selected.
- **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, plus
`/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io.
> **Re-verified 2026-11-20 (stocktake).** Steel-removal claims re-probed live: CT201 has **no** steel
> container or image, nothing on :3000/:5173/:9223, Serve routes are only `/` → 8085 and `/mgr` → 8084
> on :443 **and** :8443; `/root/_archived-steel/` holds `docker-compose.yml`, `Dockerfile.api`,
> `steel-cache-archive-20261029.tgz`. CT333 likewise has no steel container/image and nothing on :3000.
> `~/.hermes/plugins/browser-steel/` absent. Camofox healthy on CT460: `:9377/health` → `ok`
> (idle — `browserConnected:false` at probe time, no active session), unit active, config
> `cloud_provider: camofox`. All removal claims **hold**.
## Related ## Related
- [[systems/browser-backend]] - [[systems/browser-backend]]
- [[concepts/browser-automation]] - [[concepts/browser-automation]]
+12 -2
View File
@@ -3,8 +3,8 @@ title: Agent Memory
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-11-11
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: high confidence: high
tags: [concept, memory] tags: [concept, memory]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -37,6 +37,16 @@ How an agent retains facts/context beyond a single conversation turn or session
> - `profile_isolation: true` — memories do not leak across Hermes profiles, which matters now that > - `profile_isolation: true` — memories do not leak across Hermes profiles, which matters now that
> the `ops` profile has been deleted (see [[runbooks/multiplexer-setup]]). > the `ops` profile has been deleted (see [[runbooks/multiplexer-setup]]).
> **Full `memory:` block re-read 2026-11-11 — every value unchanged.** `memory_enabled: true`,
> `provider: mnemosyne`, `memory_char_limit: 5000`, `user_profile_enabled: true`,
> `user_char_limit: 3000`, `flush_min_turns: 6`, `nudge_interval: 10`; Mnemosyne sub-block
> `auto_sleep: true`, `default_scope: session`, `vector_type: int8`, `sleep_threshold: 50`,
> `profile_isolation: true`, `shared_surface_path: data/shared/mnemosyne.db`,
> `shared_surface_read: true`, `sync_roles: [user]`, `ignore_patterns: []`,
> `skip_contexts: [cron, flush, subagent, background, skill_loop]`,
> `reflect: {disabled_for_cron: true, max_calls_per_session: 3}`. The cron-skip is still why
> scheduled jobs like this stocktake leave no Mnemosyne trace.
## Model ## Model
- Per-turn conversation content accumulates up to the char limit, then flushes to the Mnemosyne store every 6 turns (rather than every turn) — a batching tradeoff between write frequency and losing very recent context if the process dies mid-batch. - Per-turn conversation content accumulates up to the char limit, then flushes to the Mnemosyne store every 6 turns (rather than every turn) — a batching tradeoff between write frequency and losing very recent context if the process dies mid-batch.
- `scope: session` means recall defaults to the current session rather than a global cross-session pool, unless explicitly widened. - `scope: session` means recall defaults to the current session rather than a global cross-session pool, unless explicitly widened.
+7 -3
View File
@@ -3,8 +3,8 @@ title: Browser Automation
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-13 updated: 2026-11-20
verified_on: 2026-10-13 verified_on: 2026-11-20
confidence: high confidence: high
tags: [concept, browser, camofox, steel] tags: [concept, browser, camofox, steel]
sources: [live-probe, session-history] sources: [live-probe, session-history]
@@ -31,8 +31,12 @@ Steel Browser (Chromium 140 on CT201) was briefly the primary backend. It offere
- Empty-stream errors from the `opencode-go/deepseek-v4-flash` provider path - Empty-stream errors from the `opencode-go/deepseek-v4-flash` provider path
- DeepSeek 0731's stricter tool name validation affected all providers - DeepSeek 0731's stricter tool name validation affected all providers
**Lesson:** A local browser backend (same host, no network hop) has fewer failure modes than a remote Docker container. The `browser-steel` plugin is still present but disabled. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]]. **Lesson:** A local browser backend (same host, no network hop) has fewer failure modes than a remote Docker container. The `browser-steel` plugin **was deleted on 2026-10-29** and the Steel deployments on CT201 and CT333 were torn down, so there is no Steel fallback left to fall back to — Camofox is the only backend. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]].
> **Re-verified 2026-11-20 (stocktake).** Camofox service active on CT460 (`systemctl --user`),
> `:9377/health` → `ok`; Xvfb `:99` at 1600×1200×24 with fluxbox, websockify on 127.0.0.1:6080.
> Config `cloud_provider: camofox` / `camera_provider: camofox` confirmed. No Steel anywhere
> (CT201/CT333 containers, images, routes, plugin all re-probed absent).
## Related ## Related
- [[systems/browser-backend]] - [[systems/browser-backend]]
- [[comparisons/steel-vs-camoufox]] - [[comparisons/steel-vs-camoufox]]
+31 -7
View File
@@ -3,8 +3,8 @@ title: Model Routing
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-11-11
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: high confidence: high
tags: [routing, model, inference] tags: [routing, model, inference]
sources: [raw/configs/hermes-config-sanitized.txt, current-profile.md] sources: [raw/configs/hermes-config-sanitized.txt, current-profile.md]
@@ -16,18 +16,42 @@ sources: [raw/configs/hermes-config-sanitized.txt, current-profile.md]
How Hermes selects providers, applies combos, and enforces context limits. How Hermes selects providers, applies combos, and enforces context limits.
## Current Flow ## Current Flow
- Main path: `custom:litellm` → `http://litellm:4000/v1` - Main path: `custom:omni-101` / `smart-route` → `http://100.78.63.20:20128` (the OmniRoute node
- Default model: `freellmapi-auto` `omniroute-1`), `api_mode: chat_completions`
- OmniRoute used for the **default chat path** (`custom:omni-101` / `smart-route`, since 2026-10-24 19:06) and for **all eight auxiliary slots** (`approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`, `title_generation`, `vision` — all `no-think/freeapi/auto` as of 2026-10-25 19:13) - OmniRoute used for the **default chat path** (`custom:omni-101` / `smart-route`, since 2026-10-24 19:06) and for **nine auxiliary slots** (`approval`, `compression`, `curator`, `mcp`, `profile_describer`, `pulse_memory`, `skills_hub`, `title_generation`, `vision` — eight of them on `no-think/freeapi/auto`, plus `pulse_memory`; see the 2026-11-11 note below)
> **Auxiliary slots re-read 2026-11-11 — the set has grown from eight to nine.** Eight slots
> (`approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`,
> `title_generation`, `vision`) are unchanged: `provider: custom:omni-101`, `model:
> no-think/freeapi/auto`, each with a two-entry fallback chain
> (`no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, both on `custom:omni-101`).
> **New: `pulse_memory`** — `provider: nous`, `model: anthropic/claude-haiku-4.5`, and it is the
> slot behind the live 404s recorded above: nous is out of credits for that **paid** model, so
> `pulse_memory` is failing right now (3 errors on 2026-11-10 18:02 and 19:19). Also live:
> `auxiliary.free_only: true`.
> **Live `model:` block re-read 2026-10-26 — the default has moved TWICE.** > **Live `model:` block re-read 2026-10-26 — the default has moved TWICE.**
> `model.default` is **`smart-route`** on provider **`custom:omni-101`** > `model.default` is **`smart-route`** on provider **`custom:omni-101`**
> (`http://100.78.63.20:20128`), `api_mode: chat_completions`. It moved to `nous` / > (`http://100.78.63.20:20128`), `api_mode: chat_completions`. It moved to `nous` /
> `meituan/longcat-2.5-preview:free` on 2026-10-20 22:01 and **back** on **2026-10-24 19:06:10**. > `meituan/longcat-2.5-preview:free` on 2026-10-20 22:01 and **back** on **2026-10-24 19:06:10**.
> A six-provider top-level `fallback_providers` chain is live again, headed by the **dead `nous` > A six-provider top-level `fallback_providers` chain is live again, headed by the **dead `nous`**
> credential** (`Invalid refresh token` since 2026-10-22). > credential (`Invalid refresh token` since 2026-10-22).
> → [[systems/current-profile]] · [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] > → [[systems/current-profile]] · [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
> >
> **Re-read 2026-11-11 — the model block is unchanged, but the `nous` failure signature has moved
> on.** `model.default` is still `smart-route` on `custom:omni-101`
> (`base_url: http://100.78.63.20:20128`, `api_mode: chat_completions`,
> `persist_switch_by_default: false`) and the six-entry `fallback_providers` chain is unchanged in
> membership and order (`nous` → `opencode-zen` → `custom:guacano` → `custom:litellm` →
> `kilocode` → `custom`). The **refresh-token failure is gone from the logs**: `~/.hermes/.env`
> has no `NOUS_API_KEY` and `~/.hermes/auth.json` is live, and no 401/refresh-token error appears
> in `~/.hermes/logs/errors.log`. What *does* appear (3×, most recent **2026-11-10 19:19:33**) is
> **404 `insufficient_credits_for_paid_model`** for `anthropic/claude-haiku-4.5` — a **paid**
> model. So `nous` is no longer dead-at-auth; it is **out of credits for paid models**. Free nous
> models (position 1 is `meituan/longcat-2.5-preview:free`) have not been observed failing. The
> slot actually hit by that paid model is **`auxiliary.pulse_memory`** — see the auxiliary note
> below, which is currently a broken slot.
>
> **The five `custom_providers` and their live base URLs:** > **The five `custom_providers` and their live base URLs:**
> >
+8 -2
View File
@@ -3,8 +3,8 @@ title: Persistent Shells
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-11-11
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: high confidence: high
tags: [concept, terminal] tags: [concept, terminal]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -31,6 +31,12 @@ A background shell session that survives across separate Hermes turns/requests,
> The container-* keys are inert while `backend: local` — the shell is not containerised on this > The container-* keys are inert while `backend: local` — the shell is not containerised on this
> box, so `container_memory`/`container_disk` describe a path that isn't taken. > box, so `container_memory`/`container_disk` describe a path that isn't taken.
> **`terminal:` block re-read 2026-11-11 — unchanged.** `backend: local`,
> `persistent_shell: true`, `timeout: 180`, `home_mode: auto`, `container_cpu: 1`,
> `container_memory: 5120`, `container_disk: 51200`, `container_persistent: true`,
> `lifetime_seconds: 300`, `docker_mount_cwd_to_workspace: false`, `docker_run_as_host_user: true`,
> `cwd: /home/hermes/workspace/`. The container-* keys are still inert while `backend: local`.
## Why it matters here ## Why it matters here
Enables multi-step CLI workflows (e.g. a long `git` operation, a build, a watch process) to be resumed across turns without Hermes re-establishing context each time. The tradeoff is a shell can be left in a bad state (stuck process, wrong directory) that silently affects the *next* turn — see [[runbooks/troubleshoot-tool-loop]] for the related tool-loop guardrails that catch some of this. Enables multi-step CLI workflows (e.g. a long `git` operation, a build, a watch process) to be resumed across turns without Hermes re-establishing context each time. The tradeoff is a shell can be left in a bad state (stuck process, wrong directory) that silently affects the *next* turn — see [[runbooks/troubleshoot-tool-loop]] for the related tool-loop guardrails that catch some of this.
+7 -2
View File
@@ -3,8 +3,8 @@ title: Tool Calling
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-11-11
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: high confidence: high
tags: [concept, tool-calling] tags: [concept, tool-calling]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -49,6 +49,11 @@ This is the mechanism behind [[runbooks/troubleshoot-tool-loop]] — that runboo
> (which watches *tool* calls) never fired. Two separate loop-detection mechanisms exist; do not > (which watches *tool* calls) never fired. Two separate loop-detection mechanisms exist; do not
> assume one covers the other. > assume one covers the other.
> **`tool_loop_guardrails` re-verified 2026-11-11 — unchanged.** `warnings_enabled: true`,
> `hard_stop_enabled: false`, `warn_after {exact_failure: 2, same_tool_failure: 3,
> idempotent_no_progress: 2}`, `hard_stop_after {exact_failure: 5, same_tool_failure: 8,
> idempotent_no_progress: 5}`. A genuinely stuck tool loop still only ever produces a warning.
## Related ## Related
- [[concepts/persistent-shells]] - [[concepts/persistent-shells]]
- [[runbooks/troubleshoot-tool-loop]] - [[runbooks/troubleshoot-tool-loop]]
+622 -9
View File
@@ -3,18 +3,631 @@ title: Current State
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-25 updated: 2026-11-20
verified_on: 2026-10-25 verified_on: 2026-11-20
review_after: 2026-11-01 review_after: 2026-11-21
confidence: high confidence: high
tags: [system] tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
--- ---
# Current State # Current State
Updated: 2026-10-25 Updated: 2026-11-20
## Recent Changes (2026-10-25, nightly audit; live probes 2026-10-25 00:1x–00:5x AWST) ## Recent Changes (2026-11-17 → 2026-11-20, three-window audit; live probes 2026-11-20 00:0x – 00:5x AWST)
**The estate's memory freeze ended by itself, its two alarm paths are still red, and the audits that would have reported either had silently died twice.**
- **Capture stall: over, unassisted.** `working_memory` **1130 → 1131**, `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST** — a **105 h 52 m** break that ended with no config change, restart or repair. ⚠ **And the "capture dead" reading was wrong:** `state.db` holds only **three** non-cron sessions since 11-14 00:00 — two `desktop` on **11-14 08:35 / 08:52** (both *before* the 09:33 boundary) and the **11-18 19:18** `desktop` whose turn *is* the captured row — so **every interactive session in the window was captured**. `mnemosyne_freshness_gate.py` reads the age of the newest non-`cron`/`subagent` message, so **silence and failure read identically**; that ambiguity is the real defect. The monitor is still failing on it (`failure_streak: 2`; slots 11-18 00:01 and 12:01, 11-19 12:01, 11-20 00:01 — 11-19 00:01 alone passed). → [[incidents/2026-11-14-mnemosyne-capture-stall]], [[systems/mnemosyne]]
- **The nightly audit missed two nights.** `cbfe434a3b5e` failed on **11-18** and **11-19** on a chained-provider credit/auth fault; **both deliveries `suppressed`**, and its incident row is still `alerted` with **`alerted_at: None`** → [[incidents/2026-11-18-wiki-audit-credit-outage]]. **This run covers 11-17, 11-18 and 11-19.**
- **The watchdog is in a threshold storm.** `96eb95ce12f2` failed **115 consecutive** 15-minute slots from **2026-11-18 20:01:06** to **2026-11-20 00:31:16** on **59 unresolved conflicts vs a threshold of 50** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]
- **Cron ledger.** 11-17 **173 rows (171 `completed` / 2 `failed`)**; 11-18 **172 (153/19)**; 11-19 **176 (78/98)** — the worst day in the retained window; 11-20 so far **8 (3 / 4 / 1 `running`)**. Deliveries since 11-18 **24 — 23 `delivered`** (9 `for_failure: 1`) / **1 `failed`** (content, Telegram timeout `2026-11-19 08:34:13`). Fleet **29 (26 enabled / 3 disabled)**, a twelfth consecutive window; non-clean `last_status` **1 → 4**; unclosed `cron_incidents` **5 → 4**. The **03:00 stocktake ran `completed` on all three days** (11 m 35 s / 23 m 07 s / 24 m 38 s).
- **Mnemosyne consolidation still zero-item:** newest `consolidation_log` row `2026-11-19T16:20:14+00:00`, `items_consolidated: 0`, **199 of the last 200 cycles zero-item** (was 120 on 11-17). Zero-item-streak threshold still **proposed, not applied**.
- **Pages touched:** [[incidents/2026-11-18-wiki-audit-credit-outage]] (new), [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] (new), [[incidents/2026-11-14-mnemosyne-capture-stall]] (new re-read + corrected title), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[index]], `log.md` — frontmatter dates → **2026-11-20**.
**Previous window:**
## Recent Changes (2026-11-16, nightly audit; live probes 2026-11-16 23:59 – 2026-11-17 00:27 AWST)
**The memory pipeline is still dead at 62 h, and the 11-15 record needed a correction: a plugin
load-timeout storm nobody had logged.**
- **Capture stall, third read — unchanged in every respect but the counters.** `working_memory` **1130**
/ `episodic_memory` **1249**, newest write `2026-11-14 09:32:59` AWST → **62 h 54 m**. Unclosed
`alerted` incidents **3 → 5** (143 rows total); monitor `failure_streak` **5**; consolidator zero-item
streak **120** cycles. No cause, no fix, `status: active` →
[[incidents/2026-11-14-mnemosyne-capture-stall]]
- ⚠ **New page — [[incidents/2026-11-15-plugin-load-timeout-storm]].** 11-15 carried **52** failed plugin
loads and **519** discarded registrations, `conduit_push` alone losing **474** (its tools and hooks were
absent for those processes), plus **11** `pre_tool_call` and **2** `pre_llm_call` hook timeouts and
**10** refused tool calls. The 11-15 audit recorded **none** of it. **Zero on 11-16 and 11-17** — it
cleared with no fix applied. The companion
[[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] gained a Recurrence 2 section; load average
is now **explicitly not** a sufficient predictor (11-17 read 6.19 vs 12.08 on 11-14, no recurrence, and
a *larger* `state.db`).
- **Cron fleet, 11th window, flat at 29 jobs** (26 enabled / 3 disabled). 11-16 ledger **175 rows —
173 `completed` / 2 `failed`**, both the health monitor; deliveries **11/11 `delivered`**. 11-17 so far:
5 / 1 / 1. The 11-15 watchdog deps-guard 600 s timeout **did not recur**.
- **Quiet where it matters, growing where it doesn't.** 11-16 ran **16 cron sessions, zero interactive** —
nothing human-side to document. `state.db` **2,093,998,080 B (2.09 GB)**, WAL **40 MB**, `/` at **88 %**,
load **6.19 / 6.64 / 7.48**, uptime 6 d 2 h (host up since 2026-11-10).
- **Freshness:** 1 stale page by design — `reports/communities-wa-social-media-audit-2026` (23 d at
11-17); it is a static deliverable with `review_after` 2026-11-26, so it was left unbumped.
## Recent Changes (2026-11-15, nightly audit; live probes 2026-11-15 23:59 – 2026-11-16 00:2x AWST)
- ⚠ **Memory capture is still dead — 38 h 46 m — and the freeze is total, not degraded.**
[[incidents/2026-11-14-mnemosyne-capture-stall]] re-read: `working_memory` **1130** /
`episodic_memory` **1249**, *the same two counts as every earlier read of the stall*; newest
`working_memory` write still the `09:32:59 AWST` (01:32:59 UTC) `[MODEL_REFRESH_PROPOSAL]` maintenance
row, newest *user-turn* capture `2026-11-14 09:04:19 AWST`. Root cause still not established.
- **The monitor's `≈4.64 h` figure is not a duration — decoded against the gate's own source.**
`mnemosyne_freshness_gate.py`'s `newest_user_message()` takes the newest `role='user'` message whose
session source is not `cron`/`subagent` and prints *its* age. On this box those are the **Pulse**
turns on `source: desktop` (~6 h cadence), and the `0 */12` slots sit ~4.6 h after one — which is why
all three alerts read alike. **Do not read it as the stall length.**
- **`cron_incidents` 1 → 3 unclosed** (138 → **141** rows) — all `ea79a2107e36`, all
`failure_type: auth`, all `alerted`, each `first_seen_at == last_seen_at`. The signature embeds the
elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), so **one durable
fault mints a fresh unclosed row every slot** and nothing escalates. **Fix proposed, not applied.**
- **The consolidator runs and does nothing — by design.** Latest `consolidation_log` row **4696** at
`2026-11-15T16:14:14.934306+00:00` (= 00:14:14 AWST), `items_consolidated: 0`, `summary_preview`
`No-op consolidation cycle — timestamp updated for watchdog`; **~74 consecutive zero-item cycles**
from `2026-11-14T02:04:42+00:00`. It bumps its own timestamp, so an age-only gate reads "fresh".
- **Fleet: 2026-11-15 — 168 rows, 165 `completed` / 3 `failed`**, deliveries **9 / 9 `delivered`**
(3 `for_failure`). The three: the Health Monitor at `00:00:53.928` and `12:01:24.719`, plus — **new
failure mode** — the Watchdog `96eb95ce12f2` at `11:31:20.699`, `finished_at 11:49:10.534610` =
**17 m 49.8 s**, `runtime deps guard … timed out after 600 seconds`; the `11:45` slot was **skipped**,
`12:01:25` completed. **2026-11-14 settled 167 / 167 `completed` / 0 `failed`.**
- **Fleet 29 jobs (26 enabled / 3 disabled)** — a **tenth consecutive window unchanged**; only non-clean
row is `ea79a2107e36` (`error`, `failure_streak` **3**).
- **No interactive work on 11-15.** All **9** sessions started that day are `source: cron` — nothing
human-side for the wiki to have missed.
- Freshness (03:00 stocktake, `raw/stocktake/freshness-20261115.txt`): **0 stale / 133 fresh / 3
no-date of 136** pages.
## Recent Changes (2026-11-14, nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST)
- **A third TTS stack landed on CT200** — new page [[systems/phonon-tts]]: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model `gradium/phonon-40m-en`), `phonon-demo` (`:8882`, WASM UI on `nginx:alpine`), `ptts-ws-server` (`:8883`, WebSocket, locally-built image). CT200 went **61 → 65 running / 68 → 72 total**; verified live from CT460 (health 200, models list, MP3 returned). ⚠ Everything after **09:03 AWST** on 11-14 is **not attributable to any retained transcript** — see the page's provenance gap before crediting it to a session.
- ⚠ **Open incident: Mnemosyne capture stalled** since **2026-11-14 09:32:59 AWST** (~15 h at this read) — new page [[incidents/2026-11-14-mnemosyne-capture-stall]]. The health monitor escalated only at **2026-11-15 00:03:35** (exit 1, `ea79a2_601037516abd`, `alerted`, unclosed), after logging the same hole as ℹ️ at 12:00 AWST. Root cause not established; the known stringified-config killer is ruled out.
- **Fleet clean apart from the monitor's own alert:** 2026-11-14 settled **167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**; `cron_incidents` **0 → 1 unclosed** (the monitor row above). Fleet **29 jobs (26 enabled / 3 disabled)**, **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, streak 1).
- [[infrastructure/docker-services]] CT200 inventory re-verified and updated — **64 non-seeder running / 72 total** (was 61 / 68); the delta is exactly the three TTS containers, no removals.
## Recent Changes (2026-11-13, nightly audit; live probes 2026-11-13 23:59 – 2026-11-14 00:2x AWST)
- ✅ **2026-11-13 was the third consecutive zero-`failed` cron day: 174 rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit, `cbfe434a3b5e`, started 23:59:41). Trend: 11-11 **172 / 172**, 11-12 **176 / 176**, 11-13 **174 / 173**. **Deliveries 11-13: 8 rows, 8 / 8 `delivered`, 0 `for_failure`** (11-12: 9 / 9). Fleet **29 jobs (26 enabled / 3 disabled)** — an **eighth consecutive window** unchanged — and **every job reads `last_status: ok` with `failure_streak` 0, so the fleet has no non-clean row at all** for the first time in weeks. `cron_incidents` **0 unclosed**. → [[systems/scheduled-tasks]]
- ⚠ **The `pre_tool_call` hook timeout is back, and the raised budget did not hold it.** `_pre_tool_call timed out after 120s` at **2026-11-14 00:05:40**, then **10** `skipped after previous timeout or while still running` lines through **00:06:38** — refusing this audit's own first tool calls (first-hand: `skill_view` and `terminal` both returned the plugin timeout) for ~1 minute before dispatch recovered unaided. `errors.log` by day: 11-12 **0 / 0**, 11-13 **0 / 0**, 11-14 **1 / 10**; it is the **first `pre_tool_call` timeout in the retained log since the 2026-11-08 cascade** (whose own lines have since rotated out — `errors.log` now starts **2026-11-09 13:18:39**). **No cron run failed** — 11-13 settled `failed`-free. **The variable is load, now ~2× the level recorded at diagnosis (`load average 12.08 / 13.02 / 12.33` on 2 vCPU vs 5.89).** The 30 → 120 s fix is **proven insufficient**; the incident is **reopened, `resolved` → `active`**. → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
- ⚠ **`state.db` is now 2.04 GB** (**2,036,908,032 B**; **2,977 sessions / 207,577 messages**) — up from **1.89 GB / 2,901 / 197,682** on 2026-11-09 — and it is the load source behind the reopened hook fault above. **Root filesystem 88 % used.** `state.db` **checkpoint/retention remains undecided** and is now the estate's most urgent open item; re-raising the hook budget again would only mask the fault. Still `journal_mode=wal`, `wal_autocheckpoint=2048`. → [[systems/hermes-agent]], [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
- ✅ **The Uber Eats Monthly Spend Report was fixed end-to-end on 2026-11-13**, clearing the fleet's last non-clean row. Two stacked root causes: (1) the job's **runtime prompt was 88 k chars** because it carried two heavy browser skills, so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest (*"remaining 4 in the report"* plus forbidden run commentary); (2) the `delivery_failed` label was a **stale `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart — which happened **2026-11-13 13:14**. Fix: a deterministic digest printer **`~/.hermes/cron/scripts/uber-eats-digest.py`**, a rewritten prompt with a hard output contract, and the skills cleared (**88 k → 1.8 k chars**); the lesson went into the `cron-job-maintenance` skill. Verified on the **18:51 → 18:58** slot: **154 unique / 153 completed** orders, Oct = **9 orders / $301.04** (unchanged), delivery row `delivered`, `hermes cron doctor` clean across all 26 enabled jobs. → [[reports/uber-eats/index]], [[systems/scheduled-tasks]]
- **Freshness scan (2026-11-13 03:00 stocktake): 0 stale / 131 fresh / 3 no-date of 134 pages** — nothing needed retiring. → `raw/stocktake/freshness-20261113.txt`
- **`executions.db` retention floor moved to 2026-11-08T00:01:20+08:00** (was 2026-11-06T05:09:30). The 11-07/11-08 cascade **runs** are still readable, but their **`errors.log` lines are not** — so the incident page's 58-timeout figure stands on the 2026-11-09 read and was **not re-verifiable** this window. → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
## Recent Changes (2026-11-12, nightly audit; live probes 2026-11-12 23:59 – 2026-11-13 00:4x AWST)
**Clean day. The estate is incident-clean, and the only open item is an upgrade that cannot be installed.**
- **The last open incident closed itself.** `cbfe43_ed01845310d8` (the 2026-11-10 audit's 429,
[[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]) went `closed_at: 2026-11-12T00:25:36.133387+08:00`,
`state: resolved`, when the **next slot of the same job ran clean** — `cbfe434a3b5e`
2026-11-11T23:59:54.486572 → 2026-11-12T00:25:36.133387 = **25 m 42 s**, `completed`, delivery carrying
**both** windows. **`cron_incidents`: 0 unclosed of 137.** No configuration produced this; the pool's
reset at ≈ 06:36:57 on 11-11 did, and the 07:0x cohort confirmed it.
- **State reads:** fleet **29 (26 enabled / 3 disabled)** — unchanged for a seventh window.
Ledger — **2026-11-11: 172 rows, 172 `completed` / 0 `failed`**; **2026-11-12: 176 rows, 175 `completed`
/ 0 `failed` / 1 `running`** (this audit); **2026-11-13 so far: 7 / 7 `completed`**. **Deliveries 11-11:
7 / 7 `delivered`; 11-12: 9 / 9 `delivered`, 0 `for_failure`.** Executions retention floor
`2026-11-07T00:00:42`, max `2026-11-13T00:30:55`, **1,001 rows** in the table.
- **New open item — a refused upgrade, not a failure.** The attempt to take **`hermes-hq-push`** past its
installed **0.9.42** was **blocked by the pre-install plugin security scan**:
*"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec))"*, and *"--force does not
override a dangerous verdict."* The critical is the plugin's **own** `skills/dashboard-tile/SKILL.md`
one-liner (`echo '<json>' | python3 …` = obfuscation). Verified in source: **there is no per-verdict
override anywhere** — `plugins.scan_on_install: false` disables scanning for *every* future install, and
the plugin guard hardcodes `trust_level="community"` (the skills-side allowlist machinery is not read by
it). Installed copy untouched, `enabled`, still delivering. **Fix = PR #2** on
`mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc`, **OPEN / MERGEABLE**,
`mergedAt: null`, +7/−2 in one file), pushed from a fork because `gh` is **read-only** upstream.
→ [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]]
- **Config untouched and consistent.** Live `config.yaml` is **byte-identical to the
2026-11-12 18:14:52 snapshot** (565,433 B). The only drift since **11-09** is the **model catalogue
block** (lines 1916-1941): 6 models removed, 9 added. No unrelated settings moved.
- **The "stored hold-and-ask instruction" is not stored.** The alleged standing rule about plugin
security-scan warnings was hunted exhaustively — a recursive sweep of `~/.hermes` (**184,716 files**,
generated/venv/browser-profile trees excluded) found **zero matches in any editable text file**; the
sentence exists only in `state.db` **conversation history** (58 rows). **Nothing was removed, nothing
was reverted** — there was no persisted policy to undo. Procedure recorded as the skill
`stored-instruction-search-and-removal`; installed skills **299 → 300**.
- **The recurring pre-dispatch-filter signature was re-confirmed** on a *second* job and a *second* pin by
the 2026-11-12 03:00 stocktake — still no failover for the free-tier cohort
([[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]).
## Recent Changes (2026-11-11, nightly audit; live probes 2026-11-11 23:59 – 2026-11-12 00:1x AWST)
**Recovery night — plus one failure, and it was the auditor's own.**
→ [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]
- **The 2026-11-10 audit died on its own 429.** `cbfe434a3b5e` ran 2026-11-10T23:59:54.486572 →
**2026-11-11T00:12:29.646987 = 12 m 35 s**, `failed`, delivery `suppressed`:
`RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch
filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` — **6 candidates, `attempted 0`**, tenant
`tn-5ud77abtl1` quota exceeded. Row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 but
**`alerted_at: NULL` — it was the estate's only open incident row; it has since closed `resolved` at 2026-11-12T00:25:36 (0 of 137 unclosed)**. Pool reset
≈ **06:36:57**; the 07:0x cohort ran clean.
⇒ **The 2026-11-10 window was never audited.** This audit covers two windows.
- **State reads:** fleet **29 (26 enabled / 3 disabled)**; **non-clean `last_status` 2** (was 1) —
`066bb5191280` Uber Eats (`delivery_failed`, streak 0) and `cbfe434a3b5e` (`error`, streak 1).
Ledger — **2026-11-09: 174 completed / 0 failed**; **2026-11-10: 157 rows, 155 completed / 2 failed**;
**2026-11-11: 172 rows, 171 completed / 1 running / 0 failed** (15 distinct jobs) — first zero-failure
day since 11-09. **Deliveries 11-11: 7 rows, 7 / 7 `delivered`, 0 `for_failure`** (last at 18:45:58).
- **Stocktake recovered with output.** `1a8e90ea2271` 03:00:56.081504 → **04:11:38.724852 = 1 h 10 m
42 s**, `completed`, **36 stale pages re-verified → 0 stale**, **commits `341dda9` + `c108eb1`**,
freshness report `raw/stocktake/freshness-20261111.txt` (6,235 B). `341dda9` also rewrote the live
fallback chain in [[runbooks/test-openrouter]] (primary back to `smart-route` / `custom:omni-101`),
corrected the CT333 `clawtest` naming claim, and replaced the nous refresh-token reading with the
404-credits signature. Closes [[incidents/2026-11-09-stocktake-completes-with-zero-edits]].
⚠ The job is **still pinned to `free-stack`**, the combo recorded broken on 2026-08-14, and succeeded.
- **Job Radar recovered with output.** Stage 1 `dca8482e4f76` 07:01:08.358004 → **07:37:34.076224 =
36 m 26 s**, `completed`, writing **`radar-ref/handoff/2026-11-11.json` (22,682 B, mtime 07:59)** — the
first handoff since **2026-11-06**. Stage 2 `5fc292ff0f3b`: recovery fire 07:35:56.632032 →
08:03:47.816811 (`delivered`), scheduled 11:30:56 → 11:35:18, 14:30:55 → 14:34:56 (`delivered`). Row
`5fc292_773822455856` auto-closed **`resolved` 2026-11-11T08:03:47.644226**, closing
[[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]].
- **Longest runs.** 11-11: stocktake **1 h 10 m 42 s**, Radar S1 **36 m 26 s**, Radar S2 **27 m 51 s**
(recovery fire), Newsletter Digest `ecc1fc13655a` 12:01:02.571781 → 12:19:38.965387 (**18 m 36 s**),
`85edf6772bca` 07:01:08.287627 → 07:11:42.858722 (**10 m 34 s**). **Nothing over 71 minutes.**
11-10: Radar S2 scheduled slot 11:30:35.267155 → 22:18:49.814215 = **10 h 48 m 14 s**, `failed` /
`interrupted`, delivery `delivered` with `for_failure: 1`.
- **Provider state (unchanged config; worsening symptoms).** `config.yaml` mtime still **2026-11-09
22:58:07** — nothing repaired. Top model `custom:omni-101` / `smart-route`
(`http://100.78.63.20:20128`). `fallback_providers` (6): nous `meituan/longcat-2.5-preview:free` →
**opencode-zen `deepseek-v4.1-flash` (EXPIRED key)** → `custom:guacano` → `custom:litellm`
`kilo-reflection-beam-free` → kilocode `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`)
`or-inclusionai_ling-2_6-1t`. **The expired opencode-zen key is no longer latent** — **7 × HTTP 401
`api_key_expired`** on 11-11 (07:01:13 … 19:19:39), each immediately after nous returned **14 × 404
`insufficient_credits_for_paid_model`** (07:01:11 … last 19:19:38, all from session
`20261031_190813_95fbde`, source `desktop`, opened 2026-10-31, **still open**, pinned to that paid
model). **Two consecutive dead hops on one chain.** → [[systems/opencode-go]]
- **Auxiliary slots:** nine slots plus `free_only: true`. Eight chain
`no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` on `custom:omni-101`.
**`pulse_memory` remains the sole exception with no `fallback_chain`**
(`{"provider": "nous", "model": "anthropic/claude-haiku-4.5"}`) — 19 aux log lines on 11-11 across
06:0x / 08:0x / 13:0x / 18:0x, including **11 ×** *"main provider `custom:omni-101` is unavailable and
no fallback_chain / fallback_providers is configured — refusing to guess another logged-in provider"*.
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
- **Error log:** `~/.hermes/logs/errors.log` = **6,179 lines** through **2026-11-12 00:10:11**.
- **Still open, untouched this window:** Uber Eats Monthly Spend Report `delivery_failed` streak 0 (no
execution row on 11-11 — monthly cadence); disabled trio unchanged (Important-mail monitor, Rhino
AgentMail Inbox Watcher, AgentMail Rhino Inbox Poll); the three inherited items below.
## Recent Changes (2026-11-10, nightly audit; live probes 2026-11-09 23:5x – 2026-11-10 01:0x AWST)
- ✅ **2026-11-09 was the first `failed`-free cron day since 2026-11-05: 174 rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit). Trend: 11-06 1 failed / 172, 11-07 3 / 163, 11-08 1 / 170. Deliveries 8/8 `delivered`, 0 `for_failure`.
- ⚠️ **The stocktake ran 9 h 10 m, closed `completed`, and landed nothing — third day running.** `1a8e90ea2271` (**03:00:43 → 12:10:47**) delivered `suppressed`; its own output file says `RUN INCOMPLETE`, tool-iteration budget exhausted during verification, **zero edits and no commit**. `failure_streak` 3 → 0 and `last_status ok` are the harness's verdict on the *process*, not the work. The job is still pinned to `custom:omni-101` / **`free-stack`** — a combo recorded broken on **2026-08-14**. → [[incidents/2026-11-09-stocktake-completes-with-zero-edits]], [[systems/scheduled-tasks]]
- ⚠️ **The Job Radar landed nothing on 11-09.** Stage 1 (`dca8482e4f76`) ran **07:00:48 → 16:10:12 = 9 h 09 m 24 s** and never wrote `radar-ref/handoff/2026-11-09.json` (newest handoff on disk is **2026-11-06** — nothing has landed since). Both Stage-2 slots (`5fc292ff0f3b`) fired **before** Stage 1 finished (11:30:38 and 14:30:34) and declined silently. The run's own last words: *"I stopped retrying because I kept running read_file 5 times without making progress."* Same silent family as 11-02/11-03, new trigger (read loop, not iteration cap), and again **no incident row was raised**. → [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]
- ✅ **The 11-08 `pre_tool_call` hook cascade did not recur.** `errors.log` (1,035 retained lines) contains **no `pre_tool_call` timeout entry at all**; its only two hook lines are benign skips (`on_session_end` 2026-11-09 22:58:35, `pre_llm_call` 2026-11-10 00:02:54). No 11-09 run outside the two documented long-runners exceeded **74 min**. → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
- **Unclosed incident rows 3 → 0** (`cron_incidents`: 134 rows, 0 open). The stocktake cluster closed **automatically at 2026-11-09T12:10:47.125205** — the instant the overrunning run finished; none of the three ever alerted. → [[systems/scheduled-tasks]]
- **Tailscale: 58 non-funnel nodes (57 peers + `hermes`), 42 online.** `octop` is **online** again (dark since `LastSeen 2026-11-08T13:13:04Z`); 8 offline ages re-drifted — cloudreve 106d → **108d**, elementsynapse 109d → **111d**, freshrss-new 82d → **84d**, conduit 62d → **65d**, octoagent 6d → **8d**, omarchybook 17d → **19d**, openmausbot 17d → **19d**, ubuntu-server-xfce 78d → **81d**; the 6 ephemeral `codex-environment-*` peers' last-seen stamps were corrected. → [[infrastructure/tailscale]]
- **OmniRoute `omniroute-1` (CT101, `100.78.63.20`): `/v1/combos` → HTTP 200, 55 combos**, `free-stack` present — one ahead of the page's last reading (54). `/v1/models` on **both** nodes and CT600's `/v1/combos` return **HTTP 401** (alive, auth required), so the **6,480-model catalog figure dated 2026-10-26 could not be re-verified this window and was not changed**. → [[systems/omniroute]]
- **Not verified this window (gap recorded, no page changed):** CT450/CT460 Hermes version strings — `pct exec 450 -- sh -lc 'hermes --version'` resolves the **known-stale root-level `/usr/local/bin/hermes` (v0.19.0)**, not the user-level install, and CT460's `hermes` is not on the PATH in that shell, so **[[infrastructure/hosts]]'s CT450/CT460 version lines stand as written**; and CT450/CT460 docker `Up` days (`lxc-attach … Failed to exec "docker"`).
## Recent Changes (2026-11-09, nightly audit; live probes 2026-11-08 23:5x – 2026-11-09 00:2x AWST)
- ⚠ **The `conduit_push` `pre_tool_call` hook was the day's fault, and it is fixed — `plugins.hook_callback_timeout` 30 → 120 s.**
The plugin is the **only** `pre_tool_call` registrant on the host, and its callback overran Hermes's 30 s
hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: 12
lines on 11-07, 46 on 11-08; plus **89** *"skipped after previous timeout or while still running"*
suppression lines). Every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST**;
the last timeout line is still 08:21:03, re-verified at audit time. **The plugin's caching was measured,
not assumed, and was innocent** — 3.1 s cold / 0.00 s warm (`session_titles.py:21 CACHE_SECONDS = 30.0`);
the trigger is **load** on the 2-core CT460 (load average **5.89**, `state.db` with a **40 MB WAL**).
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]], [[systems/conduit]]
- ⚠ **`state.db` is now 1.89 GB** (1,890,656,256 B; **2,901 sessions / 197,682 messages**, live
**40,157,672 B** WAL) — up from 1.7 GB, and it is the load source behind the spike above. The 2026-08-11
corruption remnant **`state.db.corrupted.20260811_151052` (624.1 MB)** was **deleted** in this window;
`state.db.recovered.recovery.json` (14,566 B) remains, and `mnemosyne.db.corrupted.20260809_120404`
(23.6 MB) was **not** cleaned. → [[systems/hermes-agent]], [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]
- ✅ **The Mnemosyne Health Monitor's no-agent conversion is verified end-to-end.** The 18:00 slot (agent
mode) ran **12:00:51 → 18:00:42 = 5 h 59 m 51 s**; the converted job's first slot ran
**2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, **silent**, writing a **159 B** output file against
**105,982 B** for the last agent-mode one. Live coverage **working 1030/1030 = 100.0 %, episodic
1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet. → [[systems/mnemosyne]], [[systems/scheduled-tasks]]
- ⚠ **The stocktake failed a third consecutive day, on a new signature.** `1a8e90ea2271` ran
**2026-11-08 03:01:11 → 04:36:08** and died on `RuntimeError: ⚠️ No reply: the turn stopped while a tool
result was still pending and the model produced no follow-up text.` — the previous two red days read
*"lost its durable fire claim ownership"*. **`failure_streak` 3.** *Working hypothesis, not proven:* the
hook above holds a tool result open in exactly the way this error describes. **Unclosed incident rows
2 → 3**, and all three are silent (`detected`, `alerted_at: null`) — but **the alert path works**: the
*Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` at **08:06:54**.
→ [[systems/scheduled-tasks]], [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
- **2026-11-08 settled: 169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit). Deliveries
**8 / 8 `delivered`**. Fleet **29 jobs (26 / 3)** — a **sixth** consecutive audit at the same count, though
`updated_at` moved to `2026-11-09T00:06:50+08:00` with no definition change. Nothing added, retired,
disabled or repinned. → [[systems/scheduled-tasks]]
## Recent Changes (2026-11-08, nightly audit; live probes 2026-11-07 23:5x – 2026-11-08 00:5x AWST)
- **Hermes on CT460 is now `0.23.0`** (was `0.22.5`): `hermes update`, 2026-11-07 05:05:13 → 05:43:10,
`bf0203a794a07aac3ff764972ab9087baa4e4f4e` → `2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`. ⚠ **The `snapshot`
stage was skipped again** (`updates.pre_update_backup` mode `off`) → **there is still no rollback point**.
Gateway restarted 05:37:34, dashboard 05:43:04, and the gateway once more at 11:11:44 → 11:11:55
(live `pid 3109529`). → [[systems/hermes-agent]], [[agents/hermes-production]]
- ⚠ **The restart drain cost two cron runs their fire claims.** A restart requested at ~04:29:52 sat in
*after-turn* mode for the full 1800 s, timed out, and drained at 05:00:43/47 — killing *Mnemosyne Health
Monitor* (`ea79a2107e36`, 00:00:42.918 → 05:02:22.870) and *wiki-stocktake-maintenance*
(`1a8e90ea2271`, 03:00:31.515 → 05:02:22.174), both closed `failed` on
`lost its durable fire claim ownership`. systemd restarted the unit at **05:01:00**
(`restart counter is at 3`). → [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
- ⚠ **Two incident rows are open and neither alerted.** `1a8e90_628d914f1f97` (first seen 04:29:54.815) and
`1a8e90_2f370ef764f5` (05:02:21.573) both sit `detected` for *wiki-stocktake-maintenance* with
**`alerted_at: null`** — the *red row, no alert* shape first recorded for Job Radar on 11-06.
- ⚠ **The 03:00 stocktake slot has now failed two days running** (`failure_streak: 2`, `last_status: error`,
2026-11-06 + 2026-11-07). The 11-07 in-day entry committed the stocktake's own edits so the tree was not
left dirty.
- **`camofox` is now an enabled plugin (`v1.1.0`)**, not a core feature — installed by the 0.23.0 update.
No `browser.*` config key changed. → [[systems/browser-backend]]
- **`conduit_push` self-updated `0.122.2 → 0.134.4`** (`git` source, tree HEAD `ef3d2f2`) and recorded its own
last step **`restarting_gateway` = failed** — the gateway it wanted to restart is systemd-managed. →
[[systems/conduit]]
- **`hermes-hq-push` reinstalled `0.9.27 → 0.9.42`** (upstream `d26850e7`, 60/60 files) at 06:42; its web
routes only load at service start, so the 11:11:44 gateway restart is what put them on the wire. →
[[systems/hermes-hq-push]]
- ⚠ **The OpenCode Go credential is expired** — HTTP 401, `code: api_key_expired`, on
`…/zen/go/v1/models` (probed twice in-window). Configuration alive, credential dead; no 2026-11-07
execution failed on an auth error, so nothing was consuming it yet. → [[systems/opencode-go]]
- ⚠ **Skill inventory moved: 302 → 299 skills (68 → 67 categories).** Four skills went into
`~/.hermes/skills/.archive/` in one pass at **2026-11-07 07:43** — `cron-model-repair`,
`fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup` — and the parked archive grew
**46 → 50**. No active `SKILL.md` references any of them, so if this was a curator consolidation the
*"absorbed into X"* pointer is missing. `microsoft-office` appeared in the same window (Productivity
25 → 26). → [[systems/skills-index]], [[systems/scheduled-tasks]]
- **Cron day (2026-11-07): 163 rows — 159 `completed` / 3 `failed` / 1 `running`**; deliveries 7/7
`delivered`, 0 `for_failure`. Fleet unchanged at **29 jobs / 26 enabled**. →
[[systems/scheduled-tasks]]
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] **remains OPEN** — the re-check
found cron job prompts clean of secret-shaped strings, but the redaction setting could not be confirmed.
## Recent Changes (2026-11-07, stocktake; live probes 2026-11-07 03:1x-03:2x AWST)
- **The tailnet lost a peer and the m93p guest list lost a guest — both corrections came from live probes, not the age scan.**
[[infrastructure/tailscale]] was rebuilt from `tailscale status --json`: **`octop` (`100.92.163.122`) is back
online** (peer record `Online: true`, `tailscale ping` → `pong from octop via DERP(syd) in 98ms`; it remains
**no LXC or VM on either hypervisor** — an orphaned tailnet peer), and the **9 remaining offline ages** were
refreshed (+1d each). All **51 live IPv4 peers** match the table's IPs exactly, with no missing peers and no
orphan rows (**52 rows** = 51 peers + `hermes` itself). The CT460 Serve route table still matches
`tailscale serve status` **exactly (12 paths on :443)**; both previously-flagged dead targets were
**re-confirmed dead** — still no listener on `8769` (`/agent-estate`) or `9123` (`/synergy-prep`).
- **⚠ `m93p` actually runs 20 LXCs, not 19 — CT107 (`mem0`) was missing from the Proxmox page.**
Live `pct list` on `m93p` returns **20 LXCs — 18 running, 2 stopped** (CT998 `nightbridge`, CT1264
`openmausbot`). The omitted guest is **CT107 `mem0`** (`mem0-mcp.service` active on `:8890`,
`pct exec 107 -- tailscale ip -4` → `100.73.94.53`). Added to [[infrastructure/proxmox]]; the
**2026-10-31 reading of 19 was already short for the same reason**. `pve` still returns exactly
**17 LXCs + 4 VMs**, and both hosts still run the identical `pve-manager 9.2.10` build.
- **⚠ `m93p`'s uptime claim was wrong.** The page read *"up 1 day, 12:02 … rebooted ~2026-10-30 16:0x"*.
Live `uptime -s` shows the host **booted 2026-11-05 21:19:52** (up **1 day, 7:18** at probe time, load
**0.76 / 1.08 / 1.24** — normal). The 2026-10-30 reboot was real but belonged to **`pve`**
(boot **2026-10-30 16:24:00**), not `m93p`.
- **Storage percentages refreshed on both hosts** against live `pvesm status` — `pve`'s `ssd0` moved
**76.43 % → 78.29 %** (~152 GiB free), `fred-qnap` **82.77 % → 84.21 %**, `local` **79.66 % → 81.62 %**;
on `m93p` the iMac shares and `local` moved **60.25 % → 76.21 %** and `omb-internal`
**57.92 % → 64.66 %**.
- **Docker inventories diffed set-wise against live `docker ps` on all three hosts.** CT200 matches
**61 running / 68 total** exactly; CT201 matches **20 / 26** exactly; CT333 matches **8 / 9** — but its
**enumerated list still opened with `flamboyant_mahavira`, which is not running** (live name is
**`nostalgic_keldysh`**, created 2026-11-05 22:15:57). Corrected in [[infrastructure/docker-services]].
- **Two small cross-page corrections.** [[infrastructure/network-map]]'s `/gitea` route was annotated
*"Gitea (ct207 tailnet IP)"* — it is **CT221 on `m93p`** (`100.82.100.128`; ct207's own IP is
`100.82.40.23`); the target answers **HTTP 200**. [[agents/hermes-chaos]]'s version line read
*"`v0.22.3` … local `5b147ab036d6` — the same build as CT460"*, which contradicted itself: live
`hermes --version` on CT450 is **`v0.22.5` … `bf0203a794a0`**, now genuinely identical to CT460's.
[[agents/hermes-production]]'s claim that *2 cron jobs still pin `custom:omni`* is **no longer true** —
live `jobs.json` shows **0** jobs on the bare key and **21** on `custom:omni-101`.
- **Cron fleet re-verified unchanged:** **29 jobs (26 enabled / 3 disabled)**, matching the nightly audit
hours earlier. CT450's fleet is **25 jobs (13 enabled / 12 disabled)** with **9 shared names** — confirmed
by a live set-diff of both `jobs.json` files. → [[systems/scheduled-tasks]]
## Recent Changes (2026-11-07, nightly audit; live probes 2026-11-07 00:3x-00:5x AWST)
- **A red row with no alert: the Job Radar's 11:30 slot was discarded as stale while a concurrent fire
held the claim.** 2026-11-06 settled **172 rows across 16 jobs - 170 `completed` / 1 `failed` / 1
`running`** (this audit). The failure is `5fc292ff0f3b`'s **11:30:32 slot**, interrupted at **13:26:38**
and closed **13:26:44** on `Fire claim ownership lost; stale result was discarded.` - because a **`direct`
fire of the same job (08:27:09 -> 13:26:22, 4 h 59 m 13 s)** was still holding the claim and delivered
first. **Every previous occurrence of this string came from a restart orphaning a run; this one had no
restart.** The failed row raised **no `cron_incidents` row** (newest ledger activity is still the
2026-11-05T12:36:17 closures), so **nothing alerted**. The day still landed complete via the 14:30 slot
(12 m 27 s, `MISSING_COVER_LETTER_COUNT 0`) ->
[[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]].
- **The stocktake's overrun is fully unwound.** `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m 03 s)**
against 7 h 40 m / 5 h 41 m / 3 h 02 m on 11-03/11-04/11-05, and committed normally.
- **Stage 1 ran clean for a second day** (`dca8482e4f76`, 07:00:44 -> 08:29:21, 1 h 28 m 37 s, `completed`,
`suppressed`), and the **11-04 Open entry about its two lost candidate sets is cleared in substance** -
its 11-03 ledger row closed `resolved` at 2026-11-05T12:36:17 and both runs since have been clean.
- **Ledger clean, deliveries clean:** **unclosed incident rows 0 -> 0**; deliveries **10 rows / 10
`delivered` / 0 `for_failure`**; fleet **29 jobs (26 enabled / 3 disabled)** - fifth consecutive audit at
the same count, non-clean `last_status` still **1** (`066bb5191280`, `delivery_failed`, streak 0) ->
[[systems/scheduled-tasks]].
- **The previous audit's open question closed itself:** the Mnemosyne Health Monitor's 11-05 12:00:45 slot
(still in flight 12 h+ with no delivery at the last audit) **completed at 2026-11-06T00:14:39 after
12 h 13 m 54 s and delivered**; its 11-06 12:00 slot then took **7 h 47 m 56 s** and delivered too. The
**one still-open page is unchanged:** [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
## Recent Changes (2026-11-06, nightly audit; live probes 2026-11-06 00:1x-00:4x AWST)
- **A clean ledger day and an empty incident ledger.** 2026-11-05 settled **175 rows - 173 `completed` / 2
`running` / 0 `failed`** across 18 jobs: the **first zero-`failed` day since 2026-11-02**. **Unclosed
incident rows 3 -> 0** - the *Tech & AI Newsletter Digest* row closed at **07:07:42** and both *Job Radar -
Discovery* rows at **12:36:17**, when that job finally completed after **5 h 35 m 35 s**. Fleet non-clean
`last_status` **3 -> 1** (only `066bb5191280`, `delivery_failed`). Deliveries **10 rows / 10 `delivered`**
-> [[systems/scheduled-tasks]].
- **The stocktake's three-day overrun streak broke, and it committed.** `1a8e90ea2271` ran **03:00:32 ->
06:02:30 (3 h 01 m 58 s)** - **7 h 40 m -> 5 h 41 m -> 3 h 02 m** over three days - and **committed
`2f3fc99` at 05:55**, against its 11-04 state of "reported incomplete, committed nothing".
- **Long runs, not failures:** the **Mnemosyne Health Monitor**'s 00:00 slot took **7 h 01 m 28 s** (normal
17-31 m) and its **12:00:45 slot was still running 12 h later with no delivery**; **Stage 2**'s 14:30 slot
took **8 h 56 m 17 s** and delivered at 23:27:09.
- **New open incident - live secrets stored verbatim.** Three `.env` values (`TELEGRAM_BOT_TOKEN`,
`MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) sit in plaintext in Mnemosyne rows - including
the FTS mirrors) - 4 + 6 + 7 = **17 rows** - reproduced in full by each of the **7 uncompressed rotation
snapshots**; all four values also sit verbatim in the **1.7 GB session store `state.db`**, the likely capture
origin. `CAMOFOX_API_KEY` sits verbatim in the prompts of two **enabled** Facebook Birthday jobs
(`0e753e9720fd`, `fad5a86979d4`) and hence in `cron/deliveries.db`. Found, not remediated; no value is recorded
anywhere -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
- **Fleet unchanged:** **29 jobs (26 enabled / 3 disabled)**, a fourth audit at the same count; no job added,
retired, disabled or repinned.
## Recent Changes (2026-11-05, stocktake; live probes 2026-11-05 05:1x-05:3x AWST)
- **The `systems/skills-index.md` flip-flop is FIXED at the root - it had two writers, not one bad writer.**
The repo `wiki-maintain.sh` (302 skills / 68 categories) and `~/.hermes/scripts/wiki-maintain.sh`
(mtime **2026-08-17**, 349 skills / 26 categories - it used `Path.rglob()` and so counted the parked
`.archive/` tree) are **different scripts with different generators**, and `wiki-maintenance/SKILL.md`
line 569 pointed the **23:59 nightly audit** at the stale one while the **03:00 stocktake** ran the repo
one. Each job overwrote the other's page; **neither raised an error.** The stale copy is now a
**delegating wrapper** with no generator logic, and the repo script gained `--no-build` so the wrapper
honours the old interface without a second generator. Verified by running it -> 302 / 68.
-> [[incidents/2026-11-05-skills-index-generator-divergence]]
- **Docker inventory corrected against live probes on all five hosts:** CT200 **61 running / 68 total**; CT333's
first container renamed again (`nostalgic_leavitt` -> **`flamboyant_mahavira`**); CT201 20/26, CT460 1/1,
CT1001 1/9.
⚠ **Corrected 2026-11-06 (stocktake): the "the seeder does not exist on any host" claim in this entry was
WRONG - it was a point-in-time sampling artefact.** The `worldmonitor-seeder-*` containers are **transient
and systemd-timer driven** on CT200 (`worldmonitor-seed.timer` every 4h -> `worldmonitor-seeder-scheduled`;
`worldmonitor-seed-fast.timer` every 15min -> `worldmonitor-seeder-fast`), so a `docker ps` taken between
runs sees **none** of them. Live at 2026-11-06 03:1x **both were present**; live at 2026-11-05 05:1x
**neither** was. **"Absent" must never be read as "removed."** CT333's container has also been renamed
**again**: `flamboyant_mahavira` -> **`nostalgic_keldysh`** (created 2026-11-05T22:15:57Z).
-> [[infrastructure/docker-services]]
- **Tailscale node table re-verified programmatically:** 52 table rows vs 52 live IPv4 peers - **no live
peer missing, no orphan row, every online/offline state matches.** Two offline ages corrected (`conduit`
60d, `ubuntu-server-xfce` 76d). Serve table still matches `tailscale serve status` exactly (12 paths on
:443 + the :8443 funnel + the :9443 route).
⚠ **Corrected 2026-11-06 (stocktake): the "every online/offline state matches" claim in this entry no longer
holds - `anthonys-mac-mini` has come back online** (peer record `Online: true`, `tailscale ping` -> pong) and
**8 offline ages have drifted by +1d**. Two Serve routes (`/agent-estate` -> 8769, `/synergy-prep` -> 9123)
are **dead** - no listener - while still present in `tailscale serve status`.
-> [[infrastructure/tailscale]]
- **Freshness now 0 stale** (was 8 on 11-04) - all 8 re-verified and refreshed. Skills count corrected to
**302 / 68 categories** on [[infrastructure/hosts]] and [[index]].
## Recent Changes (2026-11-05, nightly audit; live probes 2026-11-05 00:0x-00:5x AWST)
- **A dead route took out two 07:00 jobs at once - and the safety-net slot saved the day.** `dca8482e4f76`
(**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**)
both died on `RuntimeError: Upstream stream failed before completion.` - **36 seconds apart**, at **09:41:44**
and **09:42:20**, each after **2 h 41 m** on `custom:omni-101` / `smart-route`. Neither produced output; both
raised and alerted a `cron_incidents` row. The 11:30 Stage 2 slot then **reconstructed the missing handoff
from Stage 1's candidate files**, re-verified every candidate live in Camofox, and wrote **5 roles (board
479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at
16:08:59 - see [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]].
- **Fleet: 29 jobs (26 enabled / 3 disabled).** In-window ledger: **170 rows - 167 `completed` / 2 `failed` /
1 `running`**. *(fleet and last row corrected 2026-11-05 01:3x - the first draft read 158 jobs / 144 / 14 and a
closing `item_suppressed` row; live `jobs.json` + `hermes cron list` give 29 / 26 / 3, and the 170th 11-04 row is
this audit, `running`. 11-03 has since settled to 171 `completed` / 1 `failed`.)* **Non-clean `last_status` 2 -> 3**; **unclosed incident rows 1 -> 3** (all
`state = alerted`, incl. the 11-03 `timeout` row `dca848_dfd4bea84f2a`). Deliveries on 11-04: **8 rows, 8
`delivered`** - the failure notices were delivered, not silent.
- **The cover-letter route is only alive by override.** `regenerate_cover_letter.py` line 82 reads `CL_MODEL`,
defaulting to **`deepseek-paid-smart`**, which returned **HTTP 402** during the 11-04 run; that process was
pinned to **`gemini/gemini-3.5-flash`**. **No `CL_MODEL` exists in `~/.hermes/.env`**, so the override did not
persist. Endpoint check 2026-11-05: `omniroute:20128` -> **401** and `100.78.63.20:20128` -> **401** (live,
key-gated - that is auth, not health). **Keyed re-probe 2026-11-05 01:33:12 AWST:** a completion request with
`deepseek-paid-smart` returns **503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`) on `omniroute:20128`,
but **200** on `100.78.63.20:20128` (`omniroute-1`), which resolved the alias to **`deepseek/deepseek-v4-flash`**.
So the model is **undispatchable on the CT600 node only** - an override or the `omniroute-1` route is required.
- **CT299 froze a second time.** A manual `vzdump 299 ... --storage ssd0` started **2026-11-04 08:30:04** and
wedged; the guest was frozen until **~00:30 on 11-05** (~16 h). Live probe **2026-11-05 00:54:21 AWST**:
`LOCK: 0`, `cgroup.freeze=0`, `ns/cgroup.freeze=0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump processes;
SFTPGo reachable from CT460 at `100.88.223.70:8080`. Abandoned temp dir
`/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299` still on disk. ->
[[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]
- **wiki-stocktake-maintenance overran for the third day running** - `1a8e90ea2271` ran **03:00:31 -> 08:42:17
(5 h 41 m 46 s)**, hit the turn ceiling, reported incomplete, delivery `suppressed`. Pin:
`custom:omni-101` / **`free-stack`**.
## Recent Changes (2026-10-30, nightly audit; live probes 2026-10-30 00:0x AWST)
- **Conduit voice calls were repaired — after three faults in series — and the first two real calls in the
product's history are now on record.** Anthony asked why Conduit never answered when he called. In order:
(A) `voice.speech_server.url` held the truncated `wss://…:8443/v1/`, which the plugin dials **verbatim**, so
the Mac answered **403** and the plugin blamed the key; (B) `dashboard.basic_auth.session_ttl_seconds` was the
**12-hour default**, and an expired phone session gets **WS 4401** with no Hermes turn and no error visible on
the phone (the iPhone re-authenticated at **08:00:59**, minutes *after* a silent call); (C) **both** host
providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq` → **403 /
Cloudflare 1010**. Fixed 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL → **2 592 000 s (30 d)**, `stt.provider:
local` (faster-whisper `base`), `tts.provider: ollama_local` (6.94 s of real speech verified); dashboard
restarted **08:03:26**. Verified end to end by driving the whole call path, then by **calls at 08:38:04 and
08:49:36** — `conduit-calls.json → history` is no longer empty.
→ [[incidents/2026-10-29-conduit-voice-silence]], [[systems/conduit]], [[systems/hermes-agent]]
- **The Job Radar's two-stage split worked end to end for the first time.** Stage 1 (`dca8482e4f76`,
`deliver: local`) ran **07:00:59 → 10:16:12** (3 h 15 m 13 s), closed **`completed`**, and wrote
`~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**,
sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Stage 2 (`5fc292ff0f3b`) consumed the
handoff at **10:40:59**, closed **10:44:48 `completed`** and delivered to topic 1030 in the same second:
**20 pages / 20 `page_ids` / 2 cover letters / `verified_cover_letter_count 2` /
`missing_cover_letter_count 0`**, Notion board **455 → 475**. Both of stage 1's incident rows
(`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12**; the 11:30 and 14:30 slots
then no-oped as designed (10 m 46 s / 1 m 51 s).
→ [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]],
[[runbooks/job-radar]]
- **Cron: 2026-10-29 settled 177 `completed` / 0 `failed`** — the first zero-failure day since 2026-10-24 — with
**10 deliveries, all 10 `delivered`, 0 failures**. Non-clean `last_status` **5 → 3**; unclosed incident rows
**4 → 1** (the two radar rows closed `resolved` at 10:16:12). Fleet unchanged at **29 jobs (26 enabled / 3
disabled)** at that audit.
- **Cron: 2026-10-30 closed 138 / 0 — after a 5 h 30 m HOST OUTAGE — and the "auto-paused" health monitor was never
dead, it was running.** Fleet **29 jobs (26 enabled / 3 disabled)**
(`jobs.json.updated_at 2026-10-31T00:29:59+08:00`), re-verified 2026-10-31 00:3x. **Zero execution rows exist
between 10:58:20 and 16:37:21** — PVE `pve` rebooted (boot **2026-10-30 16:24:00**), CT460 came back **16:28:00**,
the old gateway life (`pid=3913841`) was logged **UNCLEANLY** at **16:35:50** (*"no exit path ran — SIGKILL / OOM /
VM death"* — a candidate list, not a diagnosis; root cause **not established**, no OOM evidence), and 8 catch-up
runs fired at **16:37:21–16:38:22** → [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]. Against 10-29's
**177 / 0**, that is **≈39 lost slots and no failures**. **`ea79a2107e36` (Mnemosyne Health Monitor) is ENABLED
and clean again** — `last_status: ok`, streak **0**, incident `ea79a2_d93c3acdc4f2` closed **`resolved`
2026-10-30T10:02:48.662** when its 00:01:08 slot finished after **10 h 01 m 40 s**; its **12:01 slot was lost to
the outage** and is the still-`running` 16:37:21 catch-up. **Non-clean `last_status` 4 → 3**, **failing 1 → 0**,
**unclosed incident rows 2 → 1** (survivor: `0e753e_e7f8194a7d06`, due 11-01). **`deliveries.db` finished 10-30
with 8 rows, all `delivered`** (latest 18:46:04) — the 03:3x "zero rows / notice never delivered" reading is
superseded.
- **⚠ Superseded by the 2026-10-31 entry above — read that first.** Cron fleet changed shape overnight on 2026-10-30 — the Mnemosyne Health Monitor is paused and its failure
notice was never delivered.** `ea79a2107e36`'s 00:01:08 slot closed **`unknown`** at 01:04:22 on a
gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`:
Gateway connection failed; no local fallback"*), opening incident `ea79a2_d93c3acdc4f2` (`detected`), and the
scheduler **auto-paused the job**. Fleet is now **29 jobs (25 enabled / 4 disabled)**; non-clean
`last_status` is **4** (2 `error` + 2 `delivery_failed`); `ea79a2107e36` carries `failure_streak: 1`.
⚠ **The failure notice itself failed to send** — `hermes cron list` reports `send_path_degraded; live adapter
delivery to telegram:-1004321904721:102 failed`, and `deliveries.db` holds **zero rows for 2026-10-30**, so
the only trace is the incident row. **Unclosed incident rows 1 → 2.** Every other previously-red job
(`dca8482e4f76`, `1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`,
`c0a9fa314542`) has **cleared to `ok` with streak 0** on its own slot. Pins: **18 of the 25 enabled jobs pin
`custom:omni-101`** (11 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`); 7 enabled rows are `no_agent`
jobs with no pin. → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]
## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 00:2x AWST)
- **The radar's silent budget overrun became a recorded failure — and the radar is now two jobs.** Window
2026-10-27 settled **172 `completed` / 1 `failed`**. The failure is `dca8482e4f76` (Daily Marketing Job
Radar - Discovery): **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed **`failed`**, delivery `delivered`,
and the delivered report opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* —
discovery, verification, dedup and classification completed; **Notion ingestion, cover-letter generation and
the final cover-letter audit did not execute**. The prior day (10-26) had the same overrun but closed
`completed`, hiding it in a green row. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted`
11:03:59, **still open** at this audit (`failure_type: auth` — misclassified). → new page
[[incidents/2026-10-27-job-radar-budget-exhausted]]
- **Response: the radar was split into a two-stage pipeline the same day.** Stage 1 `dca8482e4f76` — Discovery
(`0 7 * * 1-5`, `deliver: local`, failures → topic 1030) writes a handoff and triggers stage 2; new stage 2
`5fc292ff0f3b` — Daily Marketing Job Radar - Write and Letters (`30 11,14 * * 1-5`, Telegram topic 1030,
created **2026-10-27T17:34:15**) lands the Notion entries, cover letters and audit. Two test fires of stage 2
on 10-27 (17:35:00 → 17:36:52, 17:38:38 → 17:43:42) both `completed` with delivery `suppressed`; the first
real 07:00 → 11:30 cycle had not yet run. **Fleet 28 → 29 jobs (26 / 3).** → [[runbooks/job-radar]]
- **The 2026-10-26 orphan class did not recur, and its two incident rows closed.** `ea79a2_a4661554a3c2` and
`ea79a2_c3ab2d784ba3` **auto-closed 00:38:32** (unclosed rows **3 → 2**, the 2 being the Facebook birthday
monthly scrape from 10-01 and the new radar row). No 10-27 run besides the radar's failed, so the watchdog
and every other job were clean all day.
- **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error`
(streak 1) and `08eb86f02d37` re-joined as `delivery_queued`. Still non-clean: `0e753e9720fd` (error),
`fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.**
- **Also updated:** `incidents/2026-10-27-job-radar-budget-exhausted.md` (new),
`incidents/2026-10-26-planned-restart-orphaned-cron-run.md` (recurrence check), `runbooks/job-radar.md`,
`systems/scheduled-tasks.md`, `systems/hermes-agent.md` (TTS provider correction),
`systems/conduit.md` (voice pipeline), `infrastructure/tailscale.md` (peer table re-verified),
`systems/skills-index.md` (regenerated 10-27, committed), `index.md`, `log.md`.
**Nothing removed or archived.** No secrets written.
## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 22:00 – 2026-10-29 00:3x AWST)
### The radar's two-stage split ran for real, and landed nothing — stage 1 hit the agent iteration cap
`dca8482e4f76` (stage 1, Discovery) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**) and closed **`failed`** on
*"hit the agent iteration cap before the handoff could be written"* — **`failure_streak: 2`**. Stage 1 never
produced a handoff (`~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`), so the *first* real two-stage
cycle delivered **no Notion writes and no cover letters** — the same outcome as 10-27, by a different route.
Both stage-2 slots then did exactly what they were built to do: `5fc292ff0f3b` fired **11:30:27 → 11:31:31** and
**14:30:27 → 14:31:43**, both `completed` with **`delivery_outcome: suppressed`** — *a stage-2 run with no
handoff is a silent no-op, not a failure* (the recovery rule in [[runbooks/job-radar]]), now proven in
production. The two post-split failures also stopped at different ceilings: 10-27 at the **tool-call budget**
(4 h 2 m, writes not started), 10-28 at the **iteration cap** (3 h 32 m, handoff not written) — less of the
pipeline completed despite the shorter wall clock. Open radar incident rows: `dca848_e93a538ade6b` (10-27) and
`dca848_0a2f03b5952b` (10-28), both open. Detail:
[[incidents/2026-10-28-job-radar-stage1-iteration-cap]].
### The 03:00 stocktake failed on a provider 400 — and its incident row never alerted
`1a8e90ea2271` ran **03:00:28 → 03:37:58** (37 m 30 s) and closed **`failed`** (streak 1, delivery
`suppressed`) with `RuntimeError: [400] Failed to deserialize the JSON body into the target type: duplicate
field 'reasoning_content' at line 1 column 512203` against `custom:omni-101 / free-stack` — the *response* the
provider was asked to deserialize carried a duplicated `reasoning_content` field. It performed its rebuild
(302 skills → `systems/skills-index.md` mtime 03:01:40; freshness report `raw/stocktake/freshness-20261028.txt`
→ **2 stale / 103 fresh / 4 no-date of 109 pages**) and then died **before the commit**, leaving the wiki tree
dirty for the rest of the day. Its incident row `1a8e90_f57ab4cafab8` sits in state **`detected`** with **no
alert** — the only outward sign was the job's own `error` status. The 2026-10-29 audit committed the two
artefacts.
### Steel left the stack: fully removed — Camofox is the sole backend
`browser.cloud_provider` and `browser.camera_provider` are back to **`camofox`** (`engine: auto`, `cdp_url: ""`)
in `config.yaml` (mtime **2026-10-29 00:15:13**), with backups taken at every step
(`config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z` 22:00:57, and
`config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24). The **`browser-steel` plugin was
removed** from `~/.hermes/plugins/`, and no `browser-steel` key remains in `plugins.disabled` or
`plugins.entries`, so the `plugin_steel` toolset is no longer selectable. **Steel is now fully REMOVED, not
merely down:** on **CT201** the containers `steel-browser-api-1` / `steel-browser-ui-1`, the `steel-network`
Docker network, the images (`steel-browser-api:fixed` and both ghcr.io Steel images) and the project dir
`/root/steel-browser` were deleted (~6 GB reclaimed; disk 70G used → 64G used), and the Tailscale Serve routes
`/steel` and `/steel-api` were removed — CT201's remaining routes are only `/` → `127.0.0.1:8085` and `/mgr` →
`127.0.0.1:8084` on **both** `:443` and `:8443`, with nothing listening on 3000, 5173 or 9223. **CT333's second,
separate Steel copy** (`steel-browser`, `ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30)
was removed with its image. Hermes core contains **zero** Steel references, so there is **no Steel fallback path
in the tree** — returning to Steel means redeploying it (the compose file and Dockerfile are archived on CT201
under `/root/_archived-steel/`; the images remain re-pullable from ghcr.io). ⚠ `STEEL_BASE_URL` is no longer in
`~/.hermes/.env` — it survived only in the inherited environment of processes already running at the cleanup,
and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it (a couple of older, unrelated services
still carry it) — **no core Hermes code reads it**, so it is a runtime-only leftover, **not** a config value.
**Camofox is the sole
backend** and healthy: REST `http://100.118.5.51:9377` (and `http://localhost:9377`), `/health` → **200**,
`/tabs` → `{"running":true,"tabs":[]}`. Detail: [[systems/browser-backend]],
[[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]].
> ⚠ **Superseded:** an earlier 2026-10-29 00:3x probe described Steel only as "not reachable — endpoint dark,
> still running on CT201". That was already out of date — Steel is **removed**, not down.
### Fleet and tallies
29 jobs (**26 enabled / 3 disabled**) — unchanged. Non-clean `last_status`: **5** (`0e753e9720fd` error,
`fad5a86979d4` delivery_failed, `dca8482e4f76` error **streak 2**, `066bb5191280` delivery_failed,
`1a8e90ea2271` error **streak 1**). Pins: **12 `smart-route` / 6 `free-stack` / 7 unpinned /
1 `deepseek-paid-smart`**. **2026-10-28 settled 168 completed / 2 failed / 1 running**; **2026-10-29** so far
6 completed / 1 running; **2026-10-27 settled 173 / 1** (revised up by one late run after the 2026-10-28
audit). No Hindsight service is running on this box (`:8888` closed, no `hindsight` units) — the standing
memory-backend correction still holds and needs no change.
## Recent Changes (2026-10-27, nightly audit; live probes 2026-10-27 00:1x–00:5x AWST)
- **A *planned* gateway restart still orphaned a live cron run — and this time it alerted loudly.** A planned
`SIGTERM` at **2026-10-26 14:33:35** produced a shutdown drain that reported **`timed_out=True` after
just 1.35 s** of its 30 s budget and interrupted `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot)
**2 h 34 m** in. It closed **`failed`** on a **new** string — `lost its durable fire claim ownership` —
against the 2026-10-20 orphan's silent `unknown`, and raised **two** `cron_incidents` rows (14:33:37
*unverified / reconcile admission*, 14:35:59 *fire claim*) that are still `alerted`; the failure **was
delivered** at 14:35:59. Two Mnemosyne Watchdog reds followed and **self-resolved at 15:16:12**. Window
2026-10-26 settled **169 / 3** and **all three failures are this one fault** — so the fleet's failure
count is one interruption, not a systemic turn. Fleet unchanged at **28 jobs (25 enabled / 3 disabled)**;
**non-clean `last_status` 5 → 4**; **unclosed incident rows 2 → 3**; **pins unchanged**. →
new page [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]
- **One incident row closed, and the 2026-10-16 503 burst is fully shut.** `c0a9fa_764558b5ae83`
(GitHub Trending Radar) **auto-closed 09:13:00** when its 09:00 slot ran clean — the last open row of the
resource-pressure fault → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. No execution row
carries the 503 string on 10-25 or 10-26. The job stays **`monitoring`**: single-node exposure unchanged.
- **Two quiet degradations worth naming.** `fad5a86979d4`'s 03:08 slot **completed but delivered
`delivery_failed`** — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead
thread at 14:33:37); and the **radar's 07:00 slot closed `completed` at 09:10 while delivering
“RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters, batch finished by
hand. That morning the cover-letter PDF pipeline moved to **Chromium over the canonical HTML** with model
**`deepseek-paid-smart`** and a **600 s** dashboard timeout → [[runbooks/job-radar]]
- **Also updated:** `incidents/2026-10-26-planned-restart-orphaned-cron-run.md` (new),
`incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md` (recurrence), `incidents/2026-10-16-omniroute-503-resource-pressure.md`,
`runbooks/job-radar.md`, `systems/scheduled-tasks.md`, `index.md`, `log.md`. **Nothing removed or archived.**
## Recent Changes (2026-10-25, nightly audit; live probes 2026-10-25 00:1x–00:5x AWST) *(superseded by the 2026-10-27 block above)*
- **A third consecutive clean cron day — and an off-cron fault that is not the fleet's.** Window - **A third consecutive clean cron day — and an off-cron fault that is not the fleet's.** Window
2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at
@@ -368,7 +981,7 @@ Updated: 2026-10-25
- **OmniRoute CT600/CT601 roles are INVERTED.** CT600 `omniroute` (100.88.81.19) is **running** and serving `:20128` (combo API returns live combos); CT601 `omni` (100.93.204.4) is **stopped** with `onboot=0` (Tailscale peer offline 54d). Hermes resolves `custom:omni` → `omniroute:20128` → **CT600**. → [[systems/omniroute]] - **OmniRoute CT600/CT601 roles are INVERTED.** CT600 `omniroute` (100.88.81.19) is **running** and serving `:20128` (combo API returns live combos); CT601 `omni` (100.93.204.4) is **stopped** with `onboot=0` (Tailscale peer offline 54d). Hermes resolves `custom:omni` → `omniroute:20128` → **CT600**. → [[systems/omniroute]]
- **A third OmniRoute node exists:** `omniroute-1` at `100.78.63.20:20128`, live (401 = auth-required), registered in Hermes config as provider `omni-101` (5384 models, default `smart-route`). It carries `skills_hub` (model `free-stack`) — the same provider family implicated in the 2026-10-09 timeout storm. Not present in `pct list`; identity unconfirmed. → [[systems/omniroute]] - **A third OmniRoute node exists:** `omniroute-1` at `100.78.63.20:20128`, live (401 = auth-required), registered in Hermes config as provider `omni-101` (5384 models, default `smart-route`). It carries `skills_hub` (model `free-stack`) — the same provider family implicated in the 2026-10-09 timeout storm. Not present in `pct list`; identity unconfirmed. → [[systems/omniroute]]
- **Two Ignis instances exist, on different hosts.** CT460 `ignis` (port 8080, vault `/home/hermes/wiki/agent-estate-wiki`, serving 200) and CT200 `ignis-ignis-1` (image `ignis-server:latest`, host port 8092, vault `/opt/ignis/vaults`, serving 200). Both healthy; they are separate deployments. → [[systems/ignis]] - **Two Ignis instances exist, on different hosts.** CT460 `ignis` (port 8080, vault `/home/hermes/wiki/agent-estate-wiki`, serving 200) and CT200 `ignis-ignis-1` (image `ignis-server:latest`, host port 8092, vault `/opt/ignis/vaults`, serving 200). Both healthy; they are separate deployments. → [[systems/ignis]]
- **Tailscale route drift fixed:** `/agent-estate` targets bare `http://127.0.0.1:8769` (page had `.../agent-estate-role-dashboard.html`); a tailnet-only `:9443` → `127.0.0.1:5173` Serve was undocumented. CT201's live routes are 8085-based plus `/steel` (5173) and `/steel-api` (3000) — the old `:8443` → `localhost:8082` Homepage row is wrong. → [[infrastructure/tailscale]] - **Tailscale route drift fixed:** `/agent-estate` targets bare `http://127.0.0.1:8769` (page had `.../agent-estate-role-dashboard.html`); a tailnet-only `:9443` → `127.0.0.1:5173` Serve was undocumented. CT201's live routes are 8085-based plus `/steel` (5173) and `/steel-api` (3000) — the old `:8443` → `localhost:8082` Homepage row is wrong. *(The `/steel` and `/steel-api` routes were deleted on 2026-10-29 — see [[systems/browser-backend]].)* → [[infrastructure/tailscale]]
- **Proxmox inventory rebuilt** from `pct list` + `qm list`: **CT475 `agentzero` and VM110 `omarchy` were missing entirely**; 282 `ubuntu-server-xfce` corrected to stopped; dead rows removed (310, 500, 104, 106, 101). **VM403 `open-claw` and VM401 `hermes-old` are absent from `qm list`** — flagged unconfirmed, not deleted. → [[infrastructure/proxmox]] - **Proxmox inventory rebuilt** from `pct list` + `qm list`: **CT475 `agentzero` and VM110 `omarchy` were missing entirely**; 282 `ubuntu-server-xfce` corrected to stopped; dead rows removed (310, 500, 104, 106, 101). **VM403 `open-claw` and VM401 `hermes-old` are absent from `qm list`** — flagged unconfirmed, not deleted. → [[infrastructure/proxmox]]
- **Nanobot verified on CT333:** `nanobot-gateway` (PID 2259048) alive on 8765/8766. It is **not** a systemd `nanobot.service`/`gateway.service` (both probes return `inactive` — wrong unit names); it runs as the `nanobot` user service plus the gateway process. Port **8900 is no longer listening** — previous "Serve API" claim flagged unverified. → [[agents/nanobot]] - **Nanobot verified on CT333:** `nanobot-gateway` (PID 2259048) alive on 8765/8766. It is **not** a systemd `nanobot.service`/`gateway.service` (both probes return `inactive` — wrong unit names); it runs as the `nanobot` user service plus the gateway process. Port **8900 is no longer listening** — previous "Serve API" claim flagged unverified. → [[agents/nanobot]]
- **Docker services page corrected:** the old "Docker runtime not available" line was a CT460 socket artifact (`/var/run/docker.sock` is `root:docker srw-rw----`; `hermes` user not in group `docker`; no passwordless sudo in cron). Real fleet: CT200 holds ~50 containers, CT333 holds 9. → [[infrastructure/docker-services]] - **Docker services page corrected:** the old "Docker runtime not available" line was a CT460 socket artifact (`/var/run/docker.sock` is `root:docker srw-rw----`; `hermes` user not in group `docker`; no passwordless sudo in cron). Real fleet: CT200 holds ~50 containers, CT333 holds 9. → [[infrastructure/docker-services]]
@@ -405,7 +1018,7 @@ _The manual 2026-10-10 stocktake entry follows beneath this one._
- **Cron fleet: the worst day since the repin.** 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2) — all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, eight still open; `1a8e90ea2271` sits at `failure_streak: 2`. → [[incidents/2026-10-09-cron-provider-timeout-storm]] - **Cron fleet: the worst day since the repin.** 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2) — all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, eight still open; `1a8e90ea2271` sits at `failure_streak: 2`. → [[incidents/2026-10-09-cron-provider-timeout-storm]]
- **The 2026-10-08 audit window was lost** to the same timeout (this job's own 23:59 run); the last committed audit covered 2026-10-07. The 2026-10-09 run covers both days. → [[log]] - **The 2026-10-08 audit window was lost** to the same timeout (this job's own 23:59 run); the last committed audit covered 2026-10-07. The 2026-10-09 run covers both days. → [[log]]
- **`62619aa6bcb3` (DHL Tracking — Parcel 8052838382) is still enabled** even though Anthony instructed its deletion on 2026-10-09 21:24 and the session reported it deleted; it runs at 08:00 daily. Retirement is open item #1 in [[systems/scheduled-tasks]]. - **`62619aa6bcb3` (DHL Tracking — Parcel 8052838382) is still enabled** even though Anthony instructed its deletion on 2026-10-09 21:24 and the session reported it deleted; it runs at 08:00 daily. Retirement is open item #1 in [[systems/scheduled-tasks]].
- **Memory recall relevance incident** — a stale conversational episode outranked current-state facts; capture and vector health verified (760/760 working, 1,043/1,047 episodic), ranking fix not persisted in config. → [[incidents/2026-10-09-memory-recall-relevance]] - **Memory recall relevance incident — ✅ RESOLVED 2026-11-01.** A stale conversational episode had outranked current-state facts on an open question. The ranking fix **was** persisted — the weights live in `~/.hermes/mnemosyne/config.yaml` (`vec_weight: 0.5`, `fts_weight: 0.3`, `importance_weight: 0.3`), not in the `~/.hermes/config.yaml` block the original probe read. Live tier counts: working **941** (embeddings **927/941, 98.5 %**), episodic **1,124** (embeddings **1,124/1,124, 100 %**), canonical **90**, conflicts **175** with **11 open** (extraction noise). The original regression query no longer surfaces the stale episode. → [[incidents/2026-10-09-memory-recall-relevance]]
- **Mnemosyne live counts (2026-10-09 22:27):** 773 working / 1,047 episodic / 97 canonical slots / 871 facts / 152 conflicts (1 open) / 760 working embeddings / 1,043 episodic embeddings. → [[systems/mnemosyne]] - **Mnemosyne live counts (2026-10-09 22:27):** 773 working / 1,047 episodic / 97 canonical slots / 871 facts / 152 conflicts (1 open) / 760 working embeddings / 1,043 episodic embeddings. → [[systems/mnemosyne]]
## Recent Changes (2026-10-08) ## Recent Changes (2026-10-08)
@@ -498,7 +1111,7 @@ _The manual 2026-10-10 stocktake entry follows beneath this one._
- **WebUI connected platforms:** local, api_server, discord, email, homeassistant, mattermost, ntfy, slack, telegram, webhook — Verified, WebUI session, 2026-07-22 - **WebUI connected platforms:** local, api_server, discord, email, homeassistant, mattermost, ntfy, slack, telegram, webhook — Verified, WebUI session, 2026-07-22
## Recent Changes (2026-08-02) ## Recent Changes (2026-08-02)
- **Browser backend reverted:** Camoufox restored as primary on 2026-07-30. Steel Browser (CT201) experiment ended — CDP issues + DeepSeek 0731 tool name validation. `browser.cloud_provider: camofox`, `camera_provider: camofox`. Steel plugin disabled. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]]. - **Browser backend reverted:** Camoufox restored as primary on 2026-07-30. Steel Browser (CT201) experiment ended — CDP issues + DeepSeek 0731 tool name validation. `browser.cloud_provider: camofox`, `camera_provider: camofox`. Steel plugin disabled *(the 2026-08-02 state — the plugin and the Steel deployments were later removed entirely on 2026-10-29, see [[systems/browser-backend]])*. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]].
- **Mnemosyne capture fix (2026-08-01):** Automatic conversational capture was broken by stringified YAML (`sync_roles: "['user']"`). Fixed by rewriting structured Mnemosyne settings with native YAML types. See [[incidents/2026-08-01-mnemosyne-capture-failure]]. - **Mnemosyne capture fix (2026-08-01):** Automatic conversational capture was broken by stringified YAML (`sync_roles: "['user']"`). Fixed by rewriting structured Mnemosyne settings with native YAML types. See [[incidents/2026-08-01-mnemosyne-capture-failure]].
- **Mnemosyne vector coverage gap discovered:** Only 159/1,476 working memories (10.8%) have vector embeddings. 0/458 episodic memories have embeddings. Semantic search is mostly FTS5 fallback. - **Mnemosyne vector coverage gap discovered:** Only 159/1,476 working memories (10.8%) have vector embeddings. 0/458 episodic memories have embeddings. Semantic search is mostly FTS5 fallback.
- **DeepSeek V4 Flash 0731:** Post-training bump, same model ID, same pricing. Terminal Bench 2.1: 82.7 (up from 61.8). Stricter tool name validation (`^[a-zA-Z0-9_-]+$`) — may cause issues with OmniRoute-transformed tool names. - **DeepSeek V4 Flash 0731:** Post-training bump, same model ID, same pricing. Terminal Bench 2.1: 82.7 (up from 61.8). Stricter tool name validation (`^[a-zA-Z0-9_-]+$`) — may cause issues with OmniRoute-transformed tool names.
+13 -6
View File
@@ -3,8 +3,8 @@ title: Decisions
type: collection type: collection
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-11-20
verified_on: 2026-10-20 verified_on: 2026-11-20
confidence: high confidence: high
tags: [decisions] tags: [decisions]
--- ---
@@ -19,15 +19,22 @@ Stable strategic and config decisions, with evidence and status.
- [[decisions/persistent-shell-enabled]] - [[decisions/persistent-shell-enabled]]
- [[decisions/rollback-steel-to-camofox]] - [[decisions/rollback-steel-to-camofox]]
> **Status sweep 2026-10-20:** all three listed decisions are still `status: active`, but two now > **Status sweep 2026-10-20:** all three listed decisions are still `status: active`, but each now
> carry a review note: > carries a note:
> - [[decisions/keep-openrouter-as-fallback]] — its 30-day revisit trigger is **met**; OpenRouter is > - [[decisions/keep-openrouter-as-fallback]] — its 30-day revisit trigger is **met**; OpenRouter is
> reachable only through the OmniRoute catalog, not as a standalone provider. > reachable only through the OmniRoute catalog, not as a standalone provider.
> - [[decisions/persistent-shell-enabled]] — still `true`; its `review_after: 0001-01-01` zero-date > - [[decisions/persistent-shell-enabled]] — still `true`; its `review_after: 0001-01-01` zero-date
> bug is now corrected to a real date. > bug is now corrected to a real date.
> - [[decisions/rollback-steel-to-camofox]] — the *engine* is Camoufox as decided, but > - [[decisions/rollback-steel-to-camofox]] — ~~the *engine* is Camoufox as decided, but
> `cloud_provider: steel` remains in the config while the Steel service itself is unreachable. > `cloud_provider: steel` remains in the config while the Steel service itself is unreachable.~~
> **Superseded 2026-10-29:** `cloud_provider` is `camofox` again and Steel is **fully removed** —
> plugin deleted, CT201 and CT333 deployments torn down, Serve routes deleted. Camofox is the sole
> backend.
> **Status sweep 2026-11-20:** all three decisions still `status: active` and re-verified against live
> state — `keep-openrouter-as-fallback` and `persistent-shell-enabled` were re-verified 2026-11-11
> (`terminal.persistent_shell: true` re-confirmed live today); `rollback-steel-to-camofox`'s
> 2026-10-29 update re-probed today: `cloud_provider: camofox`, Steel absent estate-wide.
## History ## History
- **Verified:** Decision pages initialized during wiki setup on 2026-07-22 - **Verified:** Decision pages initialized during wiki setup on 2026-07-22
- **Inferred:** Config version 33 and multiple named backups imply iterative changes in vision, browser, and model routing - **Inferred:** Config version 33 and multiple named backups imply iterative changes in vision, browser, and model routing
+10 -2
View File
@@ -3,9 +3,9 @@ title: Keep OpenRouter as fallback
type: decision type: decision
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-11-11
review_after: 2026-10-20 # was 0001-01-01 (bug), fixed during 2026-07-22 audit; real trigger is conditional, see Revisit when review_after: 2026-10-20 # was 0001-01-01 (bug), fixed during 2026-07-22 audit; real trigger is conditional, see Revisit when
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: high confidence: high
tags: [routing, provider, decision] tags: [routing, provider, decision]
sources: [] sources: []
@@ -50,6 +50,14 @@ A self-hosted router has operated reliably for at least 30 days.
> bursts, and the 2026-10-18 degenerate-stream delivery. A single-node fallback posture carries > bursts, and the 2026-10-18 degenerate-stream delivery. A single-node fallback posture carries
> that concentration risk with it. > that concentration risk with it.
> **Re-verified 2026-11-11 — the decision still needs the re-affirmation it was flagged for on
> 2026-10-20, and the position is unchanged.** `custom_providers` still holds exactly
> `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101` — **no standalone
> OpenRouter provider**. OmniRoute remains the primary path (`smart-route` on
> `custom:omni-101`), so the "revisit trigger" (a self-hosted router running reliably >30 days) is
> still met and the literal reading of the decision is still unsatisfied. Nothing new to decide
> here — the page still says what needs a human call.
## Related ## Related
- [[model-routing]] - [[model-routing]]
- [[systems/omniroute]] - [[systems/omniroute]]
+8 -2
View File
@@ -3,9 +3,9 @@ title: Persistent shell enabled
type: decision type: decision
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-11-11
review_after: 2026-10-20 review_after: 2026-10-20
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: high confidence: high
tags: [automation, shell, decision] tags: [automation, shell, decision]
sources: [] sources: []
@@ -34,5 +34,11 @@ Use case becomes critical again.
> is torn down after 5 minutes idle, so "state persists across turns" is true within a burst of > is torn down after 5 minutes idle, so "state persists across turns" is true within a burst of
> work but not indefinitely. See [[concepts/persistent-shells]]. > work but not indefinitely. See [[concepts/persistent-shells]].
> **Re-verified 2026-11-11 — decision still `active`, setting still `true`.**
> `terminal.persistent_shell: true`, `backend: local`, `timeout: 180`, `home_mode: auto`,
> `container_cpu: 1`, `lifetime_seconds: 300`, `cwd: /home/hermes/workspace/`. The 5-minute idle
> teardown caveat recorded on 2026-10-20 still applies: state persists across a burst of work, not
> indefinitely.
## Related ## Related
- [[persistent-shells]] - [[persistent-shells]]
+65 -13
View File
@@ -3,8 +3,8 @@ title: Rollback Steel to Camoufox
type: decision type: decision
status: active status: active
created: 2026-08-02 created: 2026-08-02
updated: 2026-10-20 updated: 2026-11-20
verified_on: 2026-10-20 verified_on: 2026-11-20
confidence: high confidence: high
tags: [decision, browser, steel, camofox, rollback] tags: [decision, browser, steel, camofox, rollback]
sources: [session-history, live-config] sources: [session-history, live-config]
@@ -28,23 +28,29 @@ Steel Browser was deployed on CT201 on 2026-07-29 as the primary browser automat
## Decision ## Decision
**Revert to Camoufox as primary.** Steel plugin disabled but left running on CT201. **Revert to Camoufox as primary.** Steel plugin disabled but left running on CT201.
> **Status check 2026-10-20 — the decision is *partially* undone in configuration.** > ⚠ **Superseded 2026-10-29:** the "left running on CT201" half of this decision no longer holds — Steel is
> The live `browser:` block in `~/.hermes/config.yaml` reads > now **fully removed** from the estate. See the Update below.
> `cloud_provider: steel` with `engine: camofox`. The **engine** is Camoufox (decision honoured),
> but the **cloud_provider** key still says `steel`, and `camera_provider: camofox`. > **Status check 2026-10-20 — the decision is *partially* undone in configuration.** *(superseded 2026-10-29)*
> The live `browser:` block in `~/.hermes/config.yaml` read
> `cloud_provider: steel` with `engine: camofox`. The **engine** was Camoufox (decision honoured),
> but the **cloud_provider** key said `steel`, and `camera_provider: camofox`.
> This is the same split seen in [[comparisons/camoufox-vs-chrome]]. > This is the same split seen in [[comparisons/camoufox-vs-chrome]].
> >
> Meanwhile the Steel service itself is **not reachable**: `http://steel.martinwa.org` returns > Meanwhile the Steel service itself was **not reachable**: `http://steel.martinwa.org` returned
> **000** (connection failed) on 2026-10-20. Combined with the degraded CT201 stack recorded in > **000** (connection failed) on 2026-10-20. Combined with the degraded CT201 stack recorded in
> [[systems/homepage-dashboard]] (main container removed, asset proxy 502), it looks like the > [[systems/homepage-dashboard]] (main container removed, asset proxy 502), it looked like the
> "left running on CT201" part of this decision is no longer true. > "left running on CT201" part of this decision was no longer true.
> >
> **Recommendation, not actioned here:** the `cloud_provider: steel` value appears to be residue > **Recommendation, not actioned here:** the `cloud_provider: steel` value appeared to be residue
> from the reverted deployment. Confirm against the live Steel service and either restore it or > from the reverted deployment. Confirm against the live Steel service and either restore it or
> set `cloud_provider: camofox` so the config matches the decision this page records. > set `cloud_provider: camofox` so the config matches the decision this page records.
>
> **Resolved 2026-10-29:** `cloud_provider` is `camofox` again and Steel has been removed entirely — no
> restoration was needed or wanted.
## Rationale ## Rationale
1. **Reliability:** Camoufox runs on the same host as Hermes (CT450) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues. 1. **Reliability:** Camoufox runs on the same host as Hermes (~~CT450~~ **CT460** — host attribution corrected 2026-11-20; the service is the local `camofox-browser.service` on the production Hermes host) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues.
2. **Auth persistence:** Camoufox's persistent profile `anthony` is already authenticated into key services (Uber Eats, LinkedIn). Steel required cookie re-exports. 2. **Auth persistence:** Camoufox's persistent profile `anthony` is already authenticated into key services (Uber Eats, LinkedIn). Steel required cookie re-exports.
3. **Failure mode:** The `opencode-go/deepseek-v4-flash` empty-stream errors were hitting all providers, but Steel added an extra failure surface (CDP connection, plugin layer) on top of the same problem. 3. **Failure mode:** The `opencode-go/deepseek-v4-flash` empty-stream errors were hitting all providers, but Steel added an extra failure surface (CDP connection, plugin layer) on top of the same problem.
4. **DeepSeek 0731 compatibility:** The new model's stricter tool name validation (`^[a-zA-Z0-9_-]+$`) affected all providers — not a Steel-specific issue, but another reason to minimise the toolchain. 4. **DeepSeek 0731 compatibility:** The new model's stricter tool name validation (`^[a-zA-Z0-9_-]+$`) affected all providers — not a Steel-specific issue, but another reason to minimise the toolchain.
@@ -52,13 +58,59 @@ Steel Browser was deployed on CT201 on 2026-07-29 as the primary browser automat
## Consequences ## Consequences
- **Positive:** Browser automation is stable again. Uber Eats report ran successfully via Camofox on 2026-08-01. - **Positive:** Browser automation is stable again. Uber Eats report ran successfully via Camofox on 2026-08-01.
- **Negative:** Steel's session viewer UI and full CDP access are no longer available to Hermes. - **Negative:** Steel's session viewer UI and full CDP access are no longer available to Hermes.
- **Neutral:** Steel still runs on CT201 — can be re-enabled by uncommenting `.env` lines and enabling the plugin. - **Neutral (2026-07-30 — now superseded):** Steel still ran on CT201 and could be re-enabled by uncommenting `.env` lines and enabling the plugin. **As of 2026-10-29 that is no longer true** — Steel is removed, so re-enabling means redeploying it (compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`; images remain re-pullable from ghcr.io).
## Evidence ## Evidence
- Camoufox verified working Jul 30: `browserConnected: true`, `sessions: 1`, `tabs: 1`, active 200s - Camoufox verified working Jul 30: `browserConnected: true`, `sessions: 1`, `tabs: 1`, active 200s
- Uber Eats report (Aug 1) successfully scraped via Camofox REST API — 11 orders, $264.51 - Uber Eats report (Aug 1) successfully scraped via Camofox REST API — 11 orders, $264.51
- Steel plugin `browser-steel` still present at `/home/hermes/.hermes/plugins/browser-steel/` — disabled - Steel plugin `browser-steel` was present at `/home/hermes/.hermes/plugins/browser-steel/` — disabled *(2026-07-30 state; the plugin has since been deleted — see the Update below)*
## Update (2026-10-29) — Steel was fully removed, not just darkened
The 2026-07-30 rollback disabled the Steel plugin but left it on disk and left Steel running. Both of those
were closed on **2026-10-28 22:00 – 2026-10-29 00:15**: `browser.cloud_provider` / `camera_provider` were
returned to **`camofox`** (backups `config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z`,
`config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z`), and **`~/.hermes/plugins/browser-steel/` was
removed**, so the `plugin_steel` toolset is no longer selectable.
**Steel is now FULLY REMOVED — it is not merely "down" or "not reachable":**
- **Hermes config:** the `browser:` block reads exactly `cloud_provider: camofox`, `camera_provider: camofox`,
`engine: auto`, `cdp_url: ""`. No `browser-steel` key remains in `plugins.disabled` or `plugins.entries`, and
`STEEL_BASE_URL` is gone from `~/.hermes/.env` (it survived only in the inherited environment of processes
already running at the cleanup; the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it, while a
couple of older, unrelated services still carry it — **no core Hermes code reads it**, so it is a
runtime-only leftover, not a config value).
- **Hermes plugins:** `~/.hermes/plugins/browser-steel/` is deleted. Hermes core source contains **zero**
Steel references, so there is **no Steel fallback path in the tree** — returning to Steel means redeploying
the plugin, not flipping a flag.
- **CT201 (runtipi):** containers `steel-browser-api-1` and `steel-browser-ui-1`, the `steel-network` Docker
network, the images `steel-browser-api:fixed` / `ghcr.io/steel-dev/steel-browser-api:latest` /
`ghcr.io/steel-dev/steel-browser-ui:latest`, and the project dir `/root/steel-browser` were all removed
(~6 GB reclaimed; disk 70G used → 64G used). The Tailscale Serve routes `/steel` and `/steel-api` were
deleted with `tailscale serve --set-path <path> off`; CT201's remaining routes are only `/` → `127.0.0.1:8085`
and `/mgr` → `127.0.0.1:8084` on both `:443` and `:8443`. Nothing listens on 3000, 5173 or 9223.
- **CT333 (clawtest):** a second, separate and orphaned Steel install — the `steel-browser` container
(`ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image
were removed as well. It had no Tailscale Serve route, no entry in `~/.zeroclaw/config.toml` (which uses
`backend = "agent_browser"`), and no reference from Nanobot — its last CDP navigation was 2026-10-20 and it
received no requests after 2026-10-21.
- **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, plus
`/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io.
- **Camofox is the sole backend** and healthy — REST `http://100.118.5.51:9377` (also `http://localhost:9377`),
`/health` → **200**, `/tabs` → `{"running":true,"tabs":[]}`.
The decision itself is unchanged — Camofox is the backend — but the "Steel is still there if you need it"
assumption is dead. Detail: [[systems/browser-backend]], [[comparisons/steel-vs-camoufox]].
> **Re-verified 2026-11-20 (stocktake).** Every removal claim in the 2026-10-29 update re-probed:
> config `cloud_provider/camera_provider: camofox`, `engine: auto`; plugin dir absent; CT201 steel
> containers/images/network/project dir gone, Serve only `/`→8085 + `/mgr`→8084 on :443/:8443,
> nothing on 3000/5173/9223; CT333 steel gone, :3000 closed; `/root/_archived-steel/` archive intact
> (compose, Dockerfile.api, cache tgz). Camofox `:9377/health` → `ok` (idle). **Note:** the Rationale's
> "Camoufox runs on the same host as Hermes (CT450)" reflects the July-era attribution — the service
> runs on **CT460** (the production Hermes host), corrected in [[systems/browser-backend]] 2026-10-13.
> The decision and its consequences are unchanged.
## Related ## Related
- [[comparisons/steel-vs-camoufox]] - [[comparisons/steel-vs-camoufox]]
- [[systems/browser-backend]] - [[systems/browser-backend]]
+9 -2
View File
@@ -3,8 +3,8 @@ title: Experiments
type: collection type: collection
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-11-11
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: medium confidence: medium
tags: [experiments] tags: [experiments]
--- ---
@@ -40,6 +40,13 @@ Non-canonical tests, tuning passes, and change attempts.
> See [[incidents/2026-09-28-hermes-managed-runtime-deps]] and the runtime-deps notes in > See [[incidents/2026-09-28-hermes-managed-runtime-deps]] and the runtime-deps notes in
> [[systems/hermes-agent]]. > [[systems/hermes-agent]].
> **Backups tree re-listed 2026-11-11 — no change to report.** `~/.hermes/backups/` still holds the
> same dated incident directories named on 2026-10-20 (including
> `mnemosyne-capture-fix-20260801_180249/` and `legacy-memory-slim-20260801_180702/`, both verified
> present again during this pass), plus the rolling `config/` series. The three experiment
> snapshots this page originally listed (`pre-vision-fix-20260718-195712`, `pre-model-fix`,
> `pre-camofox-tuning`) are still gone.
## Outcomes ## Outcomes
- **Verified:** Backups exist with the above names - **Verified:** Backups exist with the above names
- **Unknown:** Whether each experiment succeeded, failed, or was reverted - **Unknown:** Whether each experiment succeeded, failed, or was reverted
+20 -4
View File
@@ -3,9 +3,8 @@ title: 'Mnemosyne Relay Scope Corruption + `tools: None` — 2026-07-29'
type: incident type: incident
status: resolved status: resolved
created: 2026-07-29 created: 2026-07-29
updated: 2026-10-24 updated: 2026-11-12
verified_on: 2026-10-24 verified_on: 2026-11-12
review_after: 2026-11-07
confidence: high confidence: high
tags: [mnemosyne, config, relay, incident, resolved] tags: [mnemosyne, config, relay, incident, resolved]
sources: [live-probe, config, session-20260729] sources: [live-probe, config, session-20260729]
@@ -51,4 +50,21 @@ Scope push/pop ordering mismatch. Something pushes a scope that isn't at the top
Filed (or should be filed) under NousResearch/hermes-agent issues. No token available to auto-create. Filed (or should be filed) under NousResearch/hermes-agent issues. No token available to auto-create.
## MONITORING ## MONITORING
Watch for recurrence. If persists, check whether the affected sessions share a common tool call pattern (subagent delegation, tool failure, etc.). Watch for recurrence. If persists, check whether the affected sessions share a common tool call pattern (subagent delegation, tool failure, etc.).
## Re-verified 2026-11-12 03:3x AWST (stocktake — page was 19d old, inside the staleness window)
Both halves of this page re-checked against live state and **the fix holds; neither symptom has recurred**
in ~3.5 months and 106 days of logs:
- `grep -n "tools:" ~/.hermes/config.yaml` → the `memory.mnemosyne` block no longer carries `tools: None`
(the literal `None` key is absent), so the `Unknown Mnemosyne tool(s)… None` schema warning has no
trigger left.
- `grep -c "Unknown Mnemosyne tool" ~/.hermes/logs/gateway.log` and `.log.1`–`.log.3` → **0 in every
rotated file** (4 generations, 0 occurrences).
- `grep -ah "scope handle is not at the top" ~/.hermes/logs/gateway.log*` → **no occurrences**; the
relay-scope corruption has not repeated since 2026-07-29 08:10 UTC.
- Mnemosyne itself is healthy on the current build: `mnemosyne stats` → **2782 memories (1062 working /
1217 episodic)** and `mnemosyne recall "tailscale serve routes"` returns substantive ranked results, so
the relay/memory path is functioning.
The `REPORTED` action above still stands (no token to auto-create a NousResearch issue); the **MONITORING**
condition is now satisfied — no recurrence across four log generations.
@@ -3,9 +3,9 @@ title: Mnemosyne Automatic Capture Failure
type: incident type: incident
status: active status: active
created: 2026-08-01 created: 2026-08-01
updated: 2026-10-20 updated: 2026-11-11
resolved_on: 2026-10-20 resolved_on: 2026-10-20
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: high confidence: high
tags: [memory, mnemosyne, hermes, incident] tags: [memory, mnemosyne, hermes, incident]
sources: [live-config, mnemosyne-db, hermes-cli-e2e] sources: [live-config, mnemosyne-db, hermes-cli-e2e]
@@ -82,6 +82,14 @@ no Mnemosyne trace.
`~/.hermes/backups/mnemosyne-capture-fix-20260801_180249/` and `~/.hermes/backups/mnemosyne-capture-fix-20260801_180249/` and
`~/.hermes/backups/legacy-memory-slim-20260801_180702/`. `~/.hermes/backups/legacy-memory-slim-20260801_180702/`.
> **Resolution re-confirmed 2026-11-11.** The whole `memory.mnemosyne` block in
> `~/.hermes/config.yaml` is still natively typed — `sync_roles: [user]` (list),
> `skip_contexts: [cron, flush, subagent, background, skill_loop]` (list),
> `reflect: {disabled_for_cron: true, max_calls_per_session: 3}` (dict), `sleep_threshold: 50`
> (int), `ignore_patterns: []` (list), `profile_isolation: true` (bool), `auto_sleep: true`,
> `default_scope: session`, `vector_type: int8`. No stringification regression. Both backup
> directories named in the repair still exist. `status: resolved` stands.
## Related ## Related
- [[systems/mnemosyne]] - [[systems/mnemosyne]]
- [[systems/memory-backend]] - [[systems/memory-backend]]
@@ -3,9 +3,8 @@ title: OmniRoute free-DeepSeek Silent Stops (Empty-After-Tool) — 2026-08-04
type: incident type: incident
status: monitoring status: monitoring
created: 2026-08-04 created: 2026-08-04
updated: 2026-10-24 updated: 2026-11-12
verified_on: 2026-10-24 verified_on: 2026-11-12
review_after: 2026-11-07
confidence: high confidence: high
tags: [omniroute, deepseek, free-tier, silent-failure, incident, mitigated] tags: [omniroute, deepseek, free-tier, silent-failure, incident, mitigated]
sources: [live-probe, omniroute-log, session-20260804] sources: [live-probe, omniroute-log, session-20260804]
@@ -53,5 +52,23 @@ Backup of both pre-change rows: `/tmp/combo_backup_20260804.json` on the Hermes
- Consider `auto`+explorationRate for self-tuning if free tier stays flaky. - Consider `auto`+explorationRate for self-tuning if free tier stays flaky.
- Hermes-side: empty-post-tool turns are nudge/retry/fallback handled, but still end silently once exhausted — possible future improvement: surface "(empty after tool calls)" more visibly. - Hermes-side: empty-post-tool turns are nudge/retry/fallback handled, but still end silently once exhausted — possible future improvement: surface "(empty after tool calls)" more visibly.
## Re-verified 2026-11-12 03:4x AWST (stocktake — page was 19d old, inside the staleness window)
Both reworked combos probed live on **`omniroute-1`** (`100.78.63.20:20128`, `custom:omni-101`) with the
correct `key_env` (`HERMES_CUSTOM_100_78_63_20_20128_API_KEY`) and the exact aliases used:
| Probe | Result |
|---|---|
| `good-free` (the 12-model rework) | **HTTP 200**, resolving to **`glm-5.2-cheap:free`** with a valid non-empty completion |
| `free-stack` | **HTTP 200**, resolving to **`stepfun-ai/step-5-preview`** with a valid completion |
**The mitigation is holding, and the 2026-08-04 follow-up can be partially closed:** both combos answer with
real content, and **neither resolves to a deepseek model** — which is exactly the intent of dropping the
proven culprits from `good-free`. ⚠ **What is *not* re-verified:** I did not individually probe the
free-DeepSeek upstreams (`238d5978/deepseek-v4-flash`, `nvidia/deepseek-ai/deepseek-v4-pro`,
`deepseek-v4-flash-thinking:free`), so the original claim *"upstream free-DeepSeek tier still degraded"*
stands **neither confirmed nor refuted** — the combos serve healthy non-deepseek free models, which may mean
the upstream recovered *or* simply that it is no longer reached. **Restoring deepseek-first ordering is
therefore still not justified** and should wait on a direct probe of those three upstreams.
## Refs ## Refs
- [[systems/omniroute]] - [[systems/omniroute]]
@@ -1,18 +1,23 @@
--- ---
title: Hermes Config Backup to Gitea — Push Has Been Dead Since 2026-08-08 While the Job Reads `ok` title: Hermes Config Backup to Gitea — Push Was Dead 2026-08-08 → 2026-11-07 While the Job Read `ok`
type: incident type: incident
status: active status: resolved
created: 2026-10-21 created: 2026-10-21
updated: 2026-10-22 updated: 2026-11-07
verified_on: 2026-10-22 resolved_on: 2026-11-07
review_after: 2026-10-29 verified_on: 2026-11-07
review_after: 2027-02-07
confidence: high confidence: high
tags: [backup, gitea, cron, git, green-vs-actual] tags: [backup, gitea, cron, git, green-vs-actual]
sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki] sources: [live-git-config-backup, live-cron-backup-log, live-jobs-json, live-git-wiki, gitea-api, github-api, sandbox-e2e-test]
--- ---
# Hermes Config Backup to Gitea — Off-Box Config Backup Dead Since 2026-08-08 # Hermes Config Backup to Gitea — Off-Box Config Backup Dead Since 2026-08-08
> **RESOLVED 2026-11-07.** The push is live again, the backlog is drained, and the script now exits
> non-zero on failure so the cron row cannot lie. See [Resolution 2026-11-07](#resolution-2026-11-07)
> at the foot of this page for the full record. The original investigation is preserved below.
## Headline ## Headline
The daily **Hermes Config Backup to Gitea** job (`e27da6c34bbb`, `0 11 * * *`, no-agent, `hermes-config-backup.sh`) The daily **Hermes Config Backup to Gitea** job (`e27da6c34bbb`, `0 11 * * *`, no-agent, `hermes-config-backup.sh`)
@@ -103,3 +108,92 @@ change as well the *only* off-box copy path was this dead push → [[infrastruct
- [[infrastructure/backups]] - [[infrastructure/backups]]
- [[systems/scheduled-tasks]] - [[systems/scheduled-tasks]]
- [[systems/hermes-agent]] - [[systems/hermes-agent]]
---
# Resolution 2026-11-07
Anthony asked to fix the failing push (and to check the Notion vault for the API key). Resolved in
one session: the push is live, the backlog is drained, and the script can no longer report a false green.
## Root causes (four, all independent)
1. **No Gitea credential.** `origin` was HTTPS with no userinfo, and the global `credential.helper=store`
file `~/.git-credentials` held only a `github.com` entry — so the cron worker could not authenticate
non-interactively. The token in the Notion vault was valid and identical to the one the wiki repo
carries in its remote URL; it simply was never wired into the backup repo.
2. **`backup.sh` had no `exit` statement at all.** It always returned 0, so the scheduler
(`success = (returncode == 0)` in `cron/scheduler_script.py`) recorded `last_status: ok` on every
failed night. This is why the outage went unnoticed for 92 days — it was a *false green*, not a
missing signal. (The incident's earlier note blamed the scheduler; the scheduler was correct. The
script was the liar.)
3. **A 1.76 GB `checkpoints/` backlog** sat in the unpushed range — 13 blobs over 100 MB, all junk
that `.gitignore` already excluded for new commits but that history still carried.
4. **Two broken secondary legs.** The `github` remote did not exist, and the fallback
`_github_clean_push` pushed a branch's full ancestry — which contains a 7.45 GB tarball blob, so
GitHub's 100 MB file limit could never accept it. The AgentMail alert called a
`python3 send_email.py` that no longer exists, via a `chaos-agent@agentmail.to` inbox that no
longer exists.
## Fix applied
- **Credential:** stored a host-scoped Gitea entry via `git credential approve` (not embedded in the
remote URL). `git ls-remote origin` now authenticates with `GIT_TERMINAL_PROMPT=0`.
- **Backlog:** `git filter-repo --partial --refs origin/main..main --path checkpoints --invert-paths
--prune-empty never --force` stripped `checkpoints/` from the unpushed range only (so `origin/main`
was untouched and the push stayed a fast-forward), then `checkpoints/` was untracked at the tip.
Pre-rewrite history is preserved under the tag `backup/pre-rewrite-20261107-032658`.
**Push set: ~2 GB → 264.8 MB.** The 7.45 GB tarball was already on `origin/main` and was never resent.
- **Script hardening** (`~/.hermes/config-backup/backup.sh`, rewritten via OpenCode + a correction pass):
- exits **0 only on verified success**, `1` otherwise — the cron row can no longer lie;
- `.last_backup_ok` is stamped **only** after the remote is confirmed to hold the commit;
- the push decision compares **remote vs local**, not "were there new working-tree changes" — a
local-ahead-of-remote with a clean tree now still pushes (this is exactly the mechanism that
created the 92-commit backlog);
- every `git ls-remote` is guarded with `|| true`, so an unreachable remote reaches the alert path
instead of killing the script at exit 128 before `notify_failure` fires;
- the GitHub leg now pushes a **parentless orphan snapshot** built from `HEAD^{tree}` via
`git commit-tree` — no history, no >100 MB blob, HEAD/branch/index untouched — and is a
warning-only secondary target that can never fail the job;
- the AgentMail leg uses the real CLI (`agentmail inboxes:messages send --inbox-id <addr>`) with the
key read from `.env` at runtime and passed via the environment, never argv;
- the Telegram leg uses `curl --fail` and requires `"ok":true` in the body, so it reports `sent`
only on a real delivery.
## Verification (live, 2026-11-07 AWST)
| Check | Result |
|---|---|
| Real `backup.sh` run | exit **0**, 47 s |
| `git rev-parse HEAD` vs `git ls-remote origin refs/heads/main` | identical — `a50a5170c1c0e2af5235fa63a95a732e2d780703` |
| `git rev-list --count origin/main..main` | **0** (was 92) |
| Gitea API repo `Anthony/hermes-config-backup` | HTTP 200, default branch `main`, updated `2026-11-07T05:10:46+08:00`, 8.1 MB |
| Gitea API `branches/main` | `a50a5170c1c0` — `🦝 nightly backup: 2026-11-07 05:10:33 AWST` |
| `.last_backup_ok` | refreshed to the run timestamp (was stale from 2026-11-06 11:00) |
| GitHub `main` | `faed9fb5c677`, **0 parents** (orphan snapshot confirmed), 30,086 files, largest blob 23.5 MB |
| GitHub `master` branch | untouched at `c99c94e6b13b` |
| GitHub force-push data loss | none — GitHub's tree was a strict **subset** of local HEAD (only `BACKUP-INDEX.md` added) |
| Token leakage scan (HEAD tree, working tree, all unpushed blobs) | **clean** — token absent everywhere |
| AgentMail leg (self-test to the agent's own inbox) | delivered, `message_id` returned |
| Telegram leg (`getMe`, no message sent to Anthony) | `"ok":true` detected → would report `sent` |
| Sandbox E2E: unreachable remote | exit **1**, `notify_failure` fired, health file **not** stamped |
| Sandbox E2E: local ahead of remote, clean working tree | pushed + verified, exit **0**, health stamped |
| Sandbox E2E: already up to date | no push, exit **0**, health stamped |
## Verification command set (for the next audit)
```
git -C ~/.hermes/config-backup log origin/main -1 --format='%ci %h %s'
git -C ~/.hermes/config-backup rev-list --count origin/main..main
tail -20 ~/.hermes/config-backup/backup.log
tail -3 ~/.hermes/config-backup/.backup_history
```
## Still open (not part of this incident)
- Gitea host SSH login is denied for `anthonymau` — free disk headroom on the Gitea box is still
unverified (the Gitea API reports repo size, not host free space).
- `hermes-gateway-runtime-health.service` still fails every 5 min because the Mattermost host
`100.98.116.14` has been offline since ~Sept 2026. Separate incident.
- The `checkpoints/` directory is still excluded from backups by policy; the live snapshots remain
at `~/.hermes/checkpoints/` on-box only.
+10 -5
View File
@@ -3,10 +3,10 @@ title: "Skills Index Drift — 4x Undercount"
type: incident type: incident
status: active status: active
created: 2026-08-15 created: 2026-08-15
updated: 2026-10-23 updated: 2026-11-05
resolved_on: 2026-08-15 resolved_on: 2026-08-15
reopened_on: 2026-10-23 reopened_on: 2026-10-23
verified_on: 2026-10-23 verified_on: 2026-11-05
confidence: high confidence: high
tags: [skills, wiki, drift, auto-regeneration] tags: [skills, wiki, drift, auto-regeneration]
sources: [session-investigation, agent-estate-wiki, live-filesystem-probe, cron-executions-db] sources: [session-investigation, agent-estate-wiki, live-filesystem-probe, cron-executions-db]
@@ -88,9 +88,14 @@ from the scan, prints the archived count as an explicit exclusion, and states th
filesystem inventory. The page was regenerated through the patched script and filesystem inventory. The page was regenerated through the patched script and
committed with the fix. committed with the fix.
**Still open:** **Still open (as of 2026-10-23):**
- The stale `~/.hermes/scripts/wiki-maintain.sh` copy must never be run — it produces - ~~The stale `~/.hermes/scripts/wiki-maintain.sh` copy must never be run — it produces
the divergent shape. It should be deleted or made to delegate to the repo script. the divergent shape. It should be deleted or made to delegate to the repo script.~~
**✅ CLOSED 2026-11-05** — the stale copy is now a **delegating wrapper** with no generator logic of its
own; it `exec`s the repo script. The repo script gained `--no-build` so the wrapper can honour the
interface its callers used without carrying a second copy of the generator. Verified by running the
wrapper: it now produces the canonical **302 skills / 68 categories** shape.
→ [[incidents/2026-11-05-skills-index-generator-divergence]]
- The page will still read higher than the loader's 329: nested collections - The page will still read higher than the loader's 329: nested collections
(`context-engineering-collection/…`), `apple/` and hub-installed trees are counted as (`context-engineering-collection/…`), `apple/` and hub-installed trees are counted as
installed files. Whether the page should instead report the loader's figure is a installed files. Whether the page should instead report the loader's figure is a
@@ -3,8 +3,8 @@ title: "Mnemosyne Capture Pipeline Broken (Aug 18)"
type: incident type: incident
status: superseded status: superseded
created: 2026-08-18 created: 2026-08-18
updated: 2026-10-12 updated: 2026-11-04
verified_on: 2026-10-12 verified_on: 2026-11-04
confidence: high confidence: high
tags: [memory, mnemosyne, hermes, incident, capture] tags: [memory, mnemosyne, hermes, incident, capture]
sources: [mnemosyne-db, health-monitor, run_agent.py] sources: [mnemosyne-db, health-monitor, run_agent.py]
@@ -68,6 +68,29 @@ embedding coverage **98.3 % working / 99.6 % episodic**, FTS aligned (`fts_worki
The three guard conditions diagnosed below were never individually confirmed as the fix — the path resumed The three guard conditions diagnosed below were never individually confirmed as the fix — the path resumed
capturing and has captured through every window since. capturing and has captured through every window since.
### Re-verified 2026-11-04 — capture still healthy, and the counts have moved on
Live state of `~/.hermes/mnemosyne/data/mnemosyne.db` (the path in the Verification snippet above is still
correct; note the DB is under `data/`, not the `~/.hermes/mnemosyne/*.db` glob path):
| Metric | 2026-10-12 (page) | 2026-11-04 (live) |
|---|---|---|
| `working_memory` | 786 | **970** |
| `episodic_memory` | 1,056 | **1,146** |
| `memoria_facts` | — | **446** |
| `memory_embeddings` | — | **943** |
| Working-memory vector coverage | 98.3 % | **98.6 %** (`vec_working` 956 / `working_memory` 970) |
| Episodic vector coverage | 99.6 % | **100 %** (`vec_episodes` 1,146 / `episodic_memory` 1,146) |
| `fts_working` | 786 | **970** — aligned with `working_memory` |
| `fts_facts` | 876 | **927** (vs `memoria_facts` 446 — FTS indexes a superset) |
| Last `working_memory` entry | 2026-10-11 | **2026-11-03 17:12:40** |
`mnemosyne stats` independently reports **Total 2,562 / Working 970 / Episodic 1,146**, and the `vec0` extension
loads cleanly (`sqlite-vec` from the agent venv), confirming vector search is live rather than silently FTS-only.
**Working-memory coverage is above the 95 % reindex threshold**, so no `mnemosyne reindex` was triggered by this
audit. **Capture has continued to work without incident for a full month** — the original failure has not
recurred.
**Caution for the next alarm:** the health monitor raised a false EXCEPTION on 2026-10-11 00:08 by comparing **Caution for the next alarm:** the health monitor raised a false EXCEPTION on 2026-10-11 00:08 by comparing
naive-UTC `created_at` against AWST `now` ("stalled 10.7 h" — real gap 2.7 h). Do not triage that as a repeat naive-UTC `created_at` against AWST `now` ("stalled 10.7 h" — real gap 2.7 h). Do not triage that as a repeat
of this incident; see [[systems/mnemosyne]]. of this incident; see [[systems/mnemosyne]].
@@ -3,8 +3,8 @@ title: Work Outlook Mailbox — Basic Auth Disabled (Himalaya)
type: incident type: incident
status: open status: open
created: 2026-10-03 created: 2026-10-03
updated: 2026-10-25 updated: 2026-11-12
verified_on: 2026-10-25 verified_on: 2026-11-12
confidence: high confidence: high
tags: [email, himalaya, office365, oauth2, auth, incident] tags: [email, himalaya, office365, oauth2, auth, incident]
sources: [live-himalaya-debug, live-tcp-probe, cron-session-20260826_180700_a94098] sources: [live-himalaya-debug, live-tcp-probe, cron-session-20260826_180700_a94098]
@@ -12,7 +12,16 @@ sources: [live-himalaya-debug, live-tcp-probe, cron-session-20260826_180700_a940
# Work Outlook Mailbox — Basic Auth Disabled (Himalaya) # Work Outlook Mailbox — Basic Auth Disabled (Himalaya)
**Window:** first detected 2026-08-26; re-verified 2026-10-03 and again 2026-10-25 (still open) **Re-verified 2026-11-12 03:3x AWST (stocktake — page was 18d old, inside the staleness window):**
the failure is unchanged and still reproduces exactly, with the **same error class**. `himalaya envelope
list -a work -s 1` → `Error: cannot build IMAP client / 1: login not supported` — i.e. the server rejects
the basic-auth `LOGIN` command itself rather than the credential, which is consistent with the SMTP/IMAP
basic-auth shutdown recorded above rather than a changed password. **No remediation has landed since
2026-10-25:** the account still authenticates by password (there is no OAuth2 token in the himalaya
config), so path 1 remains the only viable route and still requires Anthony's sign-in. The capability gap
described below is unchanged — no cron job polls this mailbox.
**Window:** first detected 2026-08-26; re-verified 2026-10-03, 2026-10-25 and 2026-11-12 (still open)
**Status:** Open — blocked pending an OAuth2 path **Status:** Open — blocked pending an OAuth2 path
**Impact:** The agent cannot read or send from the `work` mailbox. Every work-email **Impact:** The agent cannot read or send from the `work` mailbox. Every work-email
request is unanswerable until this is fixed. request is unanswerable until this is fixed.
+21 -2
View File
@@ -3,8 +3,8 @@ title: "Gmail Read-State Incident (2026-08-29)"
type: incident type: incident
status: active status: active
created: 2026-08-29 created: 2026-08-29
updated: 2026-10-20 updated: 2026-11-11
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: high confidence: high
tags: [email, gmail, cron, himalaya, incident] tags: [email, gmail, cron, himalaya, incident]
sources: [session:default/20260829_204103_9e1087] sources: [session:default/20260829_204103_9e1087]
@@ -80,6 +80,25 @@ Despite using `--preview`, Himalaya may still mark messages as read depending on
> **not** read-state errors — but the read-state exposure persists regardless, because the > **not** read-state errors — but the read-state exposure persists regardless, because the
> newsletter fetch path still issues `message read`. > newsletter fetch path still issues `message read`.
> >
> **Re-verified 2026-11-11 — the cron error state has cleared; the read-state root cause has not.**
>
> | Item | State on 2026-10-20 | State on 2026-11-11 |
> |---|---|---|
> | `goimapnotify.service` | active (PID 383) | **still active**, now PID **441**, same binary/conf path |
> | Important-mail monitor (`d40085da631c`) | paused | **still paused** — `enabled: False`, `last_status: ok` |
> | `fetch_newsletters.py` body-fetch call | `himalaya message read --preview …` | **identical, still unfixed** — the unsafe call is still in the file |
> | `85edf6772bca` Tech & AI Newsletter Digest | `error` | **`ok`**, `failure_streak: 0`, last run 2026-11-10 07:05 |
> | `08eb86f02d37` Gmail Action Digest | `error` | **`ok`**, `failure_streak: 0`, last run 2026-11-10 09:18 |
> | `06a0a1c8704f` AI Newsletter Curation for Patrick | `error` | **`ok`**, `failure_streak: 0`, last run 2026-11-05 10:41 |
>
> So 74 days after the incident: **both** original mitigations that were applied on 2026-08-29 are
> now **inert** — the monitor is still paused, but `goimapnotify` is back up, so IMAP IDLE is live
> again — and the root cause (`himalaya message read` on Anthony's INBOX) is untouched. The three
> email cron jobs that were in `error` on 2026-10-20 are now green, but that says nothing about
> the read-state exposure: a job that marks emails read and *succeeds* still pollutes the mailbox.
> Per the skill's own rule (a delivered/successful run is not evidence of correct behaviour), the
> exposure is best judged from the code path, not the cron status — and the code path is unchanged.
>
> **Recommended action (not taken here — this is a wiki job, and the fix touches Anthony's mail):** > **Recommended action (not taken here — this is a wiki job, and the fix touches Anthony's mail):**
> rewrite `fetch_newsletters.py` to use readonly IMAP, and decide whether `goimapnotify` should be > rewrite `fetch_newsletters.py` to use readonly IMAP, and decide whether `goimapnotify` should be
> running. See [[systems/scheduled-tasks]] and the Gmail IMAP lessons in > running. See [[systems/scheduled-tasks]] and the Gmail IMAP lessons in
+16 -2
View File
@@ -3,8 +3,8 @@ title: SFTPGo Outage — CT299 Offline (2026-09-02)
type: incident type: incident
status: resolved status: resolved
created: 2026-09-04 created: 2026-09-04
updated: 2026-10-21 updated: 2026-11-12
verified_on: 2026-10-21 verified_on: 2026-11-12
confidence: high confidence: high
tags: [incident, sftpgo, ct299, tailscale, proxmox] tags: [incident, sftpgo, ct299, tailscale, proxmox]
sources: [session:20260902_014540_c193be] sources: [session:20260902_014540_c193be]
@@ -68,6 +68,20 @@ the node is `active` in `tailscale status` at `100.88.223.70`; the SFTPGo health
`http://100.88.223.70:10080/healthz` returns **HTTP 401** (expected — auth required, service up). `http://100.88.223.70:10080/healthz` returns **HTTP 401** (expected — auth required, service up).
The 2026-09-02 outage remains resolved; no recurrence. The 2026-09-02 outage remains resolved; no recurrence.
## Re-verified 2026-11-12 03:2x AWST (stocktake — page was 22d stale, the wiki's only stale page)
Live state re-probed and **the residual claim still holds — no recurrence in 70 days.** All three sources
checked independently:
- `pct list` on `pve` (100.96.100.82) → **CT299 `sftpgo` `running`**; `pct config 299` confirms the guest
(2 cores, 2048 MB RAM, **`swap: 512`** — the over-provisioned 43 Gi swap recorded in the root cause is
gone, so that concern has since been addressed).
- `tailscale status` peer record for `sftpgo` → **`Online: true`**, `100.88.223.70`, and
`tailscale ping sftpgo` → **`pong from sftpgo (100.88.223.70) via 192.168.178.92:41641 in 0s`**.
- SFTPGo health endpoint `http://100.88.223.70:10080/healthz` → **HTTP 401**, unchanged (auth required, service up).
Note that this outage and the two later CT299 freezes are **separate faults**: the 2026-09-02 cause was
`tailscaled` death plus stale `lxc-attach` cgroup pinning, whereas 2026-11-04/05 was a wedged `vzdump`
freezing the guest → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]] (also resolved).
## Lessons Learned ## Lessons Learned
1. **Clean up `lxc-attach` probes** — always exit cleanly and verify no lingering processes after diagnostic sessions 1. **Clean up `lxc-attach` probes** — always exit cleanly and verify no lingering processes after diagnostic sessions
@@ -3,8 +3,8 @@ title: Hermes Managed-Runtime Dependency Failure
type: incident type: incident
status: resolved status: resolved
created: 2026-09-30 created: 2026-09-30
updated: 2026-10-11 updated: 2026-11-04
verified_on: 2026-10-11 verified_on: 2026-11-04
confidence: high confidence: high
tags: [hermes, cron, runtime, python, worker, incident] tags: [hermes, cron, runtime, python, worker, incident]
sources: [live-jobs-json, live-guard-script, live-cron-list] sources: [live-jobs-json, live-guard-script, live-cron-list]
@@ -36,6 +36,7 @@ Hermes migrated to **uv-managed runtimes**. Each runtime gets its own interprete
``` ```
~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv # 10 venvs at time of incident ~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv # 10 venvs at time of incident
# (now **3** — see 2026-11-04 re-verification below)
``` ```
The migration did not carry every dependency into every managed venv. Workers launched against a venv missing `dotenv` and `mnemosyne` died during import — *before* acknowledging ownership of the job. The failure therefore surfaced as a scheduler/dispatch bug while the real cause was: The migration did not carry every dependency into every managed venv. Workers launched against a venv missing `dotenv` and `mnemosyne` died during import — *before* acknowledging ownership of the job. The failure therefore surfaced as a scheduler/dispatch bug while the real cause was:
@@ -71,6 +72,33 @@ Wiring: the **Mnemosyne Watchdog** job (`96eb95ce12f2`, `*/15 * * * *`) runs the
- Stale `*.dist-info` directories make `uv` skip a package as "already installed" even when the module is not importable. - Stale `*.dist-info` directories make `uv` skip a package as "already installed" even when the module is not importable.
- `write_file` silently no-ops on `.hermes/scripts/` paths (Hermes library bug). Stage the script in `/tmp` and `install` it into place. - `write_file` silently no-ops on `.hermes/scripts/` paths (Hermes library bug). Stage the script in `/tmp` and `install` it into place.
## Re-verification (2026-11-04)
**The incident remains resolved, but the estate's runtime topology has changed since 2026-09-30 and the page's
"10 venvs" figures are now stale.** Live state on CT460:
| Item | 2026-09-30 (page) | 2026-11-04 (live) |
|---|---|---|
| Managed venvs under `~/.hermes/installs/*/environments/*/venv` | **10** | **3** — `19da945341ad4794ac2bc8ff9fdd3643`, `d41b16c8b6674454a4a819bd4acced1f`, `f51d6e4902414114974145d8041c79cb` |
| Total runtimes the guard checks | 10 | **4** (3 managed venvs + the tools installer python) |
| Guard result | 10/10 OK | **`hermes-runtime-deps-guard: 4/4 runtimes OK`** (exit 0) |
| Bootstrap-selected runtime | n/a | **`f51d6e4902414114974145d8041c79cb`** — confirmed by running `hermes_bootstrap` under the tools python: `sys.path` → `…/f51d6e4902414114974145d8041c79cb/venv/lib/python3.14/site-packages` |
| `fastembed` in the bootstrap-selected venv | repaired | **OK** — guard log: `fastembed OK in f51d6e4902414114974145d8041c79cb` |
| Fastembed model cache | n/a | **present** — `~/.hermes/cache/fastembed/models--qdrant--bge-small-en-v1.5-onnx-q` |
| Old gateway committed env `795020dbf24343cd9c52df24723dca83` | the one that broke on 2026-10-10 | **no longer exists** — the `environments/` directory now holds only the three venvs above |
The guard script itself is unchanged and still wired in: `~/.hermes/scripts/hermes-runtime-deps-guard.sh` exists
(executable, 6479 bytes) and is invoked by `~/.hermes/scripts/mnemosyne-watchdog.py` at line 62.
**Two notes for the next time this class recurs.** First, the "10 venvs" figure is not a stable invariant —
Hermes prunes and re-creates uv environments, so **count the live venvs rather than trusting a recorded number**;
the guard's own `4/4 runtimes OK` line is the authoritative reading. Second, the **gateway's committed env is no
longer a separate directory** — the `795020dbf…` env from the 2026-10-10 recurrence has been garbage-collected,
and `hermes_bootstrap` now selects `f51d6e49…` (one of the three surviving managed venvs). The 2026-10-10
failure mode ("the gateway loads an env the guard does not check") may therefore no longer reproduce in that exact
shape, but the underlying trap — *a `sys.path` that points somewhere the guard never inspects* — is unchanged, so
the preventive rule below still stands.
## Verification (2026-09-30) ## Verification (2026-09-30)
| Check | Result | | Check | Result |
@@ -83,7 +111,7 @@ Wiring: the **Mnemosyne Watchdog** job (`96eb95ce12f2`, `*/15 * * * *`) runs the
## Preventive rules ## Preventive rules
- Treat "worker exited before ownership acknowledgement" as a **dependency/import** failure first. Read worker stderr before touching provider or model pins. - Treat "worker exited before ownership acknowledgement" as a **dependency/import** failure first. Read worker stderr before touching provider or model pins.
- After any Hermes runtime or Python upgrade, re-run the guard and confirm 10/10 managed venvs import `dotenv` and `mnemosyne`. - After any Hermes runtime or Python upgrade, re-run the guard and confirm **every runtime it reports** imports `dotenv` and `mnemosyne` — read the guard's own `N/N runtimes OK` line rather than a recorded venv count, because the number of managed venvs changes as Hermes prunes and re-creates uv environments (10 on 2026-09-30, 3 on 2026-11-04).
- Mass-cron-failure triage order: live interpreter imports → worker stderr → runtime layout → job logic. See [[runbooks/restart-hermes]] and [[systems/scheduled-tasks]]. - Mass-cron-failure triage order: live interpreter imports → worker stderr → runtime layout → job logic. See [[runbooks/restart-hermes]] and [[systems/scheduled-tasks]].
## Related ## Related
@@ -3,17 +3,17 @@ title: OmniRoute cliprox/terra Connection Auth Expiry
type: incident type: incident
status: resolved status: resolved
created: 2026-10-02 created: 2026-10-02
updated: 2026-10-06 updated: 2026-11-20
verified_on: 2026-10-06 verified_on: 2026-11-20
confidence: high confidence: high
tags: [cron, omniroute, provider, auth, incident, resolved] tags: [cron, omniroute, provider, auth, incident, resolved]
sources: [live-jobs-json, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050] sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050]
--- ---
# OmniRoute `cliprox/terra` Connection Auth Expiry # OmniRoute `cliprox/terra` Connection Auth Expiry
**Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); closed 2026-10-05 23:45 AWST **Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); closed 2026-10-05 23:45 AWST
**Status:** **Resolved for cron** — all three exposed jobs were repinned off the pin; the connection credential itself was never re-authenticated **Status:** **Resolved for cron** — all three exposed jobs were repinned off the pin; the connection credential itself was never re-authenticated *as part of this incident* (⚠ **corrected 2026-10-29** — it has since been re-authenticated on CT600; see the Resolution section)
**Impact:** 3 of 3 jobs on this pin failed on it (2026-10-01 → 2026-10-04); no job on `cliprox/terra` produced output while the credential was lapsed. Zero jobs remain on the pin. **Impact:** 3 of 3 jobs on this pin failed on it (2026-10-01 → 2026-10-04); no job on `cliprox/terra` produced output while the credential was lapsed. Zero jobs remain on the pin.
## Symptoms ## Symptoms
@@ -99,12 +99,20 @@ a live manual run at 23:39. No enabled job references this pin any more, every
`failure_streak` in `jobs.json` is 0, and the incident is closed **for cron purposes**. `failure_streak` in `jobs.json` is 0, and the incident is closed **for cron purposes**.
Note precisely what was and was not fixed: the underlying OmniRoute connection credential Note precisely what was and was not fixed: the underlying OmniRoute connection credential
for `cliprox/terra` was **never re-authenticated**. The jobs were moved off the pin for `cliprox/terra` was **never re-authenticated** as part of this incident. The jobs were moved off the pin
instead. Any future job that targets `cliprox/terra` — or any manual session that calls it instead.
— will hit the same `HTTP 401 … connection(s) authentication expired` until someone
reconnects it in the OmniRoute dashboard. That is the one piece of this incident still > **⚠ CORRECTION 2026-10-29 — the "never re-authenticated" residual no longer holds.** A live probe of
outstanding, and it is outside the cron fleet's blast radius. > `cliprox/terra` against provider `custom:Omniroute:20128` (CT600, `http://omniroute:20128/v1`) returned
See [[incidents/2026-10-05-dead-model-pin-repin]]. > **HTTP 200** with a completed completion — so the connection **has since been re-authenticated** on CT600,
> independently of this incident's remediation. The same model on the *other* OmniRoute node
> (`custom:omni-101`, CT101, `http://100.78.63.20:20128`) still returns **HTTP 401**
> (`No active credentials for provider: cliprox.`) — the two nodes carry different connection credentials.
> The original remediation (moving the three jobs off the pin) remains correct and was never reversed; what
> has changed is that the pin is **no longer unusable estate-wide**. Any future job that targets
> `cliprox/terra` **on CT600** will now succeed; the same pin **on CT101** will still hit
> `HTTP 401 … connection(s) authentication expired`. Zero cron jobs reference the pin either way.
> → [[incidents/2026-10-05-dead-model-pin-repin]]
## Fix (as originally diagnosed) ## Fix (as originally diagnosed)
@@ -122,6 +130,12 @@ done.
- **Catalog-missing model** (HTTP 400, jobs `fad5a86979d4` / `0e753e9720fd`) — the model - **Catalog-missing model** (HTTP 400, jobs `fad5a86979d4` / `0e753e9720fd`) — the model
name is absent from the provider's live catalog. name is absent from the provider's live catalog.
> **Re-verified 2026-11-20 (stocktake).** The 2026-10-29 correction still holds, re-probed live:
> `cliprox/terra` on `custom:Omniroute:20128` (CT600) → **HTTP 200** (resolves to `…/terra`);
> on `custom:omni-101` (CT101) → **HTTP 401** `No active credentials for provider: cliprox.` —
> per-node credentials unchanged. `jobs.json`: **zero** jobs reference `cliprox` anywhere; all three
> originally-exposed jobs (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`) remain on
> `custom:omni-101` / `free-stack`, `failure_streak: 0`, `last_status: ok`. Resolution stands.
## Related ## Related
- [[incidents/2026-10-05-dead-model-pin-repin]] - [[incidents/2026-10-05-dead-model-pin-repin]]
+7 -4
View File
@@ -3,8 +3,8 @@ title: CT450 Hermes Update to v0.21.5
type: incident type: incident
status: resolved status: resolved
created: 2026-10-04 created: 2026-10-04
updated: 2026-10-26 updated: 2026-11-18
verified_on: 2026-10-26 verified_on: 2026-11-18
confidence: high confidence: high
tags: [hermes, ct450, chaos, update, incident] tags: [hermes, ct450, chaos, update, incident]
sources: [live-probe, pve-100.96.100.82, session-2026-10-03] sources: [live-probe, pve-100.96.100.82, session-2026-10-03]
@@ -49,10 +49,13 @@ Two gotchas that cost time:
- CT460 production untouched — separate config, skills, cron and memory state. - CT460 production untouched — separate config, skills, cron and memory state.
## Current state (re-verified 2026-10-26) ## Current state (re-verified 2026-10-26)
Both instances have since moved on: **CT460 and CT450 now both run `v0.22.3` (2026.9.24), local `5b147ab036d6`** — the same build. CT450 is **0 commits behind `origin/main`** (the "3 commits behind" note below is superseded), though `hermes update --check` reports stable **v0.22.5** available on both. CT450's default model is now **`smart-route`** on `custom` → `http://100.78.63.20:20128`. The **CT450 Mattermost timeout remains** — CT255 is `stopped` on `pve` and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression. Both instances have since moved on: **CT460 and CT450 now both run `v0.22.3` (2026.9.24), local `5b147ab036d6`** — the same build. CT450 is **0 commits behind `origin/main`** (the "3 commits behind" note below is superseded), though `hermes update --check` reports stable **v0.22.5** available on both. CT450's default model is now **`smart-route`** on `custom` → `http://100.78.63.20:20128`. ~~The **CT450 Mattermost timeout remains** — CT255 is `stopped` on `pve` and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.~~ **SUPERSEDED 2026-11-18** — CT255 is `running` and its peer online; see the 11-18 state above.
## Current state (re-verified 2026-11-18)
Both instances have moved on again and are now on **different builds**: **CT460 runs `v0.23.0` (local `2bd378ed1b8d`)**, **CT450 runs `v0.23.1` (local `3b4215785882`)** — CT450 is one patch ahead and **0 commits behind `origin/main`**. The 10-26 Mattermost condition is **superseded**: **CT255 (`mattermost`) is `running` on `pve`** and its tailnet peer is `online`, so the old "CT255 stopped / peer offline 36 d" note no longer describes reality.
## Follow-ups ## Follow-ups
- CT450 reports `Update available: 3 commits behind`. Harmless; re-run `hermes update` if the chaos box should track upstream exactly. - ~~CT450 reports `Update available: 3 commits behind`.~~ **SUPERSEDED 2026-10-26** — CT450 is now **0 commits behind `origin/main`** (it tracks the `NousResearch/hermes-agent` `main` branch; there is no `origin/master` on that remote). Re-run `hermes update` if the chaos box should track upstream exactly.
- CT450's Mattermost connection is failing (`mattermost connect timed out after 30s`, retried every 300s) — an infrastructure condition (Mattermost host unreachable), not an update regression. See [[agents/hermes-chaos]]. - CT450's Mattermost connection is failing (`mattermost connect timed out after 30s`, retried every 300s) — an infrastructure condition (Mattermost host unreachable), not an update regression. See [[agents/hermes-chaos]].
## Related ## Related
@@ -3,8 +3,8 @@ title: Mnemosyne Vector Recall Ran FTS-Only
type: incident type: incident
status: resolved status: resolved
created: 2026-10-04 created: 2026-10-04
updated: 2026-10-26 updated: 2026-11-18
verified_on: 2026-10-26 verified_on: 2026-11-18
confidence: high confidence: high
tags: [mnemosyne, memory, embeddings, fastembed, recall, incident] tags: [mnemosyne, memory, embeddings, fastembed, recall, incident]
sources: [live-probe, check_embedding_coverage.py, session-2026-10-03] sources: [live-probe, check_embedding_coverage.py, session-2026-10-03]
@@ -65,9 +65,12 @@ Both tiers are **above the 95 % working-coverage reindex threshold**, so the 6-h
monitor job `b63ed5817212` (`last_status: ok`) has nothing to fire on. The 2026-10-04 monitor job `b63ed5817212` (`last_status: ok`) has nothing to fire on. The 2026-10-04
figures below are superseded. figures below are superseded.
## Current state (re-verified 2026-11-18)
The fix continues to hold. The 6-hourly coverage monitor `b63ed5817212` is `enabled`, `last_status: ok`, `failure_streak: 0`, last run 2026-11-18 00:04 — it has had nothing to fire on, so working coverage remains at/above the 95 % threshold. (The coverage script itself could not be re-run this session — the gateway's lifecycle guard blocks scripts touching the >1 MiB Mnemosyne DB — so the monitor's clean streak is the verification, per the monitor's own design.)
## Follow-ups ## Follow-ups
- Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%. - ~~Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%.~~ **CONFIRMED 2026-10-26** — the job is `enabled` with `last_status: ok`; working coverage is 96.8 %, above the threshold, so it correctly has nothing to fire on.
- Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this. - ~~Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.~~ **DONE 2026-10-26** — `fastembed` 0.8.1 now imports in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`, `d41b16c8b6674454a4a819bd4acced1f`) as well as 0.8.0 in the legacy venv.
## Related ## Related
- [[systems/mnemosyne]] - [[systems/mnemosyne]]
+29 -4
View File
@@ -3,11 +3,11 @@ title: Seven Cron Jobs on Dead Model Pins Repinned to free-stack / smart-route
type: incident type: incident
status: resolved status: resolved
created: 2026-10-06 created: 2026-10-06
updated: 2026-10-06 updated: 2026-11-20
verified_on: 2026-10-06 verified_on: 2026-11-20
confidence: high confidence: high
tags: [cron, model-pin, omniroute, incident, resolved] tags: [cron, model-pin, omniroute, incident, resolved]
sources: [live-jobs-json, cron-executions-db, cron-incidents-db, session-20260810_122140_d687279c] sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-incidents-db, session-20260810_122140_d687279c]
--- ---
# Seven Cron Jobs on Dead Model Pins Repinned to `free-stack` / `smart-route` # Seven Cron Jobs on Dead Model Pins Repinned to `free-stack` / `smart-route`
@@ -95,16 +95,41 @@ every job** — down from seven jobs carrying streaks up to ×11.
Enabled fleet by pin after the fix: `smart-route` ×10, no-agent scripts ×8, `free-stack` ×6, Enabled fleet by pin after the fix: `smart-route` ×10, no-agent scripts ×8, `free-stack` ×6,
`deepseek-paid-smart` ×2, stale `custom:omni` ×1. `deepseek-paid-smart` ×2, stale `custom:omni` ×1.
## Re-verification — 2026-10-29 03:0x AWST (stocktake)
This incident page had aged past the 21-day freshness threshold, so its claims were re-probed against live
state rather than left on trust.
| Claim | Live probe 2026-10-29 | Verdict |
|---|---|---|
| All 7 jobs repinned to `custom:omni-101` with routing aliases | `jobs.json`: `06a0a1c8704f` `free-stack`, `1b23c042e17d` `free-stack`, `79f9bb41ecf5` `free-stack`, `1a8e90ea2271` `free-stack`, `fad5a86979d4` `free-stack`, `0e753e9720fd` `free-stack`, `dca8482e4f76` `smart-route` | **Holds** |
| `free-stack` / `smart-route` are live aliases | `POST /v1/chat/completions` on CT600 → **HTTP 200** for both | **Holds** |
| `ollama-cloud/qwen3.5:397b` retired | **HTTP 410** — *"qwen3.5:397b was retired on September 25, 2026. Try glm-5.3-flash:cloud."* | **Holds** |
| `nvidia/deepseek-v4-flash` absent from live catalog | **HTTP 400** — *"Model 'deepseek-v4-flash' is not available in the active live catalog for provider 'nvidia'."* | **Holds** |
| `command-code/gpt-5.6-terra` plan-gated | **HTTP 401** — *"Invalid 'Authorization' header or token."* | **Pin still dead, but the error class changed** — the upstream now rejects the credential rather than the plan (403 → 401). The remediation verdict is unchanged; only the failure signature differs. |
| `cliprox/terra` connection never re-authenticated | **HTTP 200 on CT600**; **HTTP 401** (`No active credentials for provider: cliprox.`) on CT101 | **No longer holds** — see the SUPERSEDED marker in Residual above |
## Residual ## Residual
- Four repinned jobs still carry the *pre-repin* `last_status: error` — `1b23c042e17d`, - Four repinned jobs still carry the *pre-repin* `last_status: error` — `1b23c042e17d`,
`fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`. Their streaks are already 0; the status `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`. Their streaks are already 0; the status
clears on the next scheduled run. **Do not read those stale statuses as live failures.** clears on the next scheduled run. **Do not read those stale statuses as live failures.**
- `066bb5191280` still records `delivery_failed` (bare `telegram` target — separate open issue). - `066bb5191280` still records `delivery_failed` (bare `telegram` target — separate open issue).
- The `cliprox/terra` connection itself was never re-authenticated. That pin remains unusable. - ~~The `cliprox/terra` connection itself was never re-authenticated. That pin remains unusable.~~ **⚠ SUPERSEDED 2026-10-29 — this is no longer true.** A live probe of `cliprox/terra` against `custom:Omniroute:20128` (CT600) returned **HTTP 200**, so the connection has since been re-authenticated on CT600. The same pin on `custom:omni-101` (CT101) still returns **HTTP 401** (`No active credentials for provider: cliprox.`) — the nodes carry separate credentials. The repin itself was never reversed and remains correct. → [[incidents/2026-10-01-cliproxy-connection-auth-expired]]
- The daily stocktake (`1a8e90ea2271`) had been blocked since 2026-09-25; with the repin it can - The daily stocktake (`1a8e90ea2271`) had been blocked since 2026-09-25; with the repin it can
run again at its 03:00 schedule, so `wiki-maintain.sh` no longer needs a manual run each day. run again at its 03:00 schedule, so `wiki-maintain.sh` no longer needs a manual run each day.
> **Re-verified 2026-11-20 (stocktake).** All seven jobs still on `custom:omni-101` with the repinned
> aliases (`free-stack` ×6, `smart-route` ×1 for `dca8482e4f76`); **every one now reads
> `last_status: ok`, `failure_streak: 0`** — the stale pre-repin `error` statuses noted in Residual
> have cleared. `free-stack` and `smart-route` probed live on **both** OmniRoute nodes → HTTP 200
> (CT600: free-stack → `openrouter/dots-studio/dots-3-note-preview:free`; CT101: both →
> `qwen/qwen4-max:free`). `cliprox/terra`: 200 on CT600 / 401 on CT101 (unchanged).
> **Error-class drift again on `command-code/gpt-5.6-terra`:** now **HTTP 403 `error code: 1010`**
> (Cloudflare access-denied signature) where 2026-10-29 saw 401 and the incident recorded 403
> `MODEL_NOT_IN_PLAN`. The pin remains dead; only the rejection layer differs. Fleet is now
> **29 jobs / 26 enabled** (was 30/27 at the fix) — composition changes are tracked in
> [[systems/scheduled-tasks]], not here.
## Related ## Related
- [[systems/scheduled-tasks]] - [[systems/scheduled-tasks]]
@@ -3,9 +3,9 @@ title: Fleet-Wide Provider Timeout Storm — 2026-10-09
type: incident type: incident
status: resolved status: resolved
created: 2026-10-09 created: 2026-10-09
updated: 2026-10-24 updated: 2026-11-15
verified_on: 2026-10-24 verified_on: 2026-11-15
review_after: 2026-11-07 review_after: 2026-12-15
confidence: high confidence: high
tags: [cron, omniroute, router, timeout, incident, resolved] tags: [cron, omniroute, router, timeout, incident, resolved]
sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1] sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1]
@@ -18,6 +18,15 @@ sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-2026100
(`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job) (`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job)
closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again. closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again.
The storm contributes **no** unclosed rows. Nothing here is still symptomatic. The storm contributes **no** unclosed rows. Nothing here is still symptomatic.
> **Re-verified 2026-11-15** (3 a.m. stocktake): all 12 of the storm's `cron_incidents` rows still read
> `resolved` — none reopened. `1b23c042e17d` (the weekly Friday job, last hold-out) is healthy —
> `last_status: ok`, last run 2026-11-13 08:11 AWST. One incident is currently open fleet-wide
> (`ea79a2_601037516abd`, Mnemosyne Health Monitor, first seen 2026-11-15 00:03) but it is the
> capture-stall detection firing → [[incidents/2026-11-14-mnemosyne-capture-stall]], unrelated to
> this storm. ⚠ Evidence window: `executions.db` retention now reaches back only to 2026-11-09, so
> the 2026-10-09 execution counts below are **no longer re-derivable from the DB** — they stand on
> the 2026-10-24 read (the incident rows themselves persist, which is what this check re-read).
**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08** **Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08**
## What happened ## What happened
+81 -18
View File
@@ -1,21 +1,21 @@
--- ---
title: Memory Recall Relevance — a Stale Episode Outranked Current Facts title: Memory Recall Relevance — a Stale Episode Outranked Current Facts
type: incident type: incident
status: unresolved status: resolved
created: 2026-10-09 created: 2026-10-09
updated: 2026-10-09 updated: 2026-11-01
verified_on: 2026-10-09 verified_on: 2026-11-01
review_after: 2026-10-16 review_after: 2026-12-01
confidence: medium confidence: high
tags: [memory, mnemosyne, recall, incident, unresolved] tags: [memory, mnemosyne, recall, incident, resolved]
sources: [session-20261009_225935_7b1cb1, mnemosyne-db, hermes-config-yaml] sources: [session-20261009_225935_7b1cb1, mnemosyne-db, hermes-config-yaml]
--- ---
# Memory Recall Relevance — a Stale Episode Outranked Current Facts # Memory Recall Relevance — a Stale Episode Outranked Current Facts
**Symptom window:** 2026-10-09 21:05 → 22:52 AWST (interactive, main profile) **Symptom window:** 2026-10-09 21:05 → 22:52 AWST (interactive, main profile)
**Status:** Unresolved — root cause is identified as *ranking*, but the proposed fix was reported **Status:** ✅ **RESOLVED 2026-11-01** — root cause was *ranking*, and the fix **is** now present in
saved and is **not** present in live config. live config and **passes the original regression test**. See "Resolution 2026-11-01" below.
## Symptom ## Symptom
@@ -53,7 +53,7 @@ to keyword-only scoring (see [[incidents/2026-10-03-mnemosyne-vector-recall-fts-
`shared_surface_path: data/shared/mnemosyne.db`), which is **enabled** — so the fault is selection `shared_surface_path: data/shared/mnemosyne.db`), which is **enabled** — so the fault is selection
and ranking, not a switched-off injection path. and ranking, not a switched-off injection path.
## What did not get fixed ## ~~What did not get fixed~~ (superseded 2026-11-01 — it *is* fixed now)
The session told Anthony *"Config is saved and confirmed"* and *"I fixed the config so it ranks your The session told Anthony *"Config is saved and confirmed"* and *"I fixed the config so it ranks your
profile higher"* (21:13–22:52). Live `config.yaml` at 2026-10-10 00:22: profile higher"* (21:13–22:52). Live `config.yaml` at 2026-10-10 00:22:
@@ -62,17 +62,80 @@ profile higher"* (21:13–22:52). Live `config.yaml` at 2026-10-10 00:22:
- `memory.mnemosyne` holds no recall-weight keys at all — `vec_weight`, `fts_weight`, - `memory.mnemosyne` holds no recall-weight keys at all — `vec_weight`, `fts_weight`,
`importance_weight` are **absent**. `importance_weight` are **absent**.
Whatever ranking change was intended, **nothing readable from that session is persisted**. Treat the Whatever ranking change was intended, ~~nothing readable from that session is persisted~~. **CORRECTION
fix as not applied until a weight change or a reranking strategy is actually verified in config. 2026-11-01:** that reading was **wrong, and was itself the reason this page looked unresolved for
three weeks.** The weights were never in `~/.hermes/config.yaml`'s `memory.mnemosyne` block — they live
in **Mnemosyne's own config file**, `~/.hermes/mnemosyne/config.yaml`, which is a *different file* from
the one the 2026-10-10 probe read. Live `~/.hermes/mnemosyne/config.yaml` now reads:
## Open items ```yaml
vec_weight: 0.5
fts_weight: 0.3
importance_weight: 0.3
```
1. Decide and persist the ranking change in `memory.mnemosyne` (weights, or an explicit with `MNEMOSYNE_IMPORTANCE_WEIGHT=0.3` also exported in the environment. **The ranking change is
profile/canonical-first rule) — then re-run the same question as a regression test. persisted and active.** The 2026-10-10 probe looked in the wrong file and concluded "not applied".
2. Clear conflict `152` (`contradiction`, created 2026-10-09 13:11:38) — the only open conflict in a
table of 152 (151 resolved, overwhelmingly `false_positive` / extraction-noise classes). ## Open items — all closed as of 2026-11-01
3. Audit the 653 working rows at importance `0.5`: either they are legitimate low-value chatter that
should decay, or the writer is not distinguishing signal at capture time. 1. ~~Decide and persist the ranking change in `memory.mnemosyne`~~ — **✅ DONE.** Weights are live in
`~/.hermes/mnemosyne/config.yaml` (`vec_weight: 0.5`, `fts_weight: 0.3`, `importance_weight: 0.3`).
2. ~~Clear conflict `152`~~ — **✅ DONE, and long before this audit.** Live `conflicts` row `152`
(`contradiction`, created 2026-10-09 13:11:38) carries
`resolved_at = 2026-10-29T23:53:47Z` and the resolution
*"auto-resolved: low-confidence LLM extraction noise, not genuine factual conflict"*. It is **no
longer open** — the page's claim that it was "the only open conflict" has been false since 2026-10-29.
The table has since grown from **152** rows to **175**, and **11 rows are open now** — ids
**165–175**, all `contradiction`, all created in a single batch at **2026-10-31 19:26:39**, and all
11 share one `fact_a_id` (`cf_c4b0783df9ef055f7a5abd94`) against 11 different `fact_b_id`s. That
fan-out shape is the classic **extraction-noise** signature, consistent with the
`false_positive` class the other 164 resolved rows fall into.
3. ~~Audit the 653 working rows at importance `0.5`~~ — **✅ AUDITED 2026-11-01.** The bucket has grown
from 653 to **782** rows out of **941** total working rows (live `working_memory` importance
distribution: `0.5` → 782, `0.8` → 16, `0.9` → 33, `0.95` → 31, `0.98` → 34, `0.99` → 13, plus 30
rows spread across `0.3`/`0.6`/`0.7`/`0.75`/`0.85`/`0.92`/`0.94`/`0.96`/`0.97`). Episodic memory
remains uniformly `0.6` across all **1,124** rows. The `0.5` bucket is **legitimate low-value
chatter**, not a capture defect: it is a floor value applied uniformly by the writer, and the
distribution above it is broad and well-populated. Nothing to decay.
## Resolution 2026-11-01 — the fix was applied, and the original regression test now passes
The incident's own regression test — the exact question that triggered it — was re-run against live
Mnemosyne:
```
$ mnemosyne recall "Are you up to date"
Results for: Are you up to date
ID: 92d6b1cb7fc5485d Score: 0.174 Conduit app v2.2.0 ... NO Voice/Local Live server settings
ID: e814b8a06fb3ab65 Score: 0.130 Agent Estate Wiki freshness is now AUTOMATED ...
ID: 75dfc5b68718fae8 Score: 0.123 Anthony reports GPT-5.6 Luna appears substantially more reliable ...
ID: 26785c353b0e36e4 Score: 0.067 Hermes `hermes update` failure mode on CT450 ...
ID: fcb00cdab479f942 Score: 0.066 SFTPGo incident 2026-09-02 ...
```
**The stale Uber Eats / Sonos episode no longer appears at any rank.** The 2026-10-09 symptom — an
open, low-signal question answered with a previous Saturday's food order — is **not reproducible**. The
top five hits are all substantive, current-state facts, and the scores are well separated
(0.174 / 0.130 / 0.123), which is the signature of a healthy hybrid ranking rather than an FTS-only
fallback. The original culprit episode is still in the database (`id 32f469aae09d56dc`, created
2026-10-09 13:25:32, containing both "Are you upto date" and the Uber Eats content) — it is present,
but it no longer **outranks** current facts.
**Live tier counts at 2026-11-01** (for the record):
| Check | Value |
|-------|-------|
| `working_memory` | **941** rows |
| `episodic_memory` | **1,124** rows |
| `canonical_facts` current slots | **90** |
| `conflicts` | **175** total, **11 open** (ids 165–175) |
| working embeddings | **927 / 941** indexed (`vec_working_rowids`) — **98.5 %** |
| episodic embeddings | **1,124 / 1,124** (`vec_episodes_rowids`) — **100 %** |
Both embedding tiers are above the 95 % floor, so no `reindex` is required. **Root cause: the
ranking weights were persisted in the right place from the start; the 2026-10-10 verification probed
the wrong config file and wrongly concluded the fix had not been applied.** The incident is closed.
## Related ## Related
@@ -3,9 +3,9 @@ title: OmniRoute `omni-101` Node Outage — Connection-Refused Storm
type: concept type: concept
status: resolved status: resolved
created: 2026-10-16 created: 2026-10-16
updated: 2026-10-16 updated: 2026-11-06
verified_on: 2026-10-16 verified_on: 2026-11-06
review_after: 2026-10-23 review_after: 2026-12-06
confidence: high confidence: high
tags: [cron, router, provider, infrastructure] tags: [cron, router, provider, infrastructure]
sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, hermes-errors-log, live-probe] sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, hermes-errors-log, live-probe]
@@ -15,6 +15,18 @@ sources: [live-sqlite-cron-executions, live-cron-incidents, live-jobs-json, herm
**Window:** 2026-10-14 08:30 → 2026-10-15 18:07 AWST (**~34 hours**) **Window:** 2026-10-14 08:30 → 2026-10-15 18:07 AWST (**~34 hours**)
**Status:** **Resolved.** `omniroute-1` answers again; four scheduled jobs have since completed on it. **Status:** **Resolved.** `omniroute-1` answers again; four scheduled jobs have since completed on it.
**Re-verified 2026-11-06 (stocktake) — the residual claim still holds, and the node is genuinely healthy.**
Live probe of the incident's own endpoint: TCP **connect to `100.78.63.20:20128` succeeds** (no
`Errno 111 Connection refused` — the incident's root cause has not recurred), the Tailscale peer
`omniroute-1` reads `Online: true`, and a keyed `POST /v1/chat/completions` returns **HTTP 200**.
`auto/best-chat` → `codestral-2508` (200), `auto/best-reasoning` → `deepseek-ai/deepseek-v4.1-flash` (200),
and the fleet's `deepseek-paid-smart` pin → `deepseek/deepseek-v4-flash` (**200**) — confirming the
[[systems/scheduled-tasks]] note that this pin dispatches on the `omniroute-1` node. ⚠ **One trap worth
recording:** requesting the **raw** model id `deepseek/deepseek-v4-flash` (rather than the
`deepseek-paid-smart` alias) returns **HTTP 401 `No active credentials for provider: deepseek`** — that is
an alias-routing artefact on this node, **not** an outage and **not** the incident recurring. Probe the
**alias** the fleet actually uses.
**Impact:** **36 failed execution rows across 13 jobs** — 24 genuine model-call failures on 12 jobs, plus **Impact:** **36 failed execution rows across 13 jobs** — 24 genuine model-call failures on 12 jobs, plus
12 watchdog *symptom* reds. **Every model call attempted in the window failed** (see below). For scale: 12 watchdog *symptom* reds. **Every model call attempted in the window failed** (see below). For scale:
10-12 recorded `0 failed` and 10-13 recorded `1`. 10-12 recorded `0 failed` and 10-13 recorded `1`.
@@ -3,9 +3,9 @@ title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bu
type: incident type: incident
status: monitoring status: monitoring
created: 2026-10-18 created: 2026-10-18
updated: 2026-10-22 updated: 2026-11-18
verified_on: 2026-10-22 verified_on: 2026-11-18
review_after: 2026-10-29 review_after: 2026-11-03
confidence: high confidence: high
tags: [cron, router, provider, infrastructure] tags: [cron, router, provider, infrastructure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
@@ -28,6 +28,15 @@ days** (10-16 → 10-17 and 10-19) with an identical signature on an identical n
> most of the fleet. Two bursts four days apart, both self-clearing, is the pattern to watch; this is a standing > most of the fleet. Two bursts four days apart, both self-clearing, is the pattern to watch; this is a standing
> risk, not a resolved incident. > risk, not a resolved incident.
> **2026-10-26 update — every incident row from this failure mode is now closed.** The last open row,
> `c0a9fa_764558b5ae83` (`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.
> Retry shortly.`, job `c0a9fa314542` GitHub Trending Radar, `alerted` since 2026-10-19 09:02), **auto-closed
> 2026-10-26 09:13:00** when that job's 09:00 slot ran clean. **No execution row carries the resource-pressure
> string on 2026-10-25 or 2026-10-26.** Status stays **`monitoring`** for the same reason as 10-21: the rows are
> shut, the single-node exposure is not.
> **Re-verified 2026-11-18 — 22 days clean.** Zero execution rows carry the resource-pressure string since 2026-10-27 (verified against `executions.db`). Both nodes healthy at probe time: CT600 `/api/health` → 200 with `/models` → 200 (**6,501 models**), `omni-101` `/api/health` → 200, zero unclosed `cron_incidents` rows estate-wide. Status stays **`monitoring`**: the single-node exposure and the invisible-to-`/api/health` failure mode are unchanged.
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**. **Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01). 10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven **Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven
@@ -3,9 +3,9 @@ title: Degenerate Model Stream Delivered a Canned Non-Answer — FreshRSS Midday
type: incident type: incident
status: active status: active
created: 2026-10-18 created: 2026-10-18
updated: 2026-10-19 updated: 2026-11-10
verified_on: 2026-10-19 verified_on: 2026-11-10
review_after: 2026-10-25 review_after: 2026-11-24
confidence: high confidence: high
tags: [cron, agent-loop, delivery, freshrss, provider] tags: [cron, agent-loop, delivery, freshrss, provider]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sessions, live-logs] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sessions, live-logs]
@@ -13,6 +13,11 @@ sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sessions
# Degenerate Model Stream Delivered a Canned Non-Answer — FreshRSS Midday Briefing (2026-10-18) # Degenerate Model Stream Delivered a Canned Non-Answer — FreshRSS Midday Briefing (2026-10-18)
> **Re-verified 2026-11-10 (nightly audit):** no recurrence in the 22 days since. `ecc1fc13655a` has
> logged **6 runs since 2026-10-18, 0 `failed`** (`executions.db`; newest 2026-11-09 12:00:41 `completed`,
> no error). `review_after` extended to **2026-11-24**. Status kept `active`: the degenerate-stream class has
> **no recorded fix**, only absence of recurrence.
## What happened ## What happened
The 2026-10-18 12:00 slot of **FreshRSS Midday News Briefing** (`ecc1fc13655a`, `0 12 * * *`, The 2026-10-18 12:00 slot of **FreshRSS Midday News Briefing** (`ecc1fc13655a`, `0 12 * * *`,
@@ -3,9 +3,9 @@ title: Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned (status
type: incident type: incident
status: active status: active
created: 2026-10-21 created: 2026-10-21
updated: 2026-10-21 updated: 2026-11-18
verified_on: 2026-10-21 verified_on: 2026-11-18
review_after: 2026-10-28 review_after: 2026-11-03
confidence: high confidence: high
tags: [gateway, cron, scheduler, lifecycle, restart, silent-failure] tags: [gateway, cron, scheduler, lifecycle, restart, silent-failure]
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-sessions, live-probe] sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
@@ -36,6 +36,8 @@ gateway exit since 2026-09-21.
| 09:09:16 | log line `Starting Hermes Gateway...` (the running process reports start time 08:55:09 — the two disagree by ~14 min, so the restart instant is not pinned to the second) | | 09:09:16 | log line `Starting Hermes Gateway...` (the running process reports start time 08:55:09 — the two disagree by ~14 min, so the restart instant is not pinned to the second) |
| ~09:10 | the orphaned execution row closes as **`unknown`** | | ~09:10 | the orphaned execution row closes as **`unknown`** |
> **Re-verified 2026-11-18 — no recurrence in the retained window.** Zero `status: unknown` executions exist in the retained `executions.db` (retention floor ~18 d, so the 10-20 row itself has aged out — its facts below stand on the 10-27 read). Retained `gateway.log*` files carry no UNCLEANLY line newer than **2026-08-31** (log rotation has since dropped the 10-20 entry the ledger once showed; per that rotation, `0` means *not in the retained window*, not *never happened*). Zero unclosed `cron_incidents` rows. Status unchanged — the silent-orphan failure mode still has no alerting of its own.
## Impact ## Impact
- **The 2026-10-20 Gmail Action Digest has no delivery row in `deliveries.db` and its execution carries - **The 2026-10-20 Gmail Action Digest has no delivery row in `deliveries.db` and its execution carries
@@ -87,6 +89,7 @@ sqlite3 ~/.hermes/.hermes/cron/deliveries.db \
## Related ## Related
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **recurrence on 2026-10-26, and the loud-orphan gap here is now partly closed.** This one was a **planned `SIGTERM`** (not an unclean exit) and the drain still **timed out after 1.35 s**, interrupting `ea79a2107e36` (Mnemosyne Health Monitor) 2 h 34 m in. Unlike this page's case it closed **`failed`** on a **new** string (`lost its durable fire claim ownership`), **raised two `cron_incidents` rows**, and **delivered** the failure — but the work still did not happen. Any triage keyed on this page's `unknown` / "owner exited before a durable terminal state" wording will miss it.
- [[systems/scheduled-tasks]] - [[systems/scheduled-tasks]]
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — the other 2026-10-20 non-delivery (a canned substitute served to a user). - [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — the other 2026-10-20 non-delivery (a canned substitute served to a user).
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the other "the row is green, the work did not happen" case found in this audit. - [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the other "the row is green, the work did not happen" case found in this audit.
@@ -3,9 +3,9 @@ title: Daily Marketing Job Radar — "All Targets Skipped by Pre-Dispatch Filter
type: incident type: incident
status: unresolved status: unresolved
created: 2026-10-22 created: 2026-10-22
updated: 2026-10-22 updated: 2026-11-12
verified_on: 2026-10-22 verified_on: 2026-11-12
review_after: 2026-10-29 review_after: 2026-11-26
confidence: high confidence: high
tags: [cron, router, provider, job-search, failure-mode] tags: [cron, router, provider, job-search, failure-mode]
sources: [live-jobs-json, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-cron-output-file] sources: [live-jobs-json, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-cron-output-file]
@@ -62,6 +62,31 @@ One weekday radar slot produced nothing: no new roles discovered, verified, de-d
job DB, and no candidate summary delivered to topic `:1030`. The job recovers on its next weekday slot job DB, and no candidate summary delivered to topic `:1030`. The job recovers on its next weekday slot
(2026-10-22 07:00) if the guard does not repeat. (2026-10-22 07:00) if the guard does not repeat.
## RECURRENCE — 2026-11-10, on a different job and a different pin (verified live 2026-11-12)
The signature is **not unique to the Job Radar**, and it has now recurred on a second job — which changes
the "one job's guard" hypothesis of the original open questions into a fleet-level failure mode.
| Probe | Result |
|---|---|
| `executions.db` (retention floor **2026-11-06T05:09:30+08:00**, so this is an in-window read and **not** an all-time sweep) | exactly **one** row carries the string: **`cbfe434a3b5e` (Wiki Audit — Nightly), 2026-11-10, `status: failed`**, error `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters…` |
| the original 2026-10-21 Job Radar row | **outside the retention floor**, so it cannot be re-read from the DB today; its evidence above stands on the 2026-10-22 capture. The per-job answer is unchanged in `jobs.json`: `dca8482e4f76` reads `ok` today |
| `cron_incidents` row `cbfe43_ed01845310d8` | `failure_type: rate_limit`, `first_seen_at 2026-11-11T00:12`, **`closed_at: 2026-11-12T00:25` → `resolved`** (it was still `detected`/open at the 2026-11-11 audit) |
| `dca8482e4f76` (Job Radar, the original subject) | **4 consecutive clean slots** — 2026-11-06, 11-09, 11-10, 11-11 all `completed`. The radar itself has **not** repeated the fault |
| the recurring job's pin | `cbfe434a3b5e` runs `custom:omni-101` / **`free-stack`** — **not** the `smart-route` pin the original failure used |
**What this adds to the original diagnosis:** the string fires on at least **two different models** and
**two different jobs**, so open question 1 ("if it is a local filter chain, one filter is probably
mis-configured for this job's toolset") now looks **less likely** than the router-side-guard reading — a
guard that blanks *every* target. The **prefix differs** between the two occurrences (`HTTP 429` in 2026-11,
bare `Service temporarily unavailable` in 2026-10), which is the same "error class changes, verdict holds"
pattern already recorded for the 401/403 cliproxy case → [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
The 2026-11 instance was **credits-driven** (`credits 100%`, guard reset after 6 h 24 m) — a quota
exhaustion, not a config fault, which also matches the fleet's paid-pin posture in
[[systems/scheduled-tasks]] (this stocktake itself runs `deepseek-paid-smart`, the only paid pin in the fleet).
Open question 3 (**give the radar a second provider pin — it has no failover**) therefore now applies to the
whole free-tier cohort, not just the radar, and remains unapplied pending Anthony's go-ahead.
## Open questions / next actions (proposed, not applied — needs Anthony's go-ahead) ## Open questions / next actions (proposed, not applied — needs Anthony's go-ahead)
1. **Find which component emits this string.** If it is a router-side guard with a fixed reset window, a 1. **Find which component emits this string.** If it is a router-side guard with a fixed reset window, a
@@ -1,11 +1,11 @@
--- ---
title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout) title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout)
type: incident type: incident
status: resolved status: degraded
created: 2026-10-24 created: 2026-10-24
updated: 2026-10-25 updated: 2026-11-15
verified_on: 2026-10-25 verified_on: 2026-11-15
review_after: 2026-11-08 review_after: 2026-12-15
confidence: high confidence: high
tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved] tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved]
sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff] sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
@@ -14,7 +14,8 @@ sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
# Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23 # Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23
**Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences) **Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences)
**Status:** Resolved 2026-10-23 18:54 (patched and verified) **Status:** Resolved 2026-10-23 18:54 (patched and verified) — **but the fix was lost in the
plugin's 0.168.x restructure; `status: degraded` since 2026-11-15** (see the regression section below)
**Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its **Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its
memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's
memory layer. memory layer.
@@ -87,17 +88,39 @@ touching the file.
not positive proof that recall was populated on those calls. The proven part is that the call not positive proof that recall was populated on those calls. The proven part is that the call
is no longer abandoned by the memory layer. is no longer abandoned by the memory layer.
## ⚠ Regression — the fix was lost in the plugin restructure (verified 2026-11-15)
The follow-up below warned that a plugin update would revert the patch. **It happened.** Live
state at the 2026-11-15 03:00 stocktake:
- The plugin tree is **clean at upstream `0.168.2`** (`296c94b`, committed 2026-11-12 US /
2026-11-13 AWST) — the patch is gone entirely, not just uncommitted.
- The memory layer **moved**: `dashboard/plugin_api.py` (now a doc index) →
`dashboard/api/memory.py`. There, `MEMORY_TIMEOUT_S = 10.0` is hard-coded again
(`memory.py:48`), with **no** `CONDUIT_MEMORY_TIMEOUT_S` env override, and the "abandoning the
call" behaviour is intact (`memory.py:256`). The restructure landed in `memory.py` on
**2026-10-29 AWST**, so the 60 s budget has been gone for ~2.5 weeks.
- The running dashboard (started 2026-11-10) loads the restructured code, so the exposure is live.
- One real call has since occurred — 2026-11-14 08:57 AWST, the first entry in
`conduit-calls.json → history` — with no abort lines in the logs afterwards. But that call met a
warm provider; it does not exercise the cold-start path. The page's original caveat stands.
- **Not re-applied by the stocktake**: porting the fix to the restructured code is a code change to
a git-managed upstream plugin that the next update will revert again — it needs Anthony's call
(re-port + commit locally, or upstream it).
The original incident window (2026-10-22 → 23) remains resolved; `status: degraded` records that
the *protection* is gone, not that the October fault returned.
## Follow-ups ## Follow-ups
- ⚠ **The fix is still unexercised by a call** (see the correction above): `conduit-calls.json → - ~~⚠ **The fix is still unexercised by a call** …~~ **Superseded 2026-11-15**: one call has since
history` is empty at 2026-10-25 00:2x, and the plugin is now **five releases behind HEAD** occurred (2026-11-14 08:57 AWST) — but the fix itself was lost in the restructure (section above),
(`0.64.0` → `0.69.0`). so "unexercised" is now moot.
- **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which - **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which
asserted the capability did not exist **without calling the tool** → asserted the capability did not exist **without calling the tool** →
[[incidents/2026-10-24-conduit-call-request-refused]]. [[incidents/2026-10-24-conduit-call-request-refused]].
- **The patch is uncommitted.** A plugin update that resets - ~~**The patch is uncommitted.** A plugin update that resets `~/.hermes/plugins/conduit_push`
`~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream reverts it to the 10 s budget.~~ **Confirmed 2026-11-15** — see the regression section above.
(`kaishi00/hermes-conduit-notifier`).
- The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised. - The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised.
Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]] Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]]
@@ -3,9 +3,9 @@ title: Nous Portal Refresh Token Invalid — Context Summaries Unavailable
type: incident type: incident
status: open status: open
created: 2026-10-26 created: 2026-10-26
updated: 2026-10-26 updated: 2026-11-11
verified_on: 2026-10-26 verified_on: 2026-11-11
review_after: 2026-11-09 review_after: 2026-12-11
confidence: high confidence: high
tags: [hermes, router, memory, incident, unresolved] tags: [hermes, router, memory, incident, unresolved]
sources: [gateway-log, live-config, live-probe] sources: [gateway-log, live-config, live-probe]
@@ -42,7 +42,55 @@ sources: [gateway-log, live-config, live-probe]
- Re-authenticate the Nous Portal credential, or remove `nous` from `fallback_providers`. - Re-authenticate the Nous Portal credential, or remove `nous` from `fallback_providers`.
- Until then, a failover event will attempt a dead credential silently. - Until then, a failover event will attempt a dead credential silently.
> **Re-verified 2026-11-11 — the refresh-token failure is gone from the logs, and the exposure has
> changed shape rather than closed.**
>
> - **Zero `invalid_grant` / refresh-token lines** now appear in `~/.hermes/logs/errors.log` (which
> covers the window through 2026-11-10 19:19) — so the credential is no longer failing at *auth*.
> `~/.hermes/.env` carries no `NOUS_API_KEY`; `~/.hermes/auth.json` is present and live.
> - What appears instead is **404 `insufficient_credits_for_paid_model`** for
> `anthropic/claude-haiku-4.5` — 3 occurrences, most recent **2026-11-10 19:19:33**, from session
> `20261031_190813_95fbde`.
> - **A ninth auxiliary slot now points back at `nous`.** The 2026-10-25 remediation moved all
> *eight* slots off `nous` onto `custom:omni-101` / `no-think/freeapi/auto`, but
> `auxiliary.pulse_memory` (`provider: nous`, `model: anthropic/claude-haiku-4.5`) was added
> afterwards and **is** the slot producing those 404s. So the masking is intact for the original
> eight slots but a new, single-slot nous dependency exists and is **failing on credits**, not on
> auth.
> - `nous` is **still the first entry** in `fallback_providers`
> (`meituan/longcat-2.5-preview:free`) — that is a *free* model and has not been observed failing,
> so the failover risk described below is now "may burn an attempt", not "guaranteed dead".
>
> **Revised open item:** decide whether `auxiliary.pulse_memory` should be moved onto
> `custom:omni-101` like the other eight slots (it is the only active nous path left, and it is
> currently erroring), and/or add credits for paid nous models. The original re-authenticate item
> is **superseded** — auth works now.
> **Escalated 2026-11-11 (nightly audit).** The single-slot exposure described above became the
> busiest error path on the host, and it now has a **downstream casualty**.
>
> - `anthropic/claude-haiku-4.5` on `nous` returned **404 `insufficient_credits_for_paid_model`**
> **14 times** on 2026-11-11 (07:01:11 … last **19:19:38**) — up from 3 in the whole preceding
> window. All from session `20261031_190813_95fbde` (**source `desktop`, started 2026-10-31
> 19:08:13, still open**), whose pinned model is that same paid nous model.
> - The **404 is immediately followed by a 401.** The session's fallback chain moves the turn on to
> `opencode-zen` / `deepseek-v4.1-flash`, whose key expired on 2026-11-08 — **7 × HTTP 401
> `api_key_expired`** on 11-11 (07:01:13, 07:19:30, 08:03:40, 13:03:28, 13:19:33, 18:03:39,
> 19:19:39). So **both hops of this chain are dead**, and the pair repeats every time the session is
> poked. → [[systems/opencode-go]]
> - `auxiliary.pulse_memory` logged **19 lines** on 11-11 (06:0x, 08:0x, 13:0x, 18:0x), including
> **11 ×** *"main provider `custom:omni-101` is unavailable and no fallback_chain /
> fallback_providers is configured — refusing to guess another logged-in provider."* Live config
> re-confirmed at audit time: `pulse_memory` is the **only** slot carrying **no `fallback_chain`**
> (`{"provider": "nous", "model": "anthropic/claude-haiku-4.5"}`), while the other eight all chain
> `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` on `custom:omni-101`.
> - **Net effect:** pulse/voice memory retrieval failed on every attempt this window. `config.yaml`
> mtime is unchanged since **2026-11-09 22:58:07** — nothing was fixed.
>
> **Open item, unchanged and now evidenced:** move `auxiliary.pulse_memory` onto `custom:omni-101`
> like the other eight slots and give it a `fallback_chain`, and re-point or retire the stale desktop
> session pinned to a paid nous model. No credentials were minted or modified.
## Related ## Related
- [[systems/model-providers]]
- [[systems/current-profile]] - [[systems/current-profile]]
- [[systems/hermes-agent]] - [[systems/hermes-agent]]
@@ -3,8 +3,8 @@ title: Conduit Call Requests Refused by the Agent
type: incident type: incident
status: open status: open
created: 2026-10-25 created: 2026-10-25
updated: 2026-10-26 updated: 2026-11-19
verified_on: 2026-10-25 verified_on: 2026-11-19
review_after: 2026-11-08 review_after: 2026-11-08
confidence: high confidence: high
tags: [conduit, voice, tool-calling, agent-conduct, incident, unresolved] tags: [conduit, voice, tool-calling, agent-conduct, incident, unresolved]
@@ -93,3 +93,11 @@ into [[incidents/2026-10-22-conduit-memory-start-timeout]].
- The plugin **is** working for cards: `conduit_show` ×1 on 2026-10-25 14:37, delivered live. - The plugin **is** working for cards: `conduit_show` ×1 on 2026-10-25 14:37, delivered live.
- Plugin advanced to **`0.69.2`** and the dashboard gateway restarted at **18:42:33**, so upstream commits are live — see [[systems/conduit]]. - Plugin advanced to **`0.69.2`** and the dashboard gateway restarted at **18:42:33**, so upstream commits are live — see [[systems/conduit]].
- **Status stays open**: the request path is untested, not confirmed good. - **Status stays open**: the request path is untested, not confirmed good.
## Status update — 2026-11-19 (stocktake re-probe)
- `~/.hermes/conduit-calls.json` → `history` is **still empty (0 entries)** as of 2026-11-19 03:0x
AWST — the call path has now gone unused for ~26 days since the fault. The "verify a real call
end-to-end" open item remains outstanding.
- No new refusal episodes have been recorded against this page. **Status stays open** on the same
basis as before: untested, not confirmed good.
@@ -1,13 +1,13 @@
--- ---
title: Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop title: Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop
type: incident type: incident
status: open status: resolved
created: 2026-10-26 created: 2026-10-26
updated: 2026-10-26 updated: 2026-11-18
verified_on: 2026-10-26 verified_on: 2026-11-18
review_after: 2026-11-09 review_after: 2026-11-09
confidence: medium confidence: medium
tags: [cron, automation, hermes, incident, unresolved] tags: [cron, automation, hermes, incident]
sources: [live-sqlite-cron-executions, live-sqlite-sessions, gateway-log, jobs-json] sources: [live-sqlite-cron-executions, live-sqlite-sessions, gateway-log, jobs-json]
--- ---
@@ -55,6 +55,9 @@ scheduler admitted an unrelated session is unexplained and is the first thing to
execution row. execution row.
- Does `delivery_queued` clear on the 2026-10-26 08:30 slot? Check after that fire. - Does `delivery_queued` clear on the 2026-10-26 08:30 slot? Check after that fire.
## Resolution (verified 2026-11-18)
**Resolved — no recurrence in 23 days.** The `delivery_queued` state cleared (the open question above is answered: it did **not** persist past the 2026-10-26 08:30 slot). Job `08eb86f02d37` now reads `enabled`, `last_status: ok`, `last_error: null`, `failure_streak: 0`, last run 2026-11-17 08:33. In the retained `executions.db` window (2026-11-12 → 11-17; DB retention floor ~18 d means earlier slots have aged out) it ran **exactly once per day, `completed` every time, zero failed rows** — no refires since the 10-25 loop. Root cause of the 10-25 session-adoption remains hypothesised, not confirmed; the wall-clock-budget follow-up was never implemented, so the *exposure* (a runaway can recur silently) is unchanged even though the incident itself has not recurred.
## Related ## Related
- [[systems/scheduled-tasks]] - [[systems/scheduled-tasks]]
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] - [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
@@ -0,0 +1,125 @@
---
title: Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in)
type: incident
status: resolved
created: 2026-10-27
updated: 2026-11-19
verified_on: 2026-11-19
review_after: 2026-12-19
confidence: high
tags: [gateway, cron, scheduler, lifecycle, restart, memory, silent-failure]
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-probe]
---
# Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in)
## Headline
On **2026-10-26 at 14:33:35 AWST** the gateway received a **planned `SIGTERM`** (`under_systemd=yes`,
`parent_name=systemd`) — and the shutdown drain **still interrupted a running cron execution**, because
the drain reported `timed_out=True` after only **1.35 s**. The orphaned run was
`ea79a2107e36` **Mnemosyne Health Monitor** (12-hourly `0 */12 * * *`), which had been claimed at
**12:01:13** and was **2 h 34 m** into its work.
This is the **second instance** of the orphaned-cron-run class documented for 2026-10-20
([[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]), with three material differences:
| | 2026-10-20 | 2026-10-26 |
|---|---|---|
| Stop type | **unclean exit** (no exit path, `suspected_oom=False`) | **planned `SIGTERM`** under systemd |
| Execution closed as | `unknown` ("owner exited before a durable terminal state") | **`failed`** with a **new** error string |
| Incident row raised | **no** | **yes — two** |
| Failure delivered | no | **yes**, 14:35:59 |
So the loud-orphan recommendation from the 10-20 page is **now partly implemented** — the run no longer
disappears silently. What it did not do is *finish the work* or *avoid the interruption*.
## Timeline (all times AWST, 2026-10-26)
| Time | Event | Source |
|---|---|---|
| 12:01:13 | `ea79a2107e36` Mnemosyne Health Monitor execution claimed, `running` | `executions.db` |
| 00:01:46 → 00:10:45 | the same job's previous (00:01) slot — **completed in 9 m** | `executions.db` |
| 14:33:35 | `Received SIGTERM as a planned gateway stop`, `under_systemd=yes`, `parent_name=systemd`, `loadavg_1m=2.15` | `gateway.log` |
| 14:33:37 | `Thread 34795 not found` while sending the shutdown notice (same dead thread as the birthday/Uber jobs) | `gateway.log` |
| 14:33:37 | incident row `ea79a2_c3ab2d784ba3` opens — *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/724a69108b…json`"* (`alerted`) | `cron_incidents` |
| 14:33:38 | `Shutdown drain: 1 in-flight cron job(s) … drain took 1.35s, timed_out=True` → `interrupting remaining work` | `gateway.log` |
| 14:34:25 | `1 authority task(s) did not settle within 40s; left for recovery`; `Profile /home/hermes/.hermes still has live writers at shutdown; its admissions stay for recovery` | `gateway.log` |
| 14:34:27 | `ERROR gateway.session_authority: Admitted turn 28ad127ffb2a44c0bcd17283a93aa496 failed`; cron session `local-253a96…` **paused: runtime_draining** | `gateway.log` |
| 14:34:27 | that session finalises with `end_reason: cron_complete` (66 msgs, 39 tool calls, 127 API calls) | `state.db` |
| 14:34:29 | `Gateway stopped (total teardown 54.06s)`; **`Skipping .clean_shutdown marker — drain timed out with interrupted agents`** | `gateway.log` |
| 14:35:18 | next gateway life starts (`Starting Hermes Gateway...`) | `gateway.log` |
| 14:35:59 | execution closes **`failed`**: `RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership` | `executions.db` |
| 14:35:59 | incident row `ea79a2_a4661554a3c2` opens on that RuntimeError (`alerted`); execution `c5ac27e3…` **delivered** the failure | `cron_incidents`, `deliveries.db` |
| 14:46:12 / 15:01:15 | Mnemosyne Watchdog `96eb95ce12f2` fails twice (`Script exited with code 1`) reacting to the log `ERROR` — rows `96eb95_0c89bd373a0a`, `96eb95_e32174134546` | `cron_incidents` |
| 15:16:12 | both watchdog rows **`resolved`** (the error aged out of its 15-min log window); watchdog back to `ok` | `cron_incidents` |
## What is new or unexplained
1. **A planned stop is enough to orphan a run.** No crash was required. The drain had a **30 s** budget and
reported `timed_out=True` after **1.35 s** with `cron_now=0` — i.e. the drain did **not** wait out its
budget before taking the interrupt path. Logged verbatim; **not diagnosed**.
2. **New terminal error string:** `lost its durable fire claim ownership`. The 10-20 orphan closed as
`unknown` with a scheduler-side note; this one closed as `failed` from the worker's own perspective.
Any alerting or triage that pattern-matches on the 10-20 wording will miss this one.
3. **Two rows for one fault.** `ea79a2_a4661554a3c2` (fire claim) and `ea79a2_c3ab2d784ba3` (unverified /
reconcile admission) both remain **`alerted`, unclosed** as of 2026-10-27 00:1x, and the job carries
**`failure_streak: 2`** from a **single** interrupted fire.
4. **The admission was reconciled.** `~/.hermes/cron/admissions/` no longer holds `724a69108b…json`
(only the 00:00 / 00:01 files from the following night), so the "reconcile" step in the row's text has
been satisfied — the rows were simply never closed.
5. **The watchdog works.** This fault **did** alert, twice, and self-resolved — which is a genuine
improvement over 10-20 (one wakes up via the incident ledger, not via a missing digest).
## Impact
- **One 12-hourly memory consolidation cycle did not run.** `ea79a2107e36` is the Mnemosyne
health/consolidation job → [[systems/mnemosyne]]. The 00:01 slot completed normally; the 12:01 slot
produced no completed work.
- **Day tally:** 2026-10-26 settled **169 `completed` / 3 `failed`** — and **all three failures are this one
fault** (1 Health Monitor + 2 derivative Watchdog reds). 2026-10-25 was **172 / 2**.
- **Session cost sunk:** the interrupted run burned 66 messages / 39 tool calls / 127 API calls
(~597 k input, 254 k output tokens) before the interrupt.
- **Unclosed incident rows 2 → 3** (`0e753e_e7f8194a7d06` carried over from 2026-10-01, plus the two new
`ea79a2_*` rows).
## Recommended follow-ups (not yet done)
1. Decide the drain policy for **long-running cron jobs**: this job legitimately occupies 9 m → 2 h+.
Either exempt a cron execution from the drain deadline, or interrupt it with a durable
*"interrupted → re-fire this slot"* marker instead of a terminal `RuntimeError`.
2. **Close or ack the two `ea79a2_*` rows** — the admitted turn failed and the admission is gone, so they
are historical.
3. **Verify the drain-timeout arithmetic** (`timed_out=True` at 1.35 s against a 30 s budget) — if the
deadline is being bypassed whenever a writer is live, every future restart can interrupt work.
4. Confirm `failure_streak` should count the *unverified* row at all — a single interrupted fire reading
**streak 2** inflates the job's apparent health.
## Related
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — first instance of this class
- [[systems/scheduled-tasks]] — fleet ledger and per-window tallies
- [[systems/mnemosyne]] — the interrupted job's purpose and settings
- [[systems/hermes-agent]] — gateway lifecycle and restart history
## Update — 2026-10-28 (nightly audit)
- **No recurrence.** Every run in the 2026-10-27 window except the radar's budget overrun completed, and the
watchdog logged no alert for the day.
- **Both incident rows this fault raised closed automatically** — `ea79a2_a4661554a3c2` and
`ea79a2_c3ab2d784ba3` **auto-closed 2026-10-27T00:38:32+08:00** (unclosed rows fleet-wide fell 3 → 2).
- The `lost its durable fire claim ownership` string did not appear in any 10-27 execution row.
- Open question from the original write-up still stands: why a *planned* stop bypasses the drain deadline
(`timed_out=True` at 1.35 s of a 30 s budget).
## Update — 2026-11-19 (stocktake re-probe)
- **Both `ea79a2_*` rows are now closed** (follow-up 2 satisfied): `ea79a2_c3ab2d784ba3`
`resolved` 2026-10-27T00:38:32, and `ea79a2_a4661554a3c2` **`closed` 2026-11-07T15:43:13**
(live probe of `cron_incidents` in `~/.hermes/cron/executions.db`).
- The `lost its durable fire claim ownership` error string has **no rows in the retained
`executions` window** — no recurrence since 2026-10-27 (subject to the DB's ~18-day retention
floor; the 10-28 update already confirmed the 10-27 window clean).
- **Status → `resolved`.** The fault did not recur and its incident rows are closed. Follow-ups
1, 3 and 4 (drain-policy decision, drain-timeout arithmetic, `failure_streak` semantics)
remain open design questions and were **not** actioned.
@@ -0,0 +1,128 @@
---
title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response
type: incident
status: resolved
created: 2026-10-28
updated: 2026-10-30
verified_on: 2026-10-30
review_after: 2026-11-13
confidence: high
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api]
---
# Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response
## Headline
On **2026-10-27** the Daily Marketing Job Radar (`dca8482e4f76`) ran **07:01:08 → 11:03:54 AWST**
(**4 h 2 m 46 s** wall clock), closed **`failed`**, and delivered a report that opens:
> **Daily Marketing Job Radar — run 2026-10-27 — INCOMPLETE (tool-call budget exhausted before Notion write stage)**
Discovery, verification, dedup and classification completed. **Notion ingestion, cover-letter generation and the
final cover-letter audit were not executed** — so the run is recorded as failed rather than successful.
Self-reported run time **10:47:57**; freshness cutoff **2026-10-20**.
This is the second consecutive budget overrun: the **2026-10-26** 07:00 slot ran 07:01:08 → 09:10:27
(**2 h 9 m 19 s**), closed **`completed`**, and *still* delivered the same `RUN INCOMPLETE` marker — the day's
backlog had to be finished by hand. 2026-10-27 is the first overrun recorded as **`failed`**, which is the
improvement: the failure reached the incident ledger instead of hiding inside a green row.
## Ledger state
| Field | Value |
|---|---|
| Execution row | `dca8482e4f76`, started `2026-10-27T07:01:08.265913+08:00`, finished `2026-10-27T11:03:54.143535+08:00`, `failed` |
| Delivery | `delivered` (the incomplete report reached Telegram topic 1030) |
| Incident row | `dca848_e93a538ade6b` — `state: alerted`, `failure_type: auth`, first/last seen `2026-10-27T11:03:53.715494+08:00`, alerted `11:03:59.013115` |
| Incident status | **open at the 2026-10-28 00:2x audit** — no ack, no close |
| `jobs.json` | `last_status: error`, `failure_streak: 1`, pin unchanged at `custom:omni-101` / `smart-route` |
⚠ `failure_type: auth` does **not** describe this fault. A tool-call budget exhaustion is being classified as an
auth failure; the field is not trustworthy as a diagnosis for this job.
## What the run actually did (from its own report)
- Live Notion DB `0f90ba2b-8b10-4d02-a62c-6f692d5b1168`: **451 entries**, `Date Added = 2026-10-27` → **0**.
- Board composition at run time: **Agent — Rhino 402, Chaos 12, Manual 2, blank 35**; **Source — SEEK 295,
LinkedIn 88** (row text truncated in the ledger).
## Response, same day: the radar is now two jobs
At **2026-10-27 17:34:15** the single radar job was split into a two-stage pipeline (see
[[runbooks/job-radar]] for the full contract):
| Stage | Job | Schedule | Delivery | Job id |
|---|---|---|---|---|
| 1 — Discovery | Daily Marketing Job Radar - Discovery — `DISCOVERY → VERIFY → DEDUP → SCORE`, no Notion write, no cover letters; writes a handoff file | `0 7 * * 1-5` | `local` (failures → Telegram topic 1030) | `dca8482e4f76` (unchanged) |
| 2 — Write and Letters | Daily Marketing Job Radar - Write and Letters — lands the handoff: Notion entries, cover letters, audit, report | `30 11,14 * * 1-5` | Telegram group `-1004321904721`, topic `1030` | `5fc292ff0f3b` (created 17:34:15) |
Stage 1 keeps its id, pin (`custom:omni-101` / `smart-route`) and its own tool budget; stage 2 is a separate job
whose prompt lands the handoff rather than repeating the discovery narrative. Two test fires of stage 2 ran the same
evening — **17:35:00 → 17:36:52** and **17:38:38 → 17:43:42** — both `completed`, both with delivery
`suppressed`, `last_status: ok`, streak 0.
Fleet size went **28 → 29 jobs** (26 enabled / 3 disabled).
## Why it matters
1. **The failure mode is partial work, not a lost run.** Four hours of browser-heavy discovery produced a
correct report and zero writes. Nothing in `jobs.json` distinguished it from a good day except
`last_status: error` on 10-27 (`completed` on 10-26, with the same incompleteness buried in the text).
2. **Splitting is the mitigation, not a fix.** Stage 2 inherits its own tool budget; if stage 1 still overruns
and never writes the handoff file, stage 2 has nothing to land — watch the 11:30 slot for that.
3. **The budget is consumed by discovery**, which is exactly the stage that now owns it alone.
## Open questions
1. **Verify one full cycle end-to-end** (stage 1 07:00 → stage 2 11:30): does stage 2 find stage 1's handoff
file, do the Notion writes and cover letters land, and is `MISSING_COVER_LETTER_COUNT=0` reported?
2. **Make budget exhaustion self-describing.** The classifier tagged it `auth`; a distinct budget/partial-run
type would put this on the same footing as any other failure.
3. **Do the two stages fit the same wall-clock envelope?** Stage 1 must finish inside its own budget *and* leave
the handoff before 11:30.
4. The incident row stays open until a clean stage-1 slot auto-closes it — confirm the 2026-10-28 07:00 slot
closes `dca848_e93a538ade6b`.
## Follow-up — the 2026-10-28 slot (recorded 2026-10-29)
The **first real two-stage cycle ran on 2026-10-28** and answered the open questions above — not favourably:
- **Stage 1 overran again, on a new signature.** `dca8482e4f76` ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**),
closed **`failed`**, `failure_streak` **2**, on *"hit the agent iteration cap before the handoff could be
written"* → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]].
- **No handoff file was written** — `~/.hermes/radar-ref/handoff/` held only `2026-10-27.json` at the 2026-10-29
audit, so **Notion writes and cover letters did not run again**.
- **Both stage-2 slots no-oped correctly** — `5fc292ff0f3b` fired **11:30:27 → 11:31:31** and
**14:30:27 → 14:31:43**, both `completed` with `delivery_outcome: suppressed`. Open question 1 is therefore
answered only in the negative: the recovery rule ("a stage-2 run with no handoff is a no-op") holds in
production, but nothing has yet proved the positive path end-to-end.
- **Open question 4 is answered: the row did not close.** `dca848_e93a538ade6b` stayed open because the 10-28
slot was not clean, and a second row (`dca848_0a2f03b5952b`) opened — open incident rows **2 → 4**.
## Follow-up — the 2026-10-29 slot: the split held (recorded 2026-10-30)
**Resolved.** Stage 1 (`dca8482e4f76`) ran **2026-10-29 07:00:59 → 10:16:12** — 3 h 15 m 13 s — and
closed **`completed`**. It wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` (24.8 KB):
`run_date 2026-10-29T08:23:51+08:00`, **170 discovered / 33 duplicates avoided / 37 verified**,
sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Both ledger rows this incident opened —
`dca848_e93a538ade6b` and `dca848_0a2f03b5952b` — closed **`resolved` at 10:16:12**, and the run
opened no new row. Open cron incident rows went **4 → 1**; the survivor is the Facebook monthly's
`0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot.
Stage 2 (`5fc292ff0f3b`) fired at **10:13:46**, consumed the handoff at **10:40:59**, finished
**10:44:48 `completed`** and **delivered** to topic 1030 in the same second (delivery `delivered`).
Its `stage2_result`: **20 pages written / 20 `page_ids` / 2 cover letters / 2 same-day strong matches
/ `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went
**455 → 475**. The later slots no-oped as designed (11:30 → 10 m 46 s, 14:30 → 1 m 51 s, already-consumed
handoff). Stage 1 itself no longer delivers — correct, because stage 2 owns the report.
The questions this page left open are answered for this run: the handoff **was** written inside budget and
consumed, and **no cover letter went missing on the first real run of the split**.
## Related
- [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun
- [[systems/scheduled-tasks]] — fleet ledger, job table and the 2026-10-27 window
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — the radar's previous failure mode
@@ -0,0 +1,99 @@
---
title: Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped
type: incident
status: resolved
created: 2026-10-29
updated: 2026-10-30
verified_on: 2026-10-30
review_after: 2026-11-13
confidence: high
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir]
---
# Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped
## Headline
On **2026-10-28** the Discovery stage of the Daily Marketing Job Radar (`dca8482e4f76`) ran
**07:00:32 → 10:33:18 AWST** (**3 h 32 m 46 s**) and closed **`failed`** on *"hit the agent iteration cap before
the handoff could be written"* — its **second consecutive failure** (`failure_streak: 2`).
This is the first real cycle of the **two-stage split** introduced on 2026-10-27 (see
[[incidents/2026-10-27-job-radar-budget-exhausted]] and [[runbooks/job-radar]]), and it produced a clean negative
result: **stage 1 never wrote its handoff file, so there was nothing for stage 2 to land.** Both stage-2 slots
ran anyway — **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed`, both
`delivery_outcome: suppressed` — which is the designed no-op path working correctly in production for the first
time. Notion ingestion and cover letters did not execute for the day.
## Ledger state
| Field | Value |
|---|---|
| Execution row | `dca8482e4f76`, started `2026-10-28T07:00:32.463309+08:00`, finished `2026-10-28T10:33:18.833788+08:00`, **`failed`** |
| Duration | **3 h 32 m 46 s** (10-27: 4 h 2 m 46 s; 10-26: 2 h 9 m 19 s) |
| `jobs.json` | `last_status: error`, **`failure_streak: 2`**; `last_error` opens *"Daily Marketing Job Radar — Stage 1 (Discovery) — 2026-10-28 — hit the agent iteration cap before the handoff could be…"*; pin unchanged (`custom:omni-101` / `smart-route`) |
| Incident row | `dca848_0a2f03b5952b` — `state: alerted`, `failure_type: timeout`, first_seen `2026-10-28T10:33:18`, **open** |
| Handoff | **not written** — `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json` (verified 2026-10-29 00:3x) |
| Stage 2 | `5fc292ff0f3b` fired `11:30:27 → 11:31:31` and `14:30:27 → 14:31:43`, both `completed` / `delivery_outcome: suppressed` |
The 10-27 row (`dca848_e93a538ade6b`, classified `failure_type: auth`) did **not** close — the 2026-10-28 07:00
slot was not clean. Open `cron_incidents` rows went **2 → 4**.
## What the split proved, and what it did not
1. **It proved the recovery rule.** *"A stage-2 run with no handoff is a no-op, not a failed search"* — exactly as
written in [[runbooks/job-radar]] — held in production: stage 2 invented no work, raised no error, and sent
nothing to topic 1030.
2. **It did not make discovery finish.** The two post-split runs died at *different* ceilings:
- **2026-10-27** — stopped at the **tool-call budget** after 4 h 2 m: discovery/verify/dedup/classify complete, writes never started.
- **2026-10-28** — stopped at the **agent iteration cap** after 3 h 32 m, *earlier in the pipeline*: no handoff, so nothing downstream could run at all.
The wall-clock number improving (4 h 2 m → 3 h 32 m) is **not** an improvement — less of the pipeline completed.
3. **The write stage is no longer the risk; the discovery stage still is.** Splitting removed the
"writes lost inside a green row" failure mode. What remains is that stage 1 cannot both finish discovery and
reach its handoff write inside its own limits.
## Why it matters
- **A lost day is now cheap to detect but easy to miss.** Stage 1 raises an incident row and flips `last_status`
to `error`; stage 2's *correct* silence (`suppressed`) means the only outward sign of a workless day is stage
1's own failure report — not a "0 jobs added" report.
- **Two consecutive failures, two different classifiers.** 10-27 was tagged `failure_type: auth` (wrong for a
budget fault); 10-28 was tagged `timeout` (also only approximate — an agent-side iteration cap, not a network
timeout). Neither string names the actual cause.
- **Budget arithmetic is not the fix.** If the cap fires after ~3.5 h of discovery, the pipeline needs a smaller
discovery surface (fewer sources per slot), **staged/checkpointed handoff writes** (so a capped run still
leaves something landable), or an explicit per-phase iteration allowance.
## Open questions
1. **What is the iteration cap for this job, and can stage 1 reach its handoff write before it?** The 10-28 run
died *before* the write; the 10-27 run died *after* discovery but *before* writes. Neither produced a handoff.
2. **Should stage 1 checkpoint a partial handoff** (after DEDUP/SCORE) so a capped run gives stage 2 something to
land instead of a silent no-op day?
3. **Make the abort self-describing** — a distinct `iteration-cap` / `partial-run` failure type rather than
`auth` / `timeout`.
4. Both radar incident rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) close only when a **clean** stage-1
slot runs — watch the next 07:00 weekday slot.
## Follow-up — the 2026-10-29 slot: the cap was not hit (recorded 2026-10-30)
**Resolved for this run.** The next stage-1 run (`dca8482e4f76`, **2026-10-29 07:00:59 → 10:16:12**,
3 h 15 m 13 s) closed **`completed`** *and* wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — the
first run of the split to do both. The handoff records **170 discovered / 33 duplicates avoided /
37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`,
`indeed blocked`, `wa_gov ok`. Stage 2 consumed it at **10:40:59**, delivered at **10:44:48** with
**20 Notion pages and 2 cover letters** (`missing_cover_letter_count 0`; board 455 → 475). Its two
ledger rows closed `resolved` at 10:16:12.
⚠ **Flaky, not solved.** A 3 h 15 m stage-1 run is still long enough for the cap to be reachable, and
the previous two slots in the split both died on it (2026-10-27 on the tool-call budget, 2026-10-28 on
the iteration cap). Treat a `completed` run with **no handoff file** as the same fault, whatever the
execution status says → [[incidents/2026-10-27-job-radar-budget-exhausted]].
## Related
- [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up
- [[incidents/2026-10-27-job-radar-budget-exhausted]] — the prior day's budget exhaustion and the split
- [[systems/scheduled-tasks]] — fleet ledger and the 2026-10-28 window
- [[systems/browser-backend]] — the browser layer this run depends on
@@ -0,0 +1,106 @@
---
title: Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hour Session TTL, Dead STT/TTS Providers
type: incident
status: resolved
created: 2026-10-30
updated: 2026-10-30
verified_on: 2026-10-30
confidence: high
tags: [conduit, voice, tts, stt, config, session-ttl, incident]
sources:
- live-probe-2026-10-30
- live-config-2026-10-30
- live-log-2026-10-30
- session-20260908_031541_49c051
- plugin-tree-conduit_push
---
# Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hour Session TTL, Dead STT/TTS Providers
**Window:** 2026-10-29 07:54 – 09:0x AWST · **Fixed:** 2026-10-29 08:02–08:4x AWST (dashboard restarted 08:03:26)
**Symptom:** Anthony asked *"why do you never answer when I call you on Conduit"* — every Conduit voice call connected and then died silently. No Mac connection, no Hermes turn, no error surfaced on the phone.
Three independent faults sat in series. Fixing any one alone would not have made a call work; each one masked the next.
## Evidence trail (why it was invisible)
- The Mac Mini speech server recorded **zero connections all day.** `last_request_at` on the plugin was `null` — meaning no call had *ever* reached the Hermes turn logic.
- `conduit-calls.json → history` was empty: no successful call had ever been recorded.
- The only traces were in the plugin log: a wrong-looking reason string per attempt, which is what made each fault look like the *next* fault.
## Fault A — Truncated speech-server URL (403)
`voice.speech_server.url` was set to the bare host path:
```
wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/ ⛔ (truncated)
wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime ✅
```
`plugin_api.py` (~line 2646) dials the configured value **verbatim** — it does not append `/realtime`. The bare `/v1/` returns HTTP **403**, the socket closes, and the plugin reports it as *"the speech server refused the key."* The key was fine; the path was wrong.
**Fix:** set `voice.speech_server.url` to `…:8443/v1/realtime`. Backup: `~/.hermes/config.yaml.bak.url-081525` (08:15:25).
## Fault B — Dashboard session TTL of 12 hours (WS 4401)
`dashboard.basic_auth.session_ttl_seconds` held the **12-hour default (43 200 s)**. The dashboard issues the phone its session, and when that session lapses the Conduit call socket answers **WS 4401 "Dashboard sign-in required"** — with no Hermes turn and no user-visible error.
Audit-log corroboration: the iPhone **re-authenticated at 08:00:59**, *minutes after* the silent call — i.e. it was walking around with an expired session and only logging back in when the dashboard happened to prompt.
**Fix:** raised to **30 days (2 592 000 s)** — matched to the refresh-token life so the phone's session no longer expires under it. Dashboard restarted **2026-10-29 08:03:26** (pid 4042003). Backup: `~/.hermes/config.yaml.bak.ttl-20261029080258`.
> ⚠ Two keys exist and only one was changed: `dashboard.basic_auth.session_ttl_seconds` = **2592000** (raised) and `dashboard.session_ttl_seconds` = **43200** (still 12 h, different key, different consumer). Check *both* when diagnosing a 4401 again.
## Fault C — Host STT and TTS providers both dead
Both halves of the voice loop were broken on the host, so even a correctly-signed call would have had no ears and no mouth.
| Setting | Was | Failed with | Now | Verified |
|---|---|---|---|---|
| `tts.provider` | `openai` | Mac `/v1/speech` → **502** | `ollama_local` | 6.94 s of real 64 kbps speech produced |
| `stt.provider` | `groq` | **403 / Cloudflare error 1010** | `local` | Hermes' own transcription path returned `success: True, provider: local` |
- Groq was **blocked, not mis-keyed** — it returns 403/1010 identically with a bogus key, so the failure was Cloudflare bot-blocking, not credentials.
- `tts.provider` had been flipped to `openai` by the **Conduit settings tool** earlier (that tool writes Hermes config). The app holds no `tts.provider` setting, so this fix will not be silently overwritten.
- `ollama_local` runs `~/.hermes/scripts/ollama-tts.sh` (model `phonon`); `local` STT runs `faster_whisper` (`base`).
- Backup: `~/.hermes/config.yaml.bak.stt-084536` (08:45:36); the TTS flip landed between the 08:15 and 08:45 backups.
## Verification (end-to-end, not just config)
Rather than trusting three config edits, the whole call path was driven for real: minted a valid HMAC dashboard session → got a WS ticket → let the plugin dial the Mac. Result:
- auth gate passed → `session.created` / `session.updated`
- speech in transcribed correctly ("Hello Conduit, can you hear me right now?")
- turn committed → audio streamed back → `response.done`
- **Mac counters 0 → 10 connections, 1 completed response, 2.98 s audio, 0 errors**
- plugin `connected: true`, handshake **41 ms**
- the iPhone minted its own WS tickets at **08:07** and **08:19** — i.e. against the new 30-day session
## App update to 2.2.0 changed the settings schema
The iPhone app self-updated to **2.2.0** (from 1.10.1) mid-window, and its settings schema changed with it: the phone now exposes **19 keys, none of them `voice.local.*`**. There is **no phone-side Local Live server address**, so the Local Live path cannot be configured from the host side at all — a prior assumption that it could is now obsolete.
Consequence: `voice.mode` was moved **`local` → `classic`** (request queued 08:47, approved in the app; `conduit-settings.json` now reads `classic`). Classic needs no phone-side server config because the plugin relays the whole call — which is precisely why Fault C (both host providers dead) had to be fixed for it to work at all.
## Call record
`conduit-calls.json → history` now holds **2 entries — the first real calls ever recorded**:
| Time (AWST) | Outcome |
|---|---|
| 2026-10-29 08:38:04 | No Mac connection, no Hermes turn. iPhone had to re-login at 08:39:05, a minute later. |
| 2026-10-29 08:49:36 | Placed *after* the Classic switch and the STT/TTS fixes. Session ended 08:49:35 — Anthony's verdict is unrecorded. |
## Open items
- The 08:49:36 Classic call has **no recorded outcome**. Nothing in the window proves a phone-originated call ran a real Hermes turn.
- `last_request_at` was still `null` at the end of the window: the call→Hermes-turn bridge remains **unproven** even though the transport is now healthy. Classic mode is the recommended path.
- If an app update or the settings sync ever writes the bare `/v1/` back into `voice.speech_server.url`, every call silently 403s again. Re-check it after any Conduit update.
- Watch for the Conduit settings tool re-flipping `tts.provider` should a `tts.provider` key ever appear in the app's schema.
## Related
- [Conduit](../systems/conduit.md) — service page (voice pipeline, call path, pairing)
- [Hermes Agent](../systems/hermes-agent.md) — TTS/STT provider block
- [Conduit call request refused (2026-10-24)](2026-10-24-conduit-call-request-refused.md) — the earlier, behavioural half of the "calls don't arrive" problem
- [Conduit memory-start timeout (2026-10-22)](2026-10-22-conduit-memory-start-timeout.md)
@@ -0,0 +1,103 @@
---
title: CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30)
type: incident
status: resolved
created: 2026-10-31
updated: 2026-10-31
verified_on: 2026-10-31
confidence: high
tags: [cron, gateway, host, outage, pve, ct460, incident]
sources: [live-executions-db, live-jobs-json, live-gateway-exit-diag-log, live-agent-log, live-deliveries-db, ssh-pve-uptime]
---
# CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30)
**Status: RESOLVED** — the container and scheduler came back on 2026-10-30 ~16:28–16:37 and the day
closed clean (138 executions, 136 `completed`, **0 `failed`**). **Root cause of the host reboot is not
established.** No monitor alerted; the gap was found by this audit.
## Summary
The Hermes runtime went dark at **~10:58 AWST on 2026-10-30** and did not return until
**16:28** (container) / **16:37:21** (first execution). The PVE host `pve` itself rebooted — its boot
time is **2026-10-30 16:24:00** — so this was a **host-level outage, not a cron or gateway fault**.
The gateway's previous life died with no exit path, which the lifecycle ledger logs as
`SIGKILL / OOM / VM death`. Nothing noticed for **5 h 37 m**.
## Timeline (all times AWST, 2026-10-30)
| Time | Event | Evidence |
|---|---|---|
| 10:49:54 | Last execution before the gap starts | `executions.db` last row `< 16:37` |
| ~10:58:20 | Gateway life `pid=3913841` dies; no exit path ran | `gateway-exit-diag.log` `gateway.previous_unclean_exit` (prior life started 2026-10-28T17:21:58Z) |
| 10:58:20 → 16:37:21 | **Zero execution rows** (5 h 39 m) | `executions.db` count in window = **0** |
| 16:24:00 | PVE host `pve` (`100.96.100.82`) boots | `ssh root@100.96.100.82 'uptime -s; who -b'` → both read `2026-10-30 16:24` |
| 16:28:00 | CT460 container boots | local `uptime -s` → `2026-10-30 16:28:00` |
| 16:35:50 | Unclean exit noticed and logged | `logs/agent.log`, `logs/errors.log` — `Previous gateway life (pid=3913841, started_at=2026-10-28T17:21:58Z) exited UNCLEANLY (no exit path ran — SIGKILL / OOM / VM death)` |
| **16:37:21** | Scheduler resumes — **8-run catch-up burst** (16:37:21 → 16:38:22) | `executions.db` |
| 23:59:58 | Daily wiki audit (this run) starts; the 16:37 health-monitor catch-up is still `running` | `executions.db` |
## The resume burst (16:37:21 → 16:38:22)
Eight jobs fired within 61 seconds of the scheduler coming back. Seven closed `completed`;
the eighth is still running.
| Job | Started | Closed |
|---|---|---|
| FreshRSS Midday News Briefing (`ecc1fc13655a`) | 16:37:21 | 17:45:31 — the 12:00 slot, delivered late |
| Mnemosyne Auto-Consolidate | 16:37:21 | 16:37:54 |
| Mnemosyne Embedding Coverage Monitor | 16:37:21 | 16:41:31 |
| Mnemosyne Watchdog | 16:37:21 | 16:38:33 |
| **Mnemosyne Health Monitor (`ea79a2107e36`)** | **16:37:21** | **RUNNING at audit** — the late 12:01 slot |
| Job Tracker Dashboard Refresh | 16:37:23 | 16:37:35 |
| 🦝 Hermes Config Backup to Gitea (Daily) | 16:38:06 | 16:42:02 |
| Daily Marketing Job Radar - Write and Letters | 16:38:22 | 16:41:39 |
## Impact
- **≈39 scheduled runs lost.** 2026-10-29 fired **177** executions (177 `completed`, 0 `failed`); 2026-10-30
fired **138** (136 `completed`, 0 `failed`, 2 `running` — the 16:37 catch-up plus this audit). No row exists
inside the dark window, so the deficit is unfired slots, not failures.
- **No failures were caused.** Nothing was recorded `failed` on 2026-10-30 — the outage simply removed the
opportunities to run.
- **The two long Mnemosyne runs absorbed it.** The FreshRSS midday briefing regenerated on the late fire
(16:37:21 → 17:45:31), and the health monitor's 12:01 slot is the still-running 16:37:21 fire — normal for
this job, whose 00:01 slot took **10 h 01 m 40 s** the same day.
- **Deliveries were unaffected once back up:** `deliveries.db` holds **8 rows for 2026-10-30, all `delivered`**
(latest 18:46:04) — the opposite of the 03:3x snapshot, which read zero rows because the day had barely started.
- **2026-10-31 opened clean:** 5 executions (00:01:05 ×3, 00:15:59, 00:23:57), all `completed`.
## Root cause — not established
- The host rebooted. `pve` boot time is 16:24:00; **no OOM evidence was found** on the CT460 side (local
`dmesg` grep empty, `/var/log/wtmp` absent, no `suspected_oom` marker in the webui session store).
- The ledger's own wording (`SIGKILL / OOM / VM death`) is a **list of candidates for "no exit path ran"**, not a
finding. Treating it as a diagnosis would be wrong.
- Same-day, different host: [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] records **m93p**
reboots (~15:55) that took down NFS-backed containers. The two events are **separate hosts and separate
causes as far as the evidence goes** — they are not merged here, but the timing is close enough to warrant
checking the two hypervisors' last-boot records together before either is called a one-off.
- **To close it:** PVE syslog / `journalctl -b -1` on `100.96.100.82` around 2026-10-30 10:58, and the host's
own last-boot records (`last -x reboot`). Neither was read for this page — say so rather than guess.
## Why nothing alerted
The jobs positioned to notice a host-down event are **disabled**:
| Job | ID | State |
|---|---|---|
| NFS mount monitor | `cbfe434a3b5e` | disabled |
| m93p LXC boot monitor | `1a8e90ea2271` | disabled |
Everything else is *inside* the scheduler that went dark, so it could not have noticed. **A host-level outage of
the Hermes VM currently has no in-estate detector** — the fleet learned about it from the 23:59 audit, 13 h
after the fact. This is the incident's one open item.
## Related
- [[systems/scheduled-tasks]] — the fleet entry for this window
- [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — the same day's health-monitor story: its 00:01 slot
closed `unknown` at 01:04:22, incident `ea79a2_d93c3acdc4f2` **auto-closed `resolved` at 10:02:48** when that
run finally completed, and the job is live again (`enabled: true`, `last_status: ok`, streak 0)
- [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] — sister-host reboot, same day
- [[infrastructure/hosts]] — `pve` / CT460 addressing
@@ -0,0 +1,84 @@
---
title: m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start
type: concept
status: resolved
created: 2026-10-30
updated: 2026-10-30
verified_on: 2026-10-30
review_after: 2026-11-30
confidence: high
tags: [proxmox, m93p, nfs, lxc, outage, freshrss]
sources: [live-probe, ssh-m93p, systemctl-status, pct-list, freshrss-script-output]
---
# m93p NFS Mount Loss — All NFS-Backed LXC Containers Failed to Start (2026-10-30)
**Window:** 2026-10-30 ~15:55 AWST (host boot) → 17:20 AWST (manual recovery)
**Status:** **Resolved.** All 18 LXC containers on m93p are running; self-healing recovery unit installed.
**Impact:** Every LXC whose rootfs lives on the `nfs-ssd0-39` storage was stopped and unstartable.
This included **FreshRSS (CT 1260)**, Gitea (221), NTFY (1241), LiteLLM (104), openweb (245),
rclone (105), paseo (1261), social-dashboard (1263), homebridge (1007), docker-01 (1006),
vaultwarden (102), omakase (1264), and others. The FreshRSS midday digest cron failed with
`RuntimeError: Auth failed: ... urlopen error timed out` — a misleading error; the cause was
total service unavailability, not credentials.
## Root cause
Both Proxmox hosts rebooted within minutes of each other. The NFS server (`192.168.178.39`,
the PVE host) came back **after** m93p. m93p's boot-time NFS mount therefore failed, and
`pct start` for any CT with an NFS-backed rootfs failed with:
```
volume 'nfs-ssd0-39:1260/vm-1260-disk-0.raw' does not exist
```
There is no boot retry: m93p's mount attempt was one-shot and there is no Proxmox storage entry
for `ssd0-39` (`pvesm status` does not list it — the mount is a plain filesystem mount at
`/mnt/nfs/ssd0-39`).
Secondary fault: **three** NFS mounts fail together on m93p, not one:
| Source | Mount point | Purpose |
|---|---|---|
| `192.168.178.39:/mnt/ssd0` | `/mnt/nfs/ssd0-39` | FreshRSS + most service CT rootfs |
| `192.168.178.31:/Proxmox` | `/mnt/nfs/proxmox-31` | archive/dump storage |
| `192.168.178.54:/Proxmox` | `/mnt/nfs/proxmox-54` | archive/dump storage (same export as .31) |
## Recovery performed
```
mount -t nfs -o vers=4,soft,intr 192.168.178.39:/mnt/ssd0 /mnt/nfs/ssd0-39
mount -t nfs -o vers=4,soft,intr 192.168.178.31:/Proxmox /mnt/nfs/proxmox-31
mount -t nfs -o vers=4,soft,intr 192.168.178.54:/Proxmox /mnt/nfs/proxmox-54
```
then `pct start` for each stopped CT (all with `onboot=1`). All 18 containers are up and their
Tailscale peers are online again.
## Permanent fix
`m93p-nfs-recover.service` on m93p — enabled at `multi-user.target`, ordered after
`pve-guests.service`:
- Script: `/usr/local/sbin/m93p-nfs-recover.sh`
- Retries the three NFS mounts up to 10 times, 20s apart
- Then starts any CT still `stopped` whose config has `onboot: 1`
- Log: `/var/log/m93p-nfs-recover.log`
- Verify: `systemctl status m93p-nfs-recover.service`
## Runbook
If a service on m93p is unreachable and `pct list` shows it `stopped`:
1. `ssh root@100.104.51.98 'mount | grep /mnt/nfs/'` — expect three lines
2. If a line is missing: `systemctl start m93p-nfs-recover.service` (it remounts and restarts the CTs)
3. `pct list` to confirm `running`; then check the service from the Hermes host
4. FreshRSS specifically: `curl -sS -m 15 -o /dev/null -w '%{http_code}' http://100.83.60.93/`
## Lessons
- A Google Reader API `Auth failed ... timed out` error is an availability symptom first, an auth
problem last. Probe the endpoint with curl before touching credentials.
- The digest script needs up to 240s; a 180s timeout produced a spurious exit 124.
- FreshRSS on m93p is **HTTP only** — `https://freshrss.kangaroo-eel.ts.net` fails because no
Tailscale serve/funnel is configured for CT 1260.
@@ -0,0 +1,134 @@
---
title: Mnemosyne Health Monitor auto-paused after gateway-connection failure
type: incident
status: resolved
created: 2026-10-30
updated: 2026-10-31
verified_on: 2026-10-31
confidence: high
tags: [cron, mnemosyne, gateway, incident, delivery]
sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents]
---
# Mnemosyne Health Monitor auto-paused after gateway-connection failure (2026-10-30)
**Status: RESOLVED — 2026-10-30T10:02:48.662+08:00** (updated by the 2026-10-31 audit; everything below is the
2026-10-30 03:3x snapshot and its "job is disabled / notice never delivered" reading did not survive the day).
**What actually happened:** the 00:01:08 slot was **still running**, not dead — it closed **`completed` at
10:02:48** after **10 h 01 m 40 s** and **delivered**; incident `ea79a2_d93c3acdc4f2` **auto-closed `resolved`
at 2026-10-30T10:02:48.662+08:00**, and the job is live again (`enabled: true`, `last_status: ok`,
`failure_streak: 0`). `deliveries.db` finished 2026-10-30 with **8 rows, all `delivered`** (latest 18:46:04) — the
zero-row reading was an artefact of the day having barely begun. ⚠ Its **12:01 slot was then lost to the CT460
host outage** and fired late in the 16:37:21 catch-up burst, still `running` at the 2026-10-31 audit — see
[[incidents/2026-10-30-ct460-host-outage-scheduler-gap]].
**Historical snapshot (2026-10-30 03:3x): ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
## Summary
The Mnemosyne Health Monitor cron job `ea79a2107e36` (`0 */12 * * *`, delivery topic `102`) failed its
2026-10-30 00:01 slot with a **gateway-connection** fault. The run closed with status **`unknown`**,
the scheduler **auto-paused the job**, and the job's own failure notice **failed to send** — so the
only trace of the fault is the incident row in `cron_incidents`.
## What happened
| Field | Value |
|---|---|
| Job | `ea79a2107e36` — Mnemosyne Health Monitor |
| Schedule | `0 */12 * * *` (every 12 h) |
| Started | 2026-10-30 00:01:08.343024+08:00 |
| Finished | 2026-10-30 01:04:22.510091+08:00 (1 h 03 m 14 s) |
| Execution status | **`unknown`** |
| Job `last_status` | **`error`**, `failure_streak: 1` |
| Job `enabled` | **`false`** — auto-paused 2026-10-30 01:04:22 |
| Incident | `ea79a2_d93c3acdc4f2`, `failure_type: unknown`, state **`detected`** |
| Delivery | **failed — never sent** |
Error text on the execution row:
> Cron execution unverified; reconcile
> `/home/hermes/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json`:
> Gateway connection failed; no local fallback
## The two traps in this one fault
### 1. A failure notice that was itself never delivered
`hermes cron list` reports for this job:
> The result was not delivered: live adapter send failed: send_path_degraded;
> live adapter delivery to `telegram:-1004321904721:102` failed
Confirmed at the artefact level: `deliveries.db` contains **zero rows created on 2026-10-30**
(2026-10-29 has 10 rows, all `delivered`). So the failure produced **no Telegram output at all** —
not even a notice about the failure.
This is the same class as [[incidents/2026-08-08-gitea-config-backup-push-dead]]: a job whose green
row hides a broken destination. Here the destination was broken *and* the row went red, but the
**alert path** was the part that was dead. If Anthony had only watched `hermes cron list` for a
non-clean row, he would have seen the job had paused; if he had only watched Telegram, he would have
seen nothing at all.
### 2. `unknown` is not `failed`
The execution closed `unknown`, not `failed`. Per the wiki's own convention
([[systems/scheduled-tasks]]), a `status: unknown` row with **no matching `deliveries.db` row** is a
probable non-delivery that the dashboard hides behind the previous slot's error. The scheduler then
auto-paused the job — so the fleet lost a **monitoring** job, not a reporting job.
## Root cause
**Unverified.** The error names a gateway-connection failure with "no local fallback" during the
reconciliation of an admission record. The admission file referenced in the error is
`~/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json`.
This is **not** the same failure as the 2026-10-26 planned-restart orphan class
([[incidents/2026-10-26-planned-restart-orphaned-cron-run]]), though the error string is similar
("Cron execution unverified; reconcile …"). The 10-26 rows were `alerted` and auto-closed; this row
is still `detected` and the job stayed paused. **No gateway restart was performed on 2026-10-30**
around that window, so the trigger is not established.
## Current state (verified 2026-10-30 03:3x AWST)
- Live `jobs.json`: `ea79a2107e36` → `enabled: false`, `last_status: error`, `failure_streak: 1`,
`paused_reason` = the gateway-connection text above.
- Live `cron_incidents` (table lives inside `executions.db`): `ea79a2_d93c3acdc4f2` is the **only**
new row since 2026-10-29; `closed_at` is **NULL**, `acked_at` is **NULL**, `alerted_at` is **NULL**.
- The other unclosed row is `0e753e_e7f8194a7d06` (Facebook monthly, `failure_type: agent`, `detected`
since 2026-10-01, clears on its 2026-11-01 slot) — **2 unclosed rows in total**.
- Fleet is now **29 jobs — 25 enabled / 4 disabled**; this job is the 4th disabled one.
- The rest of the fleet was healthy in the same window: 2026-10-29 settled **177 `completed` / 0
`failed`**, and 2026-10-30 (to 05:4x) shows 37 `completed`, 1 `running` (this stocktake), 1
`unknown` (this job), 0 `failed`.
## Actions taken
1. Recorded the incident and its two distinct failure modes here.
2. Updated [[systems/scheduled-tasks]] — the job's row, the Disabled Jobs table, the fleet counts
(29/25/4), the non-clean `last_status` count (3 → 4), the "Failing (last run failed outright)"
bullet (0 → 1), and the "Failing (delivery only)" bullet (1 → 2).
3. Updated [[infrastructure/hosts]] for the cron count.
## Actions NOT taken — and why
- **The job was not re-enabled.** Re-enabling a job whose gateway connection failed an hour ago
risks an immediate re-pause, and it would close the incident's purpose. The recovery belongs to
whoever owns the gateway fix.
- **No gateway restart was performed.** This was a stocktake run, and restarting the gateway is a
disruptive action with its own orphan-risk class ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]]).
The fault is recorded, not papered over.
## Next step
Re-enable `ea79a2107e36` **after** the gateway connection is confirmed healthy, then confirm the
run completes and — critically — that a delivery row appears in `deliveries.db`. The point of this
monitor is to report on exception to topic `102`; if the topic path is still degraded, re-enabling it
recreates a silent monitor.
## Related
- [[systems/scheduled-tasks]] — fleet ledger
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — the earlier "execution unverified" class
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — green row, dead destination
- [[systems/mnemosyne]] — the subsystem being monitored
@@ -0,0 +1,102 @@
---
title: Asana Is Unreachable From CT460 on Both Paths — MCP Server Never Authenticated, Hosted Connector Rejects the Call Shape
type: incident
status: active
created: 2026-11-01
updated: 2026-11-01
verified_on: 2026-11-01
review_after: 2026-11-08
confidence: high
tags: [hermes, automation, mcp, connector, credential, silent-failure]
sources: [live-mcp-list, live-config-yaml, live-mcp-tokens, live-logs, live-probe]
---
# Asana Is Unreachable From CT460 on Both Paths — MCP Server Never Authenticated, Hosted Connector Rejects the Call Shape
## Headline
The estate has **two independent Asana integrations on CT460, and neither can complete a read** — which
went unnoticed until an interactive session spent 2026-10-31 evening retrying both to answer a single
question ("where does Remembrance Day stand?").
| Path | Surface | Live state 2026-10-31/11-01 |
|---|---|---|
| **MCP server `asana`** | `https://mcp.asana.com/v2/mcp`, `auth: oauth`, `enabled: true` (`hermes mcp list` → ✓ enabled) | **Never authenticated.** Every attempt logs `MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Run \`hermes mcp login asana\` interactively first to complete initial authorization.` |
| **Hosted connector** | `connectors__asana__GET_A_TASK`, `…__SEARCH_TASKS_IN_WORKSPACE` | Reaches Asana but returns **`422 INVALID_ARGUMENTS` — "Input should be a valid list on parameter \`opt_fields\`"**. 12+ identical retries in one session. |
## Evidence
**MCP server — no token, ever.** `~/.hermes/mcp-tokens/` holds exactly one file:
```
-rw------- 1 hermes hermes 319 Oct 26 01:07 asana.client.json
```
A registered **client** with **no token file** → the OAuth flow was started (client registered
**2026-10-26 01:07**) and never completed. `config.yaml → mcp_servers.asana` reads
`{auth: oauth, enabled: true, url: https://mcp.asana.com/v2/mcp}`. The earliest Asana trace in retained
logs is an **HTTP 401** from `mcp.asana.com/v2/mcp` at **2026-10-27 16:35:05**, followed immediately by
OAuth discovery calls.
**Failure volume** (log lines across `logs/agent.log*` + `logs/errors.log*`; the failure is written to
both files, so line counts roughly double the number of events):
| Day | `MCP OAuth setup failed for 'asana'` lines |
|---|---|
| 2026-10-21 | 455 |
| 2026-10-22 | 469 |
| 2026-10-23 | 576 |
| 2026-10-24 | 576 |
| 2026-10-25 | 576 |
| 2026-10-26 | 301 |
| 2026-10-27 | 439 |
| 2026-10-28 | 1 150 |
| 2026-10-29 | 1 154 |
| 2026-10-30 | 862 |
| **2026-10-31** | **1 156** |
| 2026-11-01 (to 00:16) | 14 |
| **Total retained** | **7 728** |
The earliest retained line is **2026-10-21 05:09:43** (`errors.log.2`) — the gap **predates the retained
log window**; this is a chronic condition, not a 10-31 regression. The 10-31 peak coincides with the
evening Pulse session hammering the path.
**Hosted connector — the call, not the credential.** From `20261031_210329_b35440` (the Conduit Pulse
"Remembrance Day" session), repeated at 22:17:47, 22:20:42, 22:26:31, 22:31:25, 22:36:45, 23:14:45,
23:26:46, 23:28:55, 23:30:34, 23:39:31, 23:41:48, 23:51:19 — each call burning 11–20 s:
```
tools.managed_gateway_identity: connector connectors.execute 422 INVALID_ARGUMENTS
Tool tool_call returned error: {"results":[{"name":"connectors__asana__GET_A_TASK","error":
{"code":"INVALID_ARGUMENTS","message":"Invalid request data provided\n- Input should be a valid list
on parameter `opt_fields`…"}}]}
```
The connector is **authorised** (it is the MCP path that cannot authenticate), so the block is purely
argument shape: `opt_fields` must be a **JSON list**, and a string/comma-form is rejected.
## Impact
- **Non-interactive sessions (cron, background jobs, Conduit background investigations) cannot read
Asana at all** — the MCP path needs an interactive browser login that cron cannot provide.
- **Interactive sessions** get the hosted connector, but the agent re-issued the same mis-shaped call
a dozen times instead of adapting — ~3 minutes of model time spent failing identically.
- Asana-backed user requests (task status behind the Conduit Pulse flow) degrade to guesses.
## Remediation (not performed — this audit records, it does not re-authorise)
1. Run **`hermes mcp login asana`** from an interactive session to complete the OAuth flow; confirm
`~/.hermes/mcp-tokens/asana.json` (or equivalent token file) appears afterwards, then re-probe the
MCP server.
2. **Or** disable the `asana` MCP server if the hosted connector is the intended path — two Asana paths
on one host is itself the source of the confusion, and one of them has never worked.
3. For the connector: pass **`opt_fields` as a JSON list**, not a comma-separated string.
4. Watch for the silent-cost pattern: a path that fails **every** time still burns a tool call each
time. 1 100+ failures/day means the failure is being retried, not fixed.
## Related
- [[systems/hermes-agent]] — MCP server inventory and auth state
- [[systems/conduit]] — the Pulse flow that needed the Asana task status
- [[systems/scheduled-tasks]] — cron fleet (the surface that cannot authenticate interactively)
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the same "green row, dead destination" class
@@ -0,0 +1,103 @@
---
title: "Conduit \"voice mode: local\" never applies — Mac mini speech server's LLM proxy is disabled"
type: incident
status: active
created: 2026-11-01
updated: 2026-11-01
verified_on: 2026-11-01
confidence: high
tags: [conduit, voice, local-live, mac-mini, speech-server, investigation]
sources:
- session 20261031_193709_40b9c2 (Conduit voice session, started 2026-10-31 19:37:09 AWST)
- ~/.hermes/conduit-settings.json, ~/.hermes/conduit-calls.json, ~/.hermes/config.yaml
- live probes of the Mac mini speech server (LLM endpoint, /v1/usage, realtime socket)
---
# Conduit "voice mode: local" never applies
**Status: OPEN — investigation in progress. Nothing was changed on the Mac mini speech server.**
On 2026-10-31 Anthony reported (from the phone): switching the Conduit app's voice
setting to **"local"** shows a **successful** connection test, but the change never
takes effect. A background job launched from that voice conversation
(`20261031_193709_40b9c2`, started **19:37:09 AWST**, 30 messages / 17 tool calls) was
still open at the 2026-11-01 00:2x audit.
## Why the test succeeds and the change still does nothing
The "test connection" button checks **socket reachability only** — it never exercises
the model path. So it passes while every local turn is still impossible:
| # | Finding | Evidence |
|---|---------|----------|
| 1 | The Mac mini speech server's **LLM proxy is disabled** | its LLM endpoint answers **HTTP 501** — *"The LLM proxy is disabled. Start the server with `--enable_llm_proxy` to enable it."* |
| 2 | The Mac has never served a local turn | `/v1/usage` → **`llm_proxy.requests: 0`** across **25** observed turns |
| 3 | The realtime handshake is not understood | the Mac's realtime socket **accepts the connection** but rejects **`session.update`** as an **unknown event** |
| 4 | The app's test button proves nothing about the model path | it returns success on socket reachability alone |
Findings 1–3 are the strongest single explanation: **even with `voice.mode = local`
set on the phone, the Mac cannot call a model back.** Finding 4 is why the UI reports
success.
## `voice.mode` is phone-side, not host-side
The host never reads a `voice.mode` key — there is none to read:
- `~/.hermes/config.yaml → voice` holds **`auto_tts`**, **`local_live`**,
**`speech_server`** — **no `mode`**.
- `~/.hermes/conduit-settings.json` contains **no `voice.*` keys at all**; its only
device state is `devices/<UUID>/mode: small` for three devices.
- The device itself reports **`voice.mode: "classic"`**.
So "set mode to local and the change does not apply" cannot be diagnosed on the host
config — the selector lives in the app, and what the host does with it is a
**`local_live`** path decision.
## A misread that was caught mid-session (worth keeping)
`local_live.py:882` is the **only** writer of `_local_live_last_request`, and it sits
inside the **chat-completions route**. A "recent `last_request_at`" reading during the
investigation therefore indicated **this investigation's own probe**, not phone
traffic — the agent initially read it as phone activity and corrected the call in the
same session. Live check: `conduit-calls.json → last_request_at` is **`null`**.
Lesson: on this integration, `last_request_at` is proof of *some* HTTP request hitting
the local-live route, **not** proof of a phone call.
## Configuration at the time of writing
Changed during the 2026-10-31 session (`config.yaml` mtime **21:02:30**):
- `voice.local_live.front_model` = `{base_url: http://100.78.63.20:20128/v1, model: mistral/voxtral-small-latest}`
— was `mistral/ministral-14b-latest`.
Unchanged: `voice.speech_server.url = wss://…:8443/v1/realtime`, `voice.auto_tts: false`,
`tts.provider: ollama_local`, `stt.provider: local`.
The Conduit dashboard gateway was restarted at **19:35:02** (pid 176771), minutes
before the session opened.
## Prompting context (candidate, not proven)
The plugin **self-updated to `0.122.2` on 2026-10-31 18:05**, which drove a gateway
reload at **18:06:41** → [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]].
A newly shipped local-live path is a reasonable regression candidate, but the
disabled LLM proxy would break local mode on **any** plugin version, so the version
change is not yet implicated.
## Open items
- **Enable the Mac mini's LLM proxy** (`--enable_llm_proxy`) or confirm the intended
Local Live architecture — with the proxy off, finding 1 alone explains the symptom.
- Confirm the **realtime event vocabulary** the plugin expects vs. what the Mac's
socket implements (`session.update` is currently rejected as unknown).
- Decide whether `front_model` should be a **text** or **audio** model — a
chat-completions `base_url` pointed at an audio model may itself be the mismatch.
- Re-test from the phone after any change; the app's own test button is not evidence.
## Related
- [[systems/conduit]]
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
- [[incidents/2026-10-29-conduit-voice-silence]]
- [[incidents/2026-10-22-conduit-memory-start-timeout]]
@@ -0,0 +1,101 @@
---
title: Gateway Reload on 2026-10-31 Force-Drained After Its Full 1 800 s Cap and Orphaned a Live Cron Run — Mnemosyne Health Monitor (6 h 38 m in)
type: incident
status: active
created: 2026-11-01
updated: 2026-11-01
verified_on: 2026-11-01
review_after: 2026-11-08
confidence: high
tags: [gateway, cron, scheduler, lifecycle, restart, memory, silent-failure]
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-probe]
---
# Gateway Reload on 2026-10-31 Force-Drained After Its Full 1 800 s Cap and Orphaned a Live Cron Run — Mnemosyne Health Monitor (6 h 38 m in)
## Headline
On **2026-10-31 at 18:06:41 AWST** the gateway received a **`SIGUSR1` reload** (`systemctl reload` /
`hermes gateway restart`). It deferred its own stop for in-flight work — and this time it **waited the
whole deferral cap**, hit it at **18:37:38** with *"drain timed out with interrupted agents"*, and
force-restarted. The casualty was `ea79a2107e36` **Mnemosyne Health Monitor** (12-hourly `0 */12 * * *`),
whose 12:01:06 slot had been running for **6 h 38 m 09 s**. It closed **`failed`** at 18:39:15 on
**`RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership`**.
This is the **third variant of the orphan class** — after the 2026-10-20 *unclean crash* exit and the
2026-10-26 *planned `SIGTERM` whose drain timed out after 1.35 s* — and the first where the drain
mechanism behaved as designed and still ended in a lost run.
## Timeline (live gateway log, `~/.hermes/logs/agent.log`)
| Time (AWST) | Event |
|---|---|
| 18:06:41.772 | `SIGUSR1 received (systemctl reload / hermes gateway restart): performing a graceful gateway restart — drain active turns, exit, supervisor relaunches.` |
| 18:06:41.788 | `Restart requested with 2 active work unit(s); deferring stop() until they finish (cap=1800s) so in-flight turns are not amputated (#77184)` |
| 18:06:41.899 | Deferred on **1** unit: the cron unit `ea79a2107e36` (`elapsed_s: 21950.7`, `external: True`, `wedged: False`, **`restart_safe: True`** — *excluded* from the wait) plus `{'kind': 'managed', 'pid': 397}` |
| 18:07:11 → 18:11:12 | Countdown logged every 30 s: 1770 s → 1529 s remaining, same single `managed pid 397` unit |
| 18:11:23 | **Second `SIGUSR1`** arrives while the first restart is still deferred |
| 18:20:13 | `Restart deferred: … (988s remaining before force drain): [{'kind': 'managed', 'pid': 397}]` |
| **≈18:36:41** | **Cap reached** (18:06:41 + 1 800 s) |
| 18:37:38.341 | `Skipping .clean_shutdown marker — drain timed out with interrupted agents; next startup will recover their interrupted turns.` |
| 18:38:43.284 | New gateway starts; 18:39:14 reconnection watcher for 1 failed platform (`relay`) |
| 18:39:15.208 | Failure **delivered** (delivery row `9e7da0fa…`, `for_failure=1`, status `delivered`) |
| 18:39:15.563 | Execution `9e7da0fafd2847b69643c6f235cb8651` closes **`failed`** — 6 h 38 m 09 s |
| 18:39:23 / 18:45:23 / 19:00:22 | Three derivative `96eb95ce12f2` (log-error watchdog) reds quote the cron error; the watchdog clears on its 19:15:21 slot |
| 19:23:39 → 19:24:40 | **Second restart, clean**: `drain took 0.00s, timed_out=False`; `Previous gateway exited cleanly — skipping session suspension` |
Current gateway: `hermes-gateway.service` active since **19:23:56**, PID `175269` (4 h 43 m at the
2026-11-01 00:07 probe). Nothing failed for the remaining ~5 h of 10-31 (every `96eb95ce12f2` and
`0c771463c3cd` slot from 19:15:21 onward `completed`).
## Incident-ledger effect
| Row | Job | State | First seen | Last seen |
|---|---|---|---|---|
| `ea79a2_9197df32320b` | `ea79a2107e36` | **`alerted` (open)** | 2026-10-31T18:06:44 | 2026-10-31T18:06:44 |
| `ea79a2_a4661554a3c2` | `ea79a2107e36` | **`alerted` (open, reused row)** | **2026-10-26**T14:35:59 | 2026-10-31T18:39:15 |
| `0e753e_e7f8194a7d06` | `0e753e9720fd` | `detected` (open, unchanged) | 2026-10-01T09:00:55 | 2026-10-01T09:00:55 |
The `error_sig` match re-opened the **2026-10-26 row** rather than filing a fresh one, so two rows now
describe one event: the earlier *"Cron execution unverified; reconcile `admissions/292056e5742c1ba4…`"*
(`failure_type: unknown`) and the definitive *fire-claim* row. **Unclosed incident rows 1 → 3.**
## What is genuinely new here
1. **The deferral cap is reachable in normal operation, not just under a stuck process.** The 10-26
incident drained in **1.35 s**; this one waited the full **1 800 s** because a `managed` unit
(`pid 397`) stayed registered. Force drain is therefore a *routine* outcome of any reload that
overlaps a long-running cron job.
2. **`restart_safe: True` did not protect the execution.** The scheduler explicitly classified the cron
unit as restart-safe and *excluded* it from the wait — and the run still died with a lost claim. The
flag protects the *scheduler's* bookkeeping, not the running agent's lease.
3. **A `SIGUSR1` reload is not a lighter-weight restart.** The pre-restart log shows plugin/update
machinery active minutes earlier (`.update-<id>/plugin` capability checks at 18:05:41 and 18:19:53,
`plugin scan passed` at 18:20:22) — the reload behaved exactly like a full stop/start for the
in-flight job.
## Open questions
- Why is a `managed` (non-cron) work unit allowed to hold the gateway for the entire 30-minute cap?
What is `pid 397`, and should it be exempt from the "wait" set (or bounded separately)?
- The 12:01 slot had already run **6 h 38 m** against **10 h 01 m** for its 10-30 catch-up. Should a
reload deferral weight long cron runs differently — or should the cron lease survive a supervisor
relaunch, as the `restart_safe` flag implies?
- Two `SIGUSR1`s 4 m 42 s apart produced a **single** interruption and then a second clean restart at
19:23 — is the double signal expected?
## Impact
- One job failure + three derivative watchdog reds = **all four** of 2026-10-31's failed executions
(day closed **160 `completed` / 4 `failed`**).
- `ea79a2107e36` now reads `last_status: error`, **`failure_streak: 2`** — the only non-zero streak in
the fleet; non-clean `last_status` **3 → 4**, failing (run failed outright) **0 → 1**.
- Its next slot (2026-11-01 **00:00:41**) was already `running` at audit time.
## Related
- [[systems/scheduled-tasks]] — fleet ledger and the 10-31 window numbers
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — variant 2, the 1.35 s drain timeout
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — variant 1, the unclean crash exit
- [[systems/hermes-agent]] — gateway lifecycle record
- [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — the same job, the day before
@@ -0,0 +1,115 @@
---
title: Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State
type: incident
status: resolved
created: 2026-11-02
updated: 2026-11-02
verified_on: 2026-11-02
review_after: 2026-11-09
confidence: high
tags: [cron, scheduler, worker, sigterm, orphan, cold-start, silent-failure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-logs, live-jobs-json, live-probe]
---
# Three Restart-Safe Cron Workers Killed by SIGTERM (-15) on 2026-11-01 Without Writing a Durable Terminal State
## Headline
On **2026-11-01** three separate `cron.scheduler` **external workers** were terminated with
**`SIGTERM` (`-15`) after they had adopted a durable execution claim but before they wrote a terminal
state**, so each run closed as *"whether side effects ran is unknown"*. The three casualties were
`85edf6772bca` (Tech & AI Digest, 07:30:46), `96eb95ce12f2` (log-error watchdog, 09:02:18) and
`0e753e9720fd` (Facebook scrape, 09:02:18). **This is a fourth variant of the orphan class** — after the
2026-10-20 unclean crash, the 2026-10-26 planned `SIGTERM` whose drain timed out in 1.35 s, and the
**2026-10-31 force-drain at the full 1 800 s cap** ([[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]).
The distinguishing feature here: **no gateway restart, reload or drain occurred anywhere near these
signals.** Gateway PID `175269` had been up since **2026-10-31 19:24:40** and stayed up; the relay
reconnect counter ran continuously through both incidents (attempt 147 at 07:24:58 → 166 at 09:01:34 →
336 at 2026-11-02 00:10), which it could not do across a process restart. The `-15` came from the cron
worker supervision path itself.
## Evidence
Identical `error` text on all three incident rows (`~/.hermes/cron/executions.db` → `cron_incidents`):
```
Restart-safe cron worker exited with status -15 after adopting this execution, without writing a
durable terminal state; whether side effects ran is unknown.;
worker stderr: <frozen runpy>:130: RuntimeWarning: 'cron.scheduler' found in sys.modules after import
of package 'cron', but prior to execution of 'cron.scheduler'; this may result in unpredictable behaviour
```
| Incident id | Job | `first_seen_at` | `closed_at` | Execution outcome |
|---|---|---|---|---|
| `85edf6_94cd563ae0df` | `85edf6772bca` Tech & AI Digest | 2026-11-01 07:30:46 | 2026-11-01 11:01:12 | row **`completed`** / `delivered` (07:01:19 → 11:01:12, **4 h 00 m**) |
| `96eb95_8091fba135e9` | `96eb95ce12f2` log-error watchdog | 2026-11-01 09:02:18 | 2026-11-01 09:16:18 | row **`unknown`** (09:01:03 → 09:02:13) |
| `0e753e_2183c9040706` | `0e753e9720fd` Facebook scrape | 2026-11-01 09:02:18 | 2026-11-01 10:44:12 | row **`completed`** / `delivered` (09:01:03 → 10:44:12) |
All three rows carry `failure_type: unknown`, `acked_at: None`. Only `85edf6` and `0e753e` were
auto-resolved by their own later successful run; `96eb95`'s row stayed `unknown` — **the only
permanently non-clean execution of the whole day**.
## The 09:00 pair — the cold-start window, in full
| Time (AWST) | Event |
|---|---|
| 09:00:49.708 | `cron.scheduler: Cron external worker for job '96eb95ce12f2' did not acknowledge within 30s; leaving the durable execution claim untouched` |
| 09:00:49 → 09:01:02 | The freshly spawned worker boots a full runtime — **`Plugin discovery complete: 83 found, 69 enabled`**, MCP capacity checks |
| 09:01:13.552 | `Running job 'Facebook Birthday Monthly Scrape' (ID: 0e753e9720fd)` — the same worker then picks up a second job |
| 09:02:13 | `96eb95ce12f2` execution row closes **`unknown`** |
| 09:02:18.189 / .211 | Both incident rows written (`alerted` 09:02:29 / 09:02:32) |
| 10:44:12 | `0e753e9720fd` row closes `completed` + `delivered` → incident resolved |
| 09:15:45 → 09:16:19 | Next `96eb95ce12f2` slot runs `completed` → incident resolved |
**Mechanism, as far as the retained logs show it:** the spawn path treats a **30 s acknowledgement
window** as the health signal for an external worker. A cold worker must import the scheduler, discover
**83 plugins / 69 enabled** and register MCP tool schemas before it can acknowledge — a cost that
plainly exceeds 30 s on this host. The supervisor therefore logs the miss and **leaves the durable claim
untouched** rather than reclaiming it; the worker then starts the job anyway and is `SIGTERM`ed ~60–90 s
later, orphan-claiming the run.
**What is NOT established:** *who* sent the `SIGTERM` in these three cases. No `SIGUSR1`, no
`Restart requested`, no `drain`, no `UNCLEANLY`, and no `Previous gateway exited` marker appears in
`gateway.log`, `agent.log` or `agent.log.1` in the 07:2x–09:1x window, and the relay/polling counters
prove no process restart. Do not attribute this to a gateway reload — the signal source is unidentified.
Treat this as an open mechanism question, not a solved one.
## Blast radius and recovery
- **2026-11-01 settled 167 `completed` / 1 `running` / 2 `unknown`** against 161 / 4 on 10-31,
138 / 0 on 10-30 and 177 / 0 on 10-29. The two `unknown` rows are precisely
`96eb95ce12f2` @09:01:03 and `79f9bb41ecf5` @18:00:39 (a handoff failure — see
[[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]).
- **All three executions still reached the user** where they were meant to: the digest closed
`delivered` at 11:01:12 and the Facebook scrape `delivered` at 10:44:12 (delivery rows
`2026-11-01 11:01:12` and `2026-11-01 10:44:12`).
- **No recurrence after 09:02:18** for the remaining ~15 h of 11-01 — every later `96eb95ce12f2`
and `0c771463c3cd` slot (23 and 21 runs respectively) closed `completed`.
- The `-15` class is **not** `[SILENT]`-shaped: each instance raised an incident row and each alerted
(`85edf6` 07:31:14, `96eb95` 09:02:32, `0e753e` 09:02:29). Only the `unknown` execution status is
invisible to the dashboard's status field.
## Why it matters
A `-15`-after-adoption means the scheduler cannot say whether the job's side effects ran. For a job with
external writes (Gitea pushes, API POSTs, email sends) that is a **possible double-execution** on the
next slot. The current design resolved this properly for two of three, but the class keeps recurring in
a new form roughly every 10 days — 10-20, 10-26, 10-31, 11-01. **The recurring element across all four
variants is the same: a live cron worker is signalled mid-adoption and the durable claim is left open.**
## Open questions
1. Which supervisor sends the `-15` after a 30 s ack miss — and is the kill a timeout guard or an
unrelated cleanup path?
2. Does the ack window scale with plugin count? 83 plugins discovered in a cold worker is the visible
cost driver; if the window is fixed at 30 s, cold-start jobs on this host are structurally exposed.
3. Should a job with `failure_type: unknown` external side effects be blocked from its next slot until
`~/.hermes/cron/admissions/*.json` is reconciled? Two admission files from 11-01 are still
unreconciled (`141448f9…`, `69b5588f…`).
## Related
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]
- [[systems/scheduled-tasks]]
- [[systems/hermes-agent]]
@@ -0,0 +1,144 @@
---
title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Recovered and Delivered 7 h 26 m Later
type: incident
status: resolved
created: 2026-11-02
updated: 2026-11-03
verified_on: 2026-11-03
review_after: 2026-11-17
confidence: high
tags: [cron, gateway, handoff, delivery, weekly-review, silent-failure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-jobs-json, live-logs, live-probe]
---
# Weekly Review Lost at 18:00 on 2026-11-01 — Recovered and Delivered 7 h 26 m Later
> **Status: RESOLVED 2026-11-02T01:27:21+08:00.** The Sunday 18:00 run did *not* lose the review — it
> resumed, completed and delivered at **01:27:21**, and the incident row auto-closed. The body below is
> the record **as it stood at the 2026-11-02 audit**, before the recovery landed. See
> [Resolution](#resolution-2026-11-02) for the corrected outcome and the answers to the open questions.
## Headline
The **Sunday 18:00 Weekly Review (`79f9bb41ecf5`, `0 18 * * 0`) did not run and nothing was delivered on
2026-11-01.** At **18:05:13** its execution closed with `status: unknown`, `delivery_outcome: NULL` and
the error *"Cron execution unverified; reconcile `…/cron/admissions/141448f9…json`: **Gateway connection
failed; no local fallback**"*. Its incident row **`79f9bb_9c443b602eec` is still `alerted` at the time of
this audit** (`first_seen` 18:05:13, `alerted_at` 18:05:32, `closed_at: NULL`) and the job reads
`last_status: error`, **`failure_streak: 1`**. There is **no `deliveries.db` row for any slot after
11:01:12 on 11-01** — the last delivery of the day was the Tech & AI Digest at 11:01:12.
## Timeline (AWST, 2026-11-01)
| Time | Event |
|---|---|
| 18:00:39 | **Four jobs handed to external workers in one batch**: `2fd50c87b0d0`, `96eb95ce12f2`, `b63ed5817212`, `79f9bb41ecf5` |
| 18:00:57 / 18:01:04 | `96eb95ce12f2` and `2fd50c87b0d0` run fine (`completed`) |
| **18:05:13.697679** | `b63ed5_6f45aeea97f0` opens — *"Cron execution unverified; reconcile `…/admissions/69b5588f…json`: Gateway connection failed; no local fallback"* (`alerted_at: None` — **never alerted**) |
| **18:05:13.708153** | `79f9bb_9c443b602eec` opens with the **identical** message, pointing at `…/admissions/141448f9…json` |
| 18:05:32 | `79f9bb` incident **alerted** |
| 18:06:21 | `b63ed5817212` execution closes `completed` (18:00:39 → 18:06:21) → its incident auto-resolves **68 s later** |
| — | `79f9bb41ecf5` closes at 18:05:13 with **no terminal state, no delivery, no recovery** |
| 18:04:24 → 18:10 | `agent.log`: `[reaper] Active gateway work detected … deferring restart`, then *"Reaper: Plan A restart completed successfully (5m02s)"* — the **Plan A/B path is RSS compaction, not a process restart** (no `SIGTERM`/`SIGUSR1` and no relay-counter reset) |
## What actually reached the user
**Nothing.** Across the whole of 11-01 `deliveries.db` holds 8 rows:
| Time | Status | Note |
|---|---|---|
| 02:39:33 | delivered | Mnemosyne-era failure notice (closes the 10-26 and 10-31 rows) |
| 03:00:37 | delivered | stocktake |
| **07:18:50** | **failed** | `configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id` — job `066bb5191280` (see [[systems/messaging-integrations]]) |
| 07:34:49 | delivered | — |
| 08:36:51 | delivered | Gmail digest |
| 10:44:12 | delivered | Facebook scrape |
| 11:01:12 | delivered | Tech & AI Digest |
| — | — | **no row after 11:01:12** |
So of the day's eight delivery attempts, **seven reached their target and one (`066bb5191280`) landed in
the DM thread-less while being recorded `failed`**. The Weekly Review's absence is not a suppressed
delivery — it is an absent run.
## Second silent gap: a job that reported `error` with no execution row at all
`ecc1fc13655a` (FreshRSS Midday News Briefing) reads **`last_status: error`, `failure_streak: 1`,
`last_run_at: 2026-11-01T12:01:00.374089+08:00`** — yet **no `executions` row exists for it on 11-01**
(the last is 2026-10-31T12:01:07, `completed`/`delivered`) and **no `cron_incidents` row was raised.**
That 12:00 slot overlapped the same `18:00:39`-style handoff batch pattern (three jobs claimed at
12:00:53). This is the **second job of the day whose failure left no durable record**, and the third
appearance of the "green/red row, no artefact" anti-pattern the wiki already documents
([[systems/scheduled-tasks]], §9 of the maintenance playbook).
## Fleet state at this audit
- **Fleet 29 jobs — 26 enabled / 3 disabled** (`jobs.json`), unchanged from 10-31.
- **Unclosed incident rows: 1 → 1, but a different row.** The 10-31 audit left
`ea79a2_a4661554a3c2` / `ea79a2_9197df32320b` / `0e753e_e7f8194a7d06` open; **all three closed on
11-01** (02:39:33 for the two `ea79a2` rows, **10:44:12** for `0e753e_e7f8194a7d06` — the
Facebook-repin row that had been open since **2026-10-01 09:00:55**, i.e. 31 days). The new sole
open row is `79f9bb_9c443b602eec`.
- **Non-clean `last_status` 4 → 2** (`ea79a2107e36`, `79f9bb41ecf5`); **failing outright 1 → 1**
(`79f9bb41ecf5`).
- **2026-11-01 settled 167 `completed` / 1 `running` / 2 `unknown`.**
## Open questions
> ⚠ **Questions 1–3 are answered in [Resolution](#resolution-2026-11-02).** Question 4 remains open.
1. **What did "Gateway connection failed; no local fallback" mean?** Same message for two different
jobs 11 ms apart, both in the same handoff batch, with no gateway restart anywhere in the window.
2. **Why did `b63ed5` self-heal in 68 s and `79f9bb` never do so?** Both were in the same batch with the
same error string; one closed `completed`, the other never wrote a terminal state.
3. **Should `79f9bb41ecf5` be manually re-run for 2026-11-01?** Anthony did not receive his weekly
review.
4. Are the two unreconciled admission files (`141448f9…`, `69b5588f…`) the durable record needed to
decide whether either run had side effects? Reconcile before the next 18:00 Sunday slot.
## Resolution (2026-11-02)
**The run was not lost. It resumed, finished, and delivered — 7 h 26 m 43 s after it started.**
| Field | Value |
|---|---|
| Execution | `be6ec11a8e68428084af9ab18fb148db` — **the same execution record** the audit saw stuck at `unknown` |
| `scheduled_instant` | `2026-11-01T10:00:00+00:00` (18:00 AWST) |
| `started_at` | `2026-11-01T18:00:39.125776+08:00` |
| `finished_at` | **`2026-11-02T01:27:22.000881+08:00`** |
| `status` | **`completed`** |
| `error` | `NULL` |
| `delivery_outcome` | **`delivered`** |
| Delivery row | `2026-11-02T01:27:21.802246+08:00` — `delivered` |
| Incident row | `79f9bb_9c443b602eec` → `state: resolved`, `closed_at: 2026-11-02T01:27:21.606691+08:00` |
Anthony received the Weekly Review at 01:27 AWST on 2026-11-02. There is still exactly **one** execution
row for the 18:00:39 slot, so this is a reconciliation of the existing claim rather than a re-run — the
`unknown` terminal state at 18:05:13 was replaced when the run finally wrote its real outcome.
**Answers to the open questions:**
1. *(the "no local fallback" message)* — the handoff to the external worker did ultimately succeed; the
18:05:13 message recorded a **failed reconciliation attempt**, not a failed run. The run kept executing
for a further 7 h 26 m and produced its deliverable. The message is therefore a **false negative from
the reconciler**, not evidence that the work was abandoned.
2. **`b63ed5` closed in 68 s; `79f9bb` closed in 7 h 26 m — both eventually reached `completed`/`delivered`.**
The difference was run duration, not failure mode. The reconciler's window is shorter than the Weekly
Review's runtime, so any long-running job in a handoff batch will be reported `unknown` while it is
still working. **This is the general lesson: an `unknown` execution state on a long job is a timing
artefact, not a verdict.**
3. **No manual re-run was needed** — the run self-healed and delivered 12 minutes before the 2026-11-02
audit would have recommended one. Had the audit re-triggered the job, it would have duplicated the run.
### Consequences for the audit method
The 2026-11-02 audit read `status: unknown` + `closed_at: NULL` at 00:4x and concluded the deliverable was
lost. Both were true at the time of reading and both had inverted by 01:27. **A nightly audit that runs
~2 h after midnight will read every long-running job mid-flight and must not treat a null terminal state as
final until the job's expected runtime has elapsed** — see [[systems/scheduled-tasks]] §9.
## Related
- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]]
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
- [[systems/scheduled-tasks]]
- [[systems/messaging-integrations]]
@@ -0,0 +1,147 @@
---
title: Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete Run Recorded `completed`, Output Suppressed
type: incident
status: active
created: 2026-11-03
updated: 2026-11-04
verified_on: 2026-11-04
review_after: 2026-11-17
confidence: high
tags: [cron, job-radar, iteration-cap, silent-failure, notion, camofox, stream-timeout]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-state-db, live-jobs-json, session-history]
---
# Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete Run Recorded `completed`, Output Suppressed
## Headline
**Third occurrence of the Stage-1 budget class, and the first that the cron ledger reports as healthy.**
On **2026-11-02** the 07:00 `dca8482e4f76` (*Daily Marketing Job Radar — Stage 1 (Discovery)*) ran for
**9 h 01 m 19 s** (`started 07:00:33.378239` → `finished 16:01:52.496335 +08:00`), consumed the harness's
entire tool-calling iteration budget, and closed with:
| Field | Value |
|---|---|
| `status` | **`completed`** |
| `error` | `NULL` |
| `delivery_outcome` | **`suppressed`** |
No handoff artefact was written. **Nothing reached Notion, no cover letters were generated, and Stage 2 was
never triggered.** Because the run's own status is `completed` with no error, a health check that inspects
`executions.status` alone reports this day as green — the failure is invisible in the ledger.
## What the run did and did not do
The session (`Daily Marketing Job Radar - Discovery · Nov 02 16:01`, `local-368c8fe9…`) shows the discovery
half genuinely worked:
- Camofox was restarted at run start after its HTTP server wedged (see [[systems/browser-backend]]).
- **SEEK 51 unique roles, LinkedIn 60 unique job IDs, Indeed live, WA Government 35 results** — 197 raw
candidates, **164 new after three-level dedup** against the 479-row Notion board.
- A **URL-canonicalisation bug was found and fixed mid-run**: `au.seek.com` vs `au.seek.com.au` had been
dropping **SEEK and Indeed entries** from the consolidated list; the fix was a job-ID-based
canonicalisation.
- Adverts were individually verified fresh and active; salaries, closing dates and JD bodies were pulled.
Then the ceiling hit. The runtime injected *"You've reached the maximum number of tool-calling iterations
allowed. Please provide a final response…"* while the agent was still reading JD text for the borderline
finalists, and the run terminated in the **scoring** phase.
The agent's own closing report was explicit — and it was **not delivered**:
> *"**Status: INCOMPLETE — handoff file NOT written, Stage 2 NOT triggered.** … Because the mandatory
> deliverable (the handoff JSON) was not produced, today's roles have **not** been handed to Stage 2."*
That text is `delivery_outcome: suppressed`, so Anthony saw nothing on 11-02 — not the roles and not the
warning that the roles were missing.
## Downstream: both Stage 2 slots behaved correctly
| Slot | Session | Ran | Result |
|---|---|---|---|
| 11:30 | `local-92665337…` (*Write and Letters*) | 11:30 → 12:25 | Exited `[SILENT]` after verifying Stage 1 was alive but had no handoff |
| 14:30 (safety net) | `local-05a3f00b…` (*Write and Letters*) | 14:30 → 14:30 | Exited `[SILENT]` — 6 messages, 2 tool calls |
Both declines were correct: no handoff existed to consume. The safety-net design worked as intended — the
gap is that Stage 1 cannot finish inside its budget.
## Recurrence pattern
| Date | Incident page | Failure shape |
|---|---|---|
| 2026-10-27 | [[incidents/2026-10-27-job-radar-budget-exhausted]] | Tool-call budget exhausted before the Notion write stage |
| 2026-10-28 | [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] | Stage 1 hit the agent iteration cap before the handoff |
| **2026-11-02** | **this page** | Same cap; **run recorded `completed` and its report suppressed** |
| 2026-11-03 | [[incidents/2026-11-03-job-radar-stream-active-timeout]] | **A different ceiling** — `RuntimeError: stream active timeout` after 6 h 28 m; `failed`, notice delivered |
Three hits in seven days, each ending in the same place: discovery complete, handoff absent, Notion write
stage never reached. **A fourth run — 2026-11-03 — ended the same way (no handoff, both Stage 2 slots declined) but on a
different fault, and this time the ledger did go red; see the 2026-11-04 update below.**
## The "green row, no artefact" anti-pattern
This is the **fourth appearance** of the class [[systems/scheduled-tasks]] §9 of the maintenance playbook
already documents (a job whose durable ledger entry says success while the thing it was supposed to produce
does not exist). The new variant here is that the ledger flipped from red to green:
| Date | `cron_incidents` row for `dca8482e4f76` | `executions.status` |
|---|---|---|
| 2026-10-27 | `dca848_e93a538ade6b` (`failure_type: auth`) | non-clean |
| 2026-10-28 | `dca848_0a2f03b5952b` (`failure_type: timeout`) | non-clean |
| **2026-11-02** | **none raised** | **`completed`, `error: NULL`** |
The two earlier caps each raised an incident whose `error` text is literally the run's INCOMPLETE report.
The 2026-11-02 cap raised **nothing at all** (`cron_incidents` holds no row for `dca8482e4f76` with a
`first_seen_at` on 2026-11-02; fleet-wide unclosed rows were **0** at the 2026-11-03 audit). The run's
`delivery_outcome` is `suppressed`, so no message reached Anthony either.
**Detection rule:** for any job whose deliverable is a *file* (a handoff JSON, a report, a generated page),
`executions.status` is not sufficient evidence. Verify the artefact exists at its expected path, or that
`cron_incidents` stayed quiet for a reason, before treating the slot as served. Silence from
`cron_incidents` is not the same as success — here it is the *symptom*.
## 2026-11-04 update — the missing incident row was a property of the cap, not of long runs
**Job Radar Stage 1 lost a second consecutive day on 2026-11-03 — and this time the harness killed the
turn, so the ledger went red.** The 07:00 `dca8482e4f76` ran **07:00:33.819190 → 13:28:52.305995
(6 h 28 m 18 s)** and closed **`status: failed` / `error: RuntimeError: stream active timeout` /
`delivery_outcome: delivered`**, raising incident **`dca848_dfd4bea84f2a`** (`failure_type: timeout`,
**`alerted` 13:28:55.141618, still open**) — fleet unclosed rows **0 → 1**.
→ [[incidents/2026-11-03-job-radar-stream-active-timeout]]
**This answers open question 3 below (partly): suppression tracked the terminal state, not the run length.**
When the run ends `completed` — as the cap produces — its report is suppressed; when the harness itself
kills the turn, the terminal state is `failed` and both the incident row and the notice appear. 11-02:
nothing delivered. 11-03: the failure notice went out at **13:28:51.741724** and Pulse surfaced it to
Anthony in the desktop session `20261031_190813_95fbde` (report `p34c8e899-1`, covering Tue 03 Nov 13:01
onward): *"The Daily Marketing Job Radar — Discovery job is now failing, with its last run dying on a
stream active timeout."*
**It also narrows open question 2 below:** the ledger *can* record a non-`completed` state when a long run
dies — the iteration cap is the specific case that still escapes as `completed`.
**And it partially answers open question 4: the candidate set was lost again.** Both Stage 2 slots declined
silently (`5fc292ff0f3b` 11:30:36 → 11:32:16 and 14:30:32 → 14:33:11) and
`~/.hermes/radar-ref/handoff/` still holds **only** `2026-10-27.json`, `2026-10-29.json` and
`2026-10-30.json` — so **2026-11-02's 164 deduped candidates and 2026-11-03's discovery are both
unrecovered**, two days running.
## Open questions
1. **Is the iteration budget the right control, or is the task too large for one agent turn?** Three caps in
a row suggest Stage 1's work (four job boards → dedup → individual advert verification → JD reading →
scoring) does not fit a single turn at any budget.
2. **Should the harness record a cap-hit as something other than `completed`?** A distinct terminal status
(or a non-null `error`) would make this visible to every existing health check.
3. **Should the run's final response be delivered even when the job "succeeded"?** Suppressing an INCOMPLETE
report converts a loud failure into silence.
4. **Did the 11-02 roles get picked up on the next day's run**, or is 2026-11-02's candidate set lost?
The 164 deduped candidates were never persisted.
## Related
- [[incidents/2026-11-03-job-radar-stream-active-timeout]]
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]
- [[incidents/2026-10-27-job-radar-budget-exhausted]]
- [[systems/scheduled-tasks]]
- [[systems/browser-backend]]
@@ -0,0 +1,130 @@
---
title: Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff
type: incident
status: resolved
created: 2026-11-04
updated: 2026-11-06
verified_on: 2026-11-06
review_after: 2026-11-11
confidence: high
tags: [cron, job-radar, stream-timeout, silent-failure, notion, context-compaction]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-deliveries-db, live-jobs-json, live-state-db, live-gateway-log, live-radar-handoff-dir, session-history]
---
# Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff
## Headline
**The Stage-1 slot lost a second consecutive day — and this time the harness killed the turn, so the
ledger went red.** On **2026-11-03** the 07:00 `dca8482e4f76` (*Daily Marketing Job Radar — Stage 1
(Discovery)*) ran **6 h 28 m 18 s** (`started 07:00:33.819190` → `finished 13:28:52.305995 +08:00`) and
closed:
| Field | Value |
|---|---|
| `status` | **`failed`** |
| `error` | **`RuntimeError: stream active timeout`** |
| `delivery_outcome` | **`delivered`** (the failure notice, `for_failure: 1`) |
| `cron_incidents` row | **`dca848_dfd4bea84f2a`**, `failure_type: timeout`, **`alerted`**, still open |
No handoff artefact was written, so **nothing reached Notion, no cover letters were generated, and both
Stage 2 slots declined.** This is the **fourth Stage-1 failure in eight days** and a **new terminal
signature** — not the iteration cap that ended 10-27, 10-28 and 11-02.
## Timeline (all `+08:00`)
| Time | Event |
|---|---|
| 07:00:33.819190 | execution `dca8482e4f76` opens (07:00 weekday slot) |
| 07:01:17 | run session `local-7180cef7…` first message |
| 08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21 | **five context compactions** in one run |
| 11:30:36.617545 → 11:32:16.812897 | Stage 2 `5fc292ff0f3b` first slot: no handoff → `[SILENT]`, `suppressed` |
| 12:23:28 | **last tool result of the run** (a `terminal` call) |
| 13:28:49 | harness records *"This turn did not complete. Some actions may already have run; verify their effects before resending."* |
| 13:28:50,627 | gateway.log: `ERROR gateway.session_authority: Admitted turn 9bf9a296… failed` (traceback through `session_authority.py:1159 _execute_claimed` → `session_finite.py:80 execute_finite_admission` → `session_ingress.py:235 execute_admission`) |
| 13:28:51.463929 | incident `dca848_dfd4bea84f2a` opens |
| 13:28:51.741724 | **failure notice delivered** |
| 13:28:52.305995 | execution closes `failed` |
| 13:28:55.141618 | incident `alerted` |
| 14:30:32.261931 → 14:33:11.470879 | Stage 2 safety-net slot: same silent decline |
| in-session | Pulse report `p34c8e899-1` (covering Tue 03 Nov 13:01 onward) led with it: *"The Daily Marketing Job Radar — Discovery job is now failing, with its last run dying on a stream active timeout."* |
## The 65-minute silence is the new information
The run's recorded activity stops at **12:23:28** and the harness does not declare the turn dead until
**13:28:49** — **65 m 21 s with no tool call, no model turn and no error written to the ledger**.
Contrast with the two cap failures: on **10-28** and **11-02** the runtime *injected* the text *"You've
reached the maximum number of tool-calling iterations allowed…"* into the session before it ended. This
session contains **no such message** — it stopped mid-verification, on a provider-stream liveness timeout,
with the run's own closing plan (*"Two final verification passes, then I'll assemble the handoff"*) never
executed.
**Therefore:** the agent iteration cap is not the only ceiling that ends a long Stage-1 run, and the
"2026-11-02 raised no incident at all" behaviour was a property of *that* terminal state (`completed`,
`error: NULL`), **not** of long runs in general. Nothing in the ledger or the gateway log records the
provider's socket state during the silent window.
## Why the ledger went red this time
| Date | `executions.status` | `cron_incidents` row | Notice to Anthony |
|---|---|---|---|
| 2026-10-27 | non-clean (`failed`) | `dca848_e93a538ade6b` — `failure_type: auth` (**wrong** for a budget fault) | yes — the run's INCOMPLETE report |
| 2026-10-28 | non-clean (`failed`) | `dca848_0a2f03b5952b` — `failure_type: timeout` | yes |
| 2026-11-02 | **`completed`, `error: NULL`** | **none raised** | **no — `suppressed`** |
| **2026-11-03** | **`failed`** | **`dca848_dfd4bea84f2a` — `timeout`, open** | **yes — 13:28:51.741724** |
Fleet-wide **unclosed incident rows 0 → 1**. A run the harness itself kills produces an honest red row and
a delivered notice; the cap produces a green row and silence. That is the distinction
[[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] could only predict, now observed.
## Downstream: both Stage 2 slots behaved correctly, and two days of candidates are lost
| Slot | Job | Ran | Result |
|---|---|---|---|
| 11:30 | `5fc292ff0f3b` (*Write and Letters*) | 11:30:36 → 11:32:16 | `[SILENT]` / `suppressed` |
| 14:30 (safety net) | `5fc292ff0f3b` | 14:30:32 → 14:33:11 | `[SILENT]` / `suppressed` |
`~/.hermes/radar-ref/handoff/` holds **only** `2026-10-27.json`, `2026-10-29.json` and `2026-10-30.json` —
nothing for 10-31, **11-02** or **11-03**. So **2026-11-02's 164 deduped candidates and 2026-11-03's
discovery are both unrecovered**, and Notion received nothing on either day. The last end-to-end success
remains **2026-10-29** (handoff written inside budget; Stage 2 landed **20 pages + 2 cover letters**,
`missing_cover_letter_count 0`).
## Context churn (observation, cause not established)
**363 messages** (6 `user` / 133 `assistant` / 224 `tool`), **288 compaction-flagged messages**, and **five
compactions** inside one 6 h run. The 11-02 run compacted repeatedly as well. Whether compaction churn
contributes to the stream stall is **not established** — this is recorded because both long failures share
the pattern, not because the link is proven.
## Open questions
1. Is the **65 m 21 s** silence the stream-active watchdog's own threshold, or did the provider socket die
silently long before the harness noticed?
2. Does repeated context compaction contribute to the stall? Two long runs, both compacting heavily.
3. The 07:00 run has now overrun past the 11:30 Stage-2 slot two days running — should the pairing be
re-slotted, or Stage 2 made to wait on a handoff?
4. Where does `radar-ref/handoff/` get its retention? The newest file is 10-30.
5. Should a stream timeout on a *file-deliverable* job trigger an automatic retry instead of leaving the
day's discovery to the next slot?
## Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST)
- The last open row raised here, `dca848_dfd4bea84f2a` (`timeout`, opened **2026-11-03 13:28:51**), **closed on
2026-11-05 at 12:36:17** - together with its 11-04 successor row `dca848_3b1bb1e9b003`.
- The job behind it, `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), **completed** its 2026-11-05
07:00 run in **5 h 35 m 35 s** (07:00:42 -> 12:36:18), with no repeat of
`RuntimeError: Upstream stream failed before completion.`
- **Status -> `resolved`.** The underlying questions (whether the 65 m 21 s stream silence is the watchdog
threshold or a silently dead socket; whether a stream timeout on a file-deliverable job should retry
automatically) remain open in [[systems/scheduled-tasks]] - they are no longer tracked as an incident
against this window.
## Related
- [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]
- [[incidents/2026-10-27-job-radar-budget-exhausted]]
- [[runbooks/job-radar]]
- [[systems/scheduled-tasks]]
@@ -0,0 +1,132 @@
---
title: "Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)"
type: incident
status: resolved
created: 2026-11-05
updated: 2026-11-06
verified_on: 2026-11-06
confidence: high
tags: [cron, job-radar, newsletter-digest, omni-101, smart-route, stream-failure, incident]
sources: [live-executions-db, live-cron-incidents, live-deliveries-db, live-cron-output, live-jobs-json]
---
# Two 07:00 Jobs Died on an Upstream Stream Failure - and Stage 2 Salvaged the Day (2026-11-04)
**Window:** 2026-11-04 07:00:45 to 09:42:20 AWST (2 h 41 m 35 s) - **Status:** both incidents still open
(`state = alerted`); no data lost - the Job Radar's day was recovered by its Stage 2 safety-net slot.
## Summary
Two independent 07:00 jobs died on the **same upstream fault, 36 seconds apart** on 2026-11-04:
- `dca8482e4f76` **Daily Marketing Job Radar - Discovery** (`0 7 * * 1-5`)
- `85edf6772bca` **Tech & AI Newsletter Digest (Analytical)** (`0 7 * * *`)
Both ran **2 h 41 m** and ended with the identical error, `RuntimeError: Upstream stream failed before
completion.` Both are pinned to `custom:omni-101` / `smart-route`. Neither produced output. Two
`cron_incidents` rows were raised and alerted, and both notices were delivered.
Unlike the 2026-11-02 and 2026-11-03 incidents - where an aborted Stage 1 cost the day's candidate set -
**the Job Radar's day survived**: the 11:30 Stage 2 slot reconstructed the missing handoff, re-verified every
candidate live, and still wrote 5 roles and 1 cover letter. See *The recovery* below.
## Evidence
Live `executions.db`, `cron_incidents`, `deliveries.db` and `jobs.json`, read 2026-11-05 00:5x AWST:
| Job | Start | End | Duration | Status | Incident row |
|---|---|---|---|---|---|
| `dca8482e4f76` Daily Marketing Job Radar - Discovery | 07:00:45 | 09:42:20 | 2 h 41 m 35 s | `error` | `dca848_3b1bb1e9b003` (`unknown`, open) |
| `85edf6772bca` Tech & AI Newsletter Digest (Analytical) | 07:00:45 | 09:41:44 | 2 h 40 m 59 s | `error` | `85edf6_c52a11a62044` (`unknown`, open) |
Both error strings: `RuntimeError: Upstream stream failed before completion.`
All three open incident rows carry `state = alerted` with `closed_at` NULL - the 2026-11-03 `timeout` row
`dca848_dfd4bea84f2a` (first seen `2026-11-03T13:28:51`) plus these two (`09:41:43` and `09:42:19`).
**Unclosed incident rows 1 -> 3.**
Deliveries on 11-04: **8 rows, 8 `delivered`** - including both failure notices (09:41:43, 09:42:20). The
alerts were not silent.
## Why this is not simply the 2026-11-03 incident again
| | 2026-11-03 | 2026-11-04 |
|---|---|---|
| Error | `RuntimeError: stream active timeout` | `RuntimeError: Upstream stream failed before completion.` |
| Duration | 6 h 28 m 18 s | 2 h 41 m (both jobs) |
| Jobs hit | 1 (`dca8482e4f76`) | 2, in the same minute |
| Row | `dca848_dfd4bea84f2a` | `85edf6_c52a11a62044`, `dca848_3b1bb1e9b003` |
A timeout is a stall; "stream failed before completion" is an upstream drop. Both land in the stream layer
of the `smart-route` combo, on `custom:omni-101`. Two unrelated prompts - job discovery and a newsletter
digest - failing within 36 seconds of each other points at the route/provider, not at either job.
## The recovery
Stage 2 (`5fc292ff0f3b`) ran in its 11:30 safety-net slot and finished **15:58-16:09 AWST: Result PASS,
0 missing cover letters**:
- Stage 1 wrote **no handoff**. Stage 2 **reconstructed it from Stage 1's own candidate files** and
re-verified every surviving candidate **live in Camofox** before writing anything.
- **5 roles written to Notion** (board **479 -> 484**): Colab Resources (Group Marketing Manager, strong
match), Foundation for the WA Museum, Mindful Meditation Australia, HPH Solutions, RecruitShop.
- Discovery totals carried forward: **34 discovered / 10 duplicates avoided / 20 rejected**.
- **1 strong match -> 1 canonical cover letter** (Colab Resources) rendered through the canonical teal
template: `.txt`/`.json`/`.html`/`.docx`/`.pdf` all present and non-empty, no `{{}}` tokens, HTTP 200 on
`:9099/jobs/drafts/...`. `SAME_DAY_STRONG_COUNT 1` / `VERIFIED_COVER_LETTER_COUNT 1` /
`MISSING_COVER_LETTER_COUNT 0`.
- Handoff marked `consumed: true` at **16:08:59** (atomic write), so the 14:30 slot stayed silent; delivery
at 16:12:22.
- The run also caught **its own** drafting error: the first Colab letter misstated three resume facts (an
invented "40% sales increase", Horizon Power programmes attributed to Pacific Energy, a stray en-dash).
It was corrected against `current_resume_from_website.txt` and re-rendered - a live demonstration of the
runbook's "verify against the canonical resume" rule.
**So the 11-04 failure cost a Stage 1 run, not the day** - the first time the split pipeline's safety-net
slot has had to carry the day alone.
## The cover-letter route is only alive by override
`regenerate_cover_letter.py` line 82 reads `os.environ.get("CL_MODEL", "deepseek-paid-smart")`. During the
11-04 run that default returned **HTTP 402 (billing)**, so generation was pinned to
**`gemini/gemini-3.5-flash`** for that process. `grep` finds **no `CL_MODEL` in `~/.hermes/.env`** - the
override never persisted, so the next run inherits the default again. The same model name still serves other
cron jobs (`cbfe434a3b5e`, this nightly audit, `last_status: ok`), so treat a 402 as **route/time-specific
until re-tested** rather than as proof the route is dead. See [[runbooks/job-radar]].
## Other in-window cron faults
- `1a8e90ea2271` **wiki-stocktake-maintenance** ran **03:00:31 -> 08:42:17 (5 h 41 m 46 s)**, hit the turn
ceiling, reported incomplete, delivery `suppressed` - the **third consecutive overrun** of this job. Its
pin is `custom:omni-101` / **`free-stack`**. It overlapped the 07:00 window Stage 1 was running in.
- `066bb5191280` **Uber Eats Monthly Spend Report** carries `last_status: delivery_failed` (streak 0) - part
of the non-clean count, not a new failure.
## Open questions / next actions
1. Both 11-04 failures share a signature and a route with the 11-03 timeout. **No output file exists for
either** - the fault lands upstream of the agent's first turn.
2. `failure_type` is stored as **`unknown`** for a stream fault, so neither new row groups with the 11-03
`timeout` row. The incident table undercounts what looks like one route-level pattern (same miscoding
seen on the 2026-10-27 row, stored as `auth` for a 400).
3. All three rows are `alerted` and unclosed. Decide whether the two 11-04 rows are two incidents or one
route incident - and whether the 11-03 row should be closed.
4. The stocktake's third overrun and its `free-stack` pin need a decision - see [[systems/scheduled-tasks]].
## Resolution (2026-11-06 nightly audit; live read 2026-11-06 00:3x AWST)
- **Both affected jobs ran clean on 2026-11-05.** `85edf6772bca` (*Tech & AI Newsletter Digest (Analytical)*)
completed its 07:00 run in **7 m 01 s** (07:00:42 -> 07:07:43), its incident row closing at **07:07:42**;
`dca8482e4f76` (*Daily Marketing Job Radar - Discovery*) completed in **5 h 35 m 35 s** and its two rows
closed at **12:36:17**. **Fleet-wide unclosed incident rows: 3 -> 0** - the ledger was empty for the first
time since 2026-11-02, and 2026-11-05 was the first zero-`failed` ledger day since then too.
- The Stage 2 salvage documented above therefore stands as the whole recovery - no follow-up failure arrived.
- The `CL_MODEL` / HTTP 402 cover-letter caveat is **unchanged and still needs a decision**: the
`gemini/gemini-3.5-flash` pin was never written to `.env`, so it did not persist.
- **Status -> `resolved`.** A recurrence should be opened as a new incident with its own window.
## Related
- [[incidents/2026-11-03-job-radar-stream-active-timeout]]
- [[systems/scheduled-tasks]]
- [[runbooks/job-radar]]
@@ -0,0 +1,121 @@
---
title: vzdump Freeze Took CT299 / SFTPGo Offline for ~24h
type: incident
status: resolved
created: 2026-11-04
updated: 2026-11-05
verified_on: 2026-11-05
confidence: high
tags: [incident, proxmox, vzdump, backup, nfs, ct299, sftpgo, tailscale, manual-backup, ssd0]
sources: [live-pve-probe, live-tailscale-probe, live-ssh, live-pve-tasks, live-probe-2026-11-05]
---
# vzdump Froze CT299 / SFTPGo Twice - 11-03 (NFS target) and 11-04 (manual ssd0 run)
> **Updated 2026-11-05 (nightly audit) - CT299 was frozen twice, not once.** The page below documents the
> 2026-11-03 freeze. A **second, separate freeze** ran from **2026-11-04 08:30:04** to **2026-11-05 ~00:30**
> (~16 h), caused by a **manual** `vzdump 299 ... --storage ssd0` run. Live probe at **00:54:21 AWST**:
> `LOCK: 0`, `cgroup.freeze=0`, `ns/cgroup.freeze=0`, guest `ALIVE`, `sftpgo: active` - **service up**.
> See *Second freeze* below.
**Window:** ~2026-11-03 03:00 AWST → 2026-11-04 04:4x AWST (≈25 h)
**Status:** Resolved — wedged `vzdump` killed, guest unfrozen, SFTPGo verified healthy
**Impact:** `sftpgo` (LXC CT299 on `pve`) was **frozen solid** — the container's filesystem was suspended by the
backup job, so `sftpgo`, `tailscaled` and everything else inside it stopped. The tailnet peer went offline and the
service was unreachable for the better part of a day. **Nothing alerted.**
## Symptoms
- Tailscale peer `sftpgo` (`100.88.223.70`) offline; `tailscale ping` → timeout.
- `ssh` into `100.88.223.70` hung until timeout; `pct exec 299 -- ...` also hung (exit 124).
- The guest was not crashed — it was **frozen**: the backup job had taken an LVM snapshot and suspended the guest
filesystem, then stalled before it could release it.
## Root cause
A scheduled Proxmox backup job (`vzdump: backup-tue-g1-299`, `schedule tue 03:00`, `mode snapshot`,
`storage fred-qnap`) started on **Tue 2026-11-03 at 03:00 AWST** and **never completed**. The task wedged at
`create storage snapshot` on the **`fred-qnap` NFS storage** (`192.168.178.31:/Proxmox` mounted at `/mnt/pve/fred-qnap`):
- The NFS write path was unresponsive — a `dd` write probe to `/mnt/pve/fred-qnap` timed out while the task was stuck.
- The `vzdump` child process was in **`D` state** (uninterruptible I/O) on a FUSE `request_wait_answer` — it could not
be killed by a normal signal while blocked in the kernel.
- Its temp directory `/mnt/ssd0/vzdump-tmp/vzdumptmp274590_299/` was left behind, and the guest's filesystem
remained suspended for the whole ~24 h.
**The failure mode is the important part: a backup task that cannot finish also cannot release the guest.** Proxmox's
snapshot mode suspends the guest filesystem while the snapshot is created and the data is read out. A stalled task
therefore does not merely "fail" — it **takes the guest down with it**, silently, until someone notices.
## Remediation (2026-11-04 04:4x AWST)
1. Identified the stuck task on `pve`: `vzdump` parent PID **274590**, child PID **274760** (state `D`).
2. `kill -TERM` on both PIDs — the parent died, the child remained blocked in `D` state on the FUSE wait.
3. `kill -KILL` on the remaining child — the kernel eventually released it once the NFS request timed out.
4. **The guest unfroze immediately.** `pct exec 299 -- /bin/echo ALIVE` → `ALIVE` (previously exit 124 / timeout).
5. Verified end to end:
- `pct exec 299 -- systemctl is-active sftpgo` → **`active`**
- `pct exec 299 -- ss -ltn` → `LISTEN *:8080` (WebAdmin) and `:2022` (SFTP)
- `tailscale status` → peer `sftpgo` **Online: true** again
- `tailscale ping 100.88.223.70` → **`pong from sftpgo via 192.168.178.92:41641 in 0s`** (direct, not DERP)
- `curl http://100.88.223.70:8080/` → **HTTP 200**
## Post-remediation storage check
`fred-qnap` was healthy again by the time the task was cleared:
- `pvesm status` → `fred-qnap nfs active 14,380,169,216 KiB total / 12,084,987,904 used / 84.04 %`
- `dd` write probe of 5 MiB with `oflag=direct` → **28.6 MB/s**, exit 0 — no stall.
- The other backup task running concurrently (`vzdump: backup-sun-g1-100`, started 2026-11-03 10:36 AWST) **finished
successfully** at 2026-11-04 04:56:59 (`TASK OK`, 00:16:25) against the same storage.
So the NFS stall was **transient, not a dead mount** — it hit the one task that happened to hold a guest suspended.
## Second freeze - 2026-11-04 08:30:04 to 2026-11-05 ~00:30 (~16 h)
The 11-03 freeze documented above was **not the only one**. A live probe of pve (`100.96.100.82`) on
**2026-11-05 00:54:21 AWST**, plus the node's task history, shows a **second, separate wedge** the next morning:
- `pvesh get /nodes/pve/tasks` lists **two CT299 `vzdump` tasks on 2026-11-04** - and **neither is the 03:00
fleet job**:
- **`2026-11-04 07:36:44` - `vzdump` / status `job errors`** - a clean failure after ~11 minutes. Its log
survives on the dump target: `/mnt/ssd0/dump/vzdump-lxc-299-2026_11_04-07_36_44.log` (597 B, mtime Nov 4 07:47).
- **`2026-11-04 08:30:04` - `vzdump` / status `unexpected status`** - the wedge. `unexpected status` is what a
killed task leaves behind; the 11-03 `03:00:04` row reads the same.
- Three `push_file` tasks from client **`100.118.5.51` (CT460)** ran at **07:17:27 / 07:27:18 / 07:33:07**,
immediately before the first attempt - so the sequence reads as operator/agent-initiated, not scheduled.
- The abandoned temp dir **`/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299`** is still on disk (the dump dir holds
8.8 G total; the only other artefact is `openweb-245-preupgrade-20261101-211545.tar`).
- **Both attempts targeted the SSD that is bind-mounted into the guest** (`--storage ssd0`, `/mnt/ssd0/dump`) -
the same hazard class as the 11-03 NFS target: the guest is snapshotted onto storage it is actively using.
**Recovery state, verified live 2026-11-05 00:54:21 AWST:** `LOCK: 0` in `/etc/pve/lxc/299.conf`;
`/sys/fs/cgroup/lxc/299/cgroup.freeze = 0` **and** `ns/cgroup.freeze = 0`; `pct exec 299 -- /bin/echo ALIVE`
returns `ALIVE`; `systemctl is-active sftpgo` -> `active`; **0** `vzdump`/`lxc-freeze` processes; SFTPGo
reachable end-to-end from CT460 at `http://100.88.223.70:8080`. **CT299 is not frozen now.**
**Open:** why a 299 dump was attempted twice on 11-04 outside any schedule, and whether the `ssd0` target
should be used for this guest at all. See [[infrastructure/proxmox]] and the open items below. The abandoned
`vzdumptmp3750030_299` directory should be reclaimed once nobody needs it.
## Prevention / open items
- **Nothing monitored the backup tasks.** A `vzdump` that hangs for 24 h on a guest-holding storage is an outage
generator, and no page, cron job or dashboard covers it. A watchdog that alerts on a `vzdump` task older than
~2 h — or that checks for `vzdump` tasks holding a suspended guest — is the missing control.
- **NFS `hard` mount semantics amplify this.** `fred-qnap` is mounted `hard,fatal_neterrors=none`, so a stalled NFS
server parks I/O in uninterruptible sleep (`D` state) indefinitely rather than failing fast. A `soft`/`intr` mount
or a `timeout=` tuned for the backup path would let the task die instead of freezing the guest.
- **The temp dir must be reclaimed manually.** `/mnt/ssd0/vzdump-tmp/vzdumptmp274590_299/` was left on disk by the
killed task; the existing cleanup cron skips a directory whose task is still registered.
- **Do not assume "guest offline" means "guest crashed."** This guest was healthy and simply frozen. The first two
probes (ping, ssh) both time out in exactly the same way as a dead guest — the distinguishing signal is a
`vzdump` task in `D` state with an open snapshot on that guest.
## Related
- [[infrastructure/proxmox]] — host, guest and storage inventory
- [[infrastructure/backups]] — Hermes-side config backups (a separate mechanism from PVE `vzdump`)
- [[incidents/2026-09-02-sftpgo-outage]] — the previous CT299 outage (swap exhaustion + cgroup pinning) — a
**different** root cause with the same symptom
- [[incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure]] — prior NFS-layer failure in this estate
- [[systems/scheduled-tasks]] — cron fleet (no backup-task watchdog exists today)
@@ -0,0 +1,132 @@
---
title: Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and Two Cron Prompts (2026-11-05)
type: incident
status: open
created: 2026-11-06
updated: 2026-11-09
verified_on: 2026-11-09
review_after: 2026-11-13
confidence: high
tags: [security, credentials, mnemosyne, memory, sessions, cron, incident]
sources: [live-env-scan, live-sqlite-mnemosyne, live-jobs-json, live-state-db, live-deliveries-db, live-config-backup]
---
# Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and Two Cron Prompts (2026-11-05)
**Status: OPEN - found by the nightly audit (2026-11-05 window), not remediated.** No key was rotated, no
memory row deleted and no cron prompt scrubbed. This page records *where* the values live and how many rows
carry them; **no credential value appears on it** ([[SCHEMA]] rule 1 - record where secrets live, never their
values).
## What was found
Three exposures, all verified by direct read on **2026-11-06 00:22-00:50 AWST**.
### 1. Live `.env` values as plaintext rows in the memory store
**17 rows** of `~/.hermes/mnemosyne/data/mnemosyne.db` carry values from `~/.hermes/.env`. Method: every
TEXT/BLOB column of every table, plus the FTS mirrors, scanned for exact substring membership.
| Key (name only) | Rows | Columns carrying it |
|---|---|---|
| `TELEGRAM_BOT_TOKEN` | 4 | `memories.content` x1, `episodic_memory.content` x1, `gists.text` x1, `fts_episodes.content` x1 |
| `MATTERMOST_TOKEN` | 6 | `memories.content` x1, `episodic_memory.content` x1, `gists.text` x2, `memoria_facts.context_snippet` x1, `fts_episodes.content` x1 |
| `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` | 7 | `memories.content` x2, `working_memory.content` x1, `episodic_memory.content` x1, `gists.text` x1, `fts_episodes.content` x1, `fts_working.content` x1 |
- **Every hit is in `mnemosyne.db`** (27.7 MB, live). The `episodic_memory.db` and `memoria_facts.db` files
beside it are **0-byte legacy stubs** (created 2026-10-27) - the tables with those names live *inside*
`mnemosyne.db`. Nothing was found in the cross-agent shared surface
(`~/.hermes/data/shared/mnemosyne.db`).
- **The rotation snapshots carry them too.** Seven uncompressed snapshots under
`~/.hermes/mnemosyne/backups/` (`mnemosyne-pre-orphan-cleanup.db`, `-20261004_030035`, `-20261011_030046`,
`-20261018_030021`, `-20261025_030108`, `-20261101_030033`, `-20261105_140314`) contain all three values in
full; the 2026-09-27 snapshot predates them. The `.db.gz` snapshots are **compressed and were not
substring-tested** - assume the same until proven otherwise.
### 2. One live key embedded verbatim in two cron prompts
The `CAMOFOX_API_KEY` value appears in the `prompt` text of two **enabled** jobs in
`~/.hermes/cron/jobs.json`:
- `0e753e9720fd` - *Facebook Birthday Monthly Scrape* (`0 9 1 * *`)
- `fad5a86979d4` - *Facebook Birthday Weekly Registry Sync* (`0 3 * * 1`)
Consequence: the same value is inside delivery records in `~/.hermes/cron/deliveries.db`. Of the **10 `.db`
files under `~/.hermes/config-backup/`** (six of them 0-byte), `cron/deliveries.db` is the **only** one that
carries any tested value; `cron/executions.db`, `cron/notepad.db` and `data/shared/mnemosyne.db` carry none.
### 3. Session transcripts hold all four values
`~/.hermes/state.db` (1.7 GB *at discovery* — **1.89 GB / 1,890,656,256 B as at 2026-11-09 00:0x AWST**, with
**2,901 sessions / 197,682 messages** and a live **40,157,672 B WAL**; the store grew while nothing was
retentioned), the session store, contains all four values. Measured as 4 MB chunks
containing the value: **`CAMOFOX_API_KEY` 178, `MATTERMOST_TOKEN` 64,
`HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` 61, `TELEGRAM_BOT_TOKEN` 22**. This is the **most likely origin of
exposure 1** - the memory-capture path writes from session text, so a session in which a key was printed
becomes a memory row - and it is itself a plaintext at-rest store that deleting memory rows does not clear.
## Why it matters
- Mnemosyne rows are **recall material**: they are pulled into a session's context by semantic or FTS match,
so a secret in a row is a secret in a prompt - echoable into any cron delivery (Telegram, Mattermost,
email), any subagent transcript, or the cross-agent shared surface.
- It turns every **backup** into a credential copy. Deleting the rows does not recall the copies;
**rotation is the only remedy that fully closes it**.
- `jobs.json` is the fleet's source of truth and is exported and examined routinely; a prompt-embedded key
is exposed to anything that reads job definitions.
- `state.db` is the fleet's largest artifact (**1.89 GB as at 2026-11-09**, up from 1.7 GB) and is copied into
snapshots and repair backups. ⚠ **It is also the load source behind the 2026-11-07/08 hook-timeout
cascade** — a 40 MB WAL checkpointing on the same spindle as a 2-core guest → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]].
The **2026-08-11 corruption remnant `state.db.corrupted.20260811_151052` (624.1 MB)** was deleted in the
2026-11-08 window, so one stale copy did leave the box — but **`mnemosyne/data/mnemosyne.db.corrupted.20260809_120404` (23.6 MB) is still there**, so the rotation-snapshot exposure above is untouched.
## Origin (inferred - not established)
Almost certainly the **memory-capture path storing verbatim session text** in which a key was printed by a
tool call or a `.env` read. Not verified: no capture-time or delivery-time redaction filter is known to
exist, and the affected rows carry no marker distinguishing them from ordinary content.
## Remediation options (proposed, none executed)
1. **Rotate** the three Mnemosyne-exposed values and the Camofox key - touches live services; needs a
maintenance window and explicit approval.
2. **Redact or forget the affected rows** (`mnemosyne forget <id>` per row), then **re-take the rotation
snapshots** so the superseded values leave the backup set.
3. **Move the two Facebook Birthday jobs off prompt-embedded keys** - read the key from the environment or
config at run time.
4. **Add capture-time and delivery-time redaction** so `.env` values cannot enter memory rows, job prompts or
delivery records again.
5. **Decide a retention/serialisation policy for `state.db`** - it currently holds every transcript
verbatim at 1.89 GB (2,901 sessions / 197,682 messages) and it is growing.
## Verification evidence
- Key names and lengths extracted from `~/.hermes/.env` (values never printed), then matched as exact
substrings across every TEXT/BLOB column of every table in `mnemosyne.db` including the FTS mirrors -
row counts as tabled.
- Byte-membership scans of the 10 `.db` files under `~/.hermes/config-backup/`, the shared-surface DB, the
live `cron/deliveries.db` + `cron/executions.db`, the seven uncompressed Mnemosyne rotation snapshots, and
a chunked scan of `state.db`.
- `json.dumps()` membership test of each value against every job object in `jobs.json`.
- **Not done:** no value was printed, logged or written to any wiki page; nothing was deleted, rotated or
scrubbed.
## 2026-11-08 00:5x AWST — nightly re-check (still OPEN; one narrow improvement)
- **Cron job prompts: clean.** A regex sweep of every job in `/home/hermes/.hermes/cron/jobs.json` for
secret-shaped strings (`sk-…`, `ghp_…`, and any `<key|token|password|secret> = <16+ chars>`) returned
**0 hits** on 2026-11-08 00:5x. Only the *shape* was tested — a secret that does not look like a secret
would still pass.
- **Redaction setting: not confirmable from the file.** `~/.hermes/config.yaml` still carries a
`redact_secrets:` key, but its on-disk value reads as a masked token when printed, so the effective
on/off state could not be established by reading it. The 2026-11-05 finding — *"no capture-time or
delivery-time redaction filter is known to exist"* (line 80 above) — therefore **stands unrefuted**.
- **Nothing else moved:** no key rotation, no affected row redacted or forgotten, no remediation option
executed. Both **Remediation options 2 and 4 above remain open**.
## See also
- [[infrastructure/credentials-map]] - where secrets live, by design
- [[systems/mnemosyne]] - the memory store whose tables carry them
- [[systems/scheduled-tasks]] - the Facebook Birthday jobs' schedules
- [[SCHEMA]] - the no-values rule this violates
@@ -0,0 +1,90 @@
---
title: "Skills Index Generator Divergence — Two Scripts, One Page"
type: incident
status: resolved
created: 2026-11-05
updated: 2026-11-05
resolved_on: 2026-11-05
verified_on: 2026-11-05
confidence: high
tags: [skills, wiki, drift, auto-regeneration, root-cause]
sources: [live-filesystem-probe, script-diff, wiki-maintain-run, agent-estate-wiki]
---
# Skills Index Generator Divergence — Two Scripts, One Page
## Summary
`systems/skills-index.md` flipped between two different shapes on alternating days — **302 skills / 68
categories** on some runs and **349 skills / 26 categories** on others. The cause was **two different
scripts** writing the same page, each with its own generator, and the two scheduled jobs pointing at
different ones. Whichever job ran last won, so the page oscillated nightly.
This is the **same class of defect** as [[incidents/2026-08-15-skills-index-drift]], but a different and
previously-unidentified root cause. The 2026-08-15 incident attributed the drift to archived skills being
counted; that was a *symptom* of one of the two generators, not the mechanism.
## Root Cause
Two files both named `wiki-maintain.sh`, each containing a **complete, independent copy** of the page
generator:
| Script | Generator technique | Output |
|---|---|---|
| `/home/hermes/workspace/agent-estate-wiki/wiki-maintain.sh` (repo, canonical, mtime 2026-11-03) | `find` scan **excluding** `*/.archive/*`; dedupes by `(category, skill_name)` | **302 skills / 68 categories** |
| `/home/hermes/.hermes/scripts/wiki-maintain.sh` (stale copy, mtime **2026-08-17**) | `Path.rglob('SKILL.md')` — **includes** the parked `~/.hermes/skills/.archive/` tree; 2-segment category heuristic | **349 skills / 26 categories** |
The two scripts are **not** identical — `diff` between them returns a non-zero exit and shows entirely
different generator bodies, different category heuristics and different output formats.
**The trigger:** `~/.hermes/skills/devops/wiki-maintenance/SKILL.md` (line 569) instructed the nightly audit
job (`cbfe434a3b5e`, daily 23:59) to run:
```bash
run `bash ~/.hermes/scripts/wiki-maintain.sh` (it regenerates `systems/skills-index.md`, writes a freshness
```
— i.e. the **stale** copy. The 03:00 stocktake job (`1a8e90ea2271`) runs the **repo** script. So:
- 03:00 stocktake → repo script → page becomes **302 / 68**
- 23:59 nightly audit → stale script → page becomes **349 / 26**
- repeat, forever
Each job was individually "successful" — neither reported an error. The defect was only visible as a
day-over-day oscillation in the committed page.
**Live counts verified 2026-11-05 05:1x AWST** (filesystem probes):
- `find ~/.hermes/skills -name SKILL.md` → **349** (includes archive)
- `find ~/.hermes/skills -name SKILL.md -not -path '*/.archive/*'` → **303**
- the generator de-dupes one nested alias duplicate → **302 unique** ✅
- `~/.hermes/skills/.archive/` → **46** parked skills (excluded by the canonical generator)
## Impact
- The published skills inventory was **wrong roughly half the time**, understating the real category
topology (26 vs 68 categories) and overstating the count by including 47 archived skills.
- Any agent reading the page to reason about the estate's skill surface got a materially different
picture depending on which run last wrote it.
- **No functional impact** — skill loading was never affected; only the documentation was wrong.
## Resolution
1. **`~/.hermes/scripts/wiki-maintain.sh` is now a delegating wrapper.** It contains **no generator logic** —
it `exec`s the canonical repo script and forwards arguments. The wrapper's header comment explains why
it must stay that way, so the divergent copy is not recreated.
2. **`--no-build` is now supported by the repo script.** The stale copy accepted `--no-build` (regenerate +
audit, skip the Quartz build/deploy); the repo script did not. The repo script gained the flag so the
wrapper can honour the interface its callers were written against **without** carrying a second copy of
the generator. This closes the only functional gap the delegation could otherwise have introduced.
3. **Verified by execution, not by inspection:** running the wrapper with `--no-build` produces the canonical
shape — `302 skills in 68 categories` — and the generated page reports `46 archived skills` excluded.
Both scripts pass `bash -n`.
4. **`systems/skills-index.md` regenerated through the canonical path** and committed in the canonical shape.
## Lesson
**A generator must live in exactly one place.** Two copies of the same script, differing in one technique
(`rglob` vs `find`), produced two valid-looking outputs and a page that flipped daily — with **no error
anywhere**, because each job did exactly what it was told.
When a scheduled job's output disagrees with another job's output for the same artefact, the cause is
almost never the artefact — it is **two writers**. Diff the scripts before diffing the data.
Related: [[incidents/2026-08-15-skills-index-drift]] (the original drift), [[systems/skills-index]],
[[systems/scheduled-tasks]].
@@ -0,0 +1,86 @@
---
title: Job Radar — a concurrent fire discarded the 11:30 slot's result
type: incident
status: open
created: 2026-11-07
updated: 2026-11-07
verified_on: 2026-11-07
review_after: 2026-11-14
confidence: high
tags: [cron, job-radar, fire-claim, concurrency, scheduler]
sources: [live-executions-db, live-deliveries-db, live-agent-log, live-cron-incidents, cron-output]
---
# Job Radar — a concurrent fire discarded the 11:30 slot's result (2026-11-06)
**Summary.** On **2026-11-06** the *Daily Marketing Job Radar — Write and Letters* (`5fc292ff0f3b`,
schedule `30 11,14 * * 1-5`) had **two live runs at the same time**. A `direct`-source fire started at
**08:27:09** and ran **4 h 59 m 13 s**, completing and delivering its report at **13:26:22**. The
scheduler's own **11:30:32** slot had already started and was still running; when the direct fire settled
the claim, the 11:30 run was judged **stale**, interrupted at **13:26:38**, and closed **`failed`** at
**13:26:44** with a **new** error string:
```
Fire claim ownership lost; stale result was discarded.
```
This is the **first occurrence of a claim-loss failure with no gateway restart involved**, and it raised
**no `cron_incidents` row** — the fleet's only red row for the day is invisible to the alerting ledger.
## Observed sequence (live `executions.db`, `deliveries.db`, `agent.log.1`)
| Time (AWST) | Event | Evidence |
|---|---|---|
| 03:00:32 -> 04:11:35 | wiki stocktake (`1a8e90ea2271`) completes in 1 h 11 m 03 s — overrun streak over | executions row |
| 07:00:44 -> 08:29:21 | Stage 1 (`dca8482e4f76`) completes (1 h 28 m 37 s, `suppressed`), second clean run running | executions row |
| **08:27:09 -> 13:26:22** | **`direct` fire of `5fc292ff0f3b`** — 4 h 59 m 13 s, `completed`, **delivered** | row `ad1fe896…`; log 08:27:13 *"Running job 'Daily Marketing Job Radar - Write and Letters'"* |
| 11:30:32 -> 13:26:44 | scheduler's **own 11:30 slot** fires and runs **concurrently** for 1 h 56 m 12 s | row `95347fa5…` |
| 13:26:22 | direct fire finishes and delivers its report | delivery row 13:26:22 |
| **13:26:38** | *"Job '5fc292ff0f3b': fire claim ownership lost; interrupting stale run"* | `agent.log.1` WARNING |
| **13:26:43** | *"Job '…' failed: RuntimeError: Cron job '…' lost its durable fire claim"* | `agent.log.1` ERROR |
| **13:26:44** | 11:30 row closes **`failed`** — `Fire claim ownership lost; stale result was discarded.` | executions row |
| 14:30:32 -> 14:42:59 | 14:30 slot runs a normal cycle (**12 m 27 s**) and delivers a complete report | row `e7462cf1…` |
**No `.md` output file** exists for the discarded 11:30 run — on 11-06 that job produced only
`2026-11-06_13-26-22.md` (the direct fire) and `2026-11-06_14-42-58.md` (the 14:30 slot).
## What is *not* affected
The day still landed in full. The 14:30 report (read from `cron/output/5fc292ff0f3b/2026-11-06_14-42-58.md`)
shows `SAME_DAY_STRONG_COUNT 4`, `VERIFIED_COVER_LETTER_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`, cache
total **494**, 10 job rows dated 11-06, 20 artifacts present and non-empty, *Run status: COMPLETE*.
## Why it matters
1. **New cause for a known string.** Every earlier `lost its durable fire claim` case came from a
**restart or reload orphaning a live run** ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]],
[[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]), where the text was *"lost its durable
fire claim ownership"*. Here **no restart occurred** — the claim was lost to *another fire of the same
job*. The string is a **generic claim-loss signal, not a restart indicator**; and `executions.db`
currently holds **exactly one** row whose `error` contains `fire claim`: this one.
2. **The failure produced no alert.** No `cron_incidents` row has `last_seen_at` on or after 2026-11-06 —
the ledger's newest activity remains the 2026-11-05T12:36:17 closures. A `failed` execution with no
ledger row means **no notification path fired**; the run only surfaces as a red row in `executions.db`.
3. **~2 hours of a scheduled slot's work was discarded, and the *later, scheduler-owned* run lost** to an
earlier out-of-band run. The stale-discard rule did what it was written to do (protect against stale
results) — but the loser was the scheduler's own slot, so the guard silently ate planned work.
4. **The winner was out-of-band.** Its `source` is `direct`, not the scheduler's `builtin`, and it started
~2 minutes before Stage 1's handover run closed (08:29:21). The timing is consistent with
[[runbooks/job-radar]]'s Stage-1 handover trigger; the trigger's exact origin was **not directly verified**
in this audit.
## Open questions / candidate fixes (none applied)
- Should the scheduler **skip or suppress** a slot when a live claim for the same job already exists,
instead of launching a second concurrent run?
- Should an **out-of-band `direct` fire** (e.g. Stage 1's handover) mark the next scheduled slot consumed,
the way a consumed handoff file silences it?
- Should a **discarded-stale run raise an incident row** at all? Today the fault is silent by construction —
a `failed` execution with no ledger row and no alert.
## Related
- [[systems/scheduled-tasks]]
- [[runbooks/job-radar]]
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]
@@ -0,0 +1,129 @@
---
title: Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07)
type: incident
status: open
created: 2026-11-08
updated: 2026-11-09
verified_on: 2026-11-09
review_after: 2026-11-15
confidence: high
tags: [cron, fire-claim, scheduler, gateway-restart, drain, conduit, health-monitor, wiki-stocktake]
sources: [live-executions-db, live-cron-incidents, live-agent-log, live-systemd-journal, conduit-self-update-json, live-jobs-json]
---
# Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07)
**Summary.** A gateway restart was **requested at ~04:29:52 AWST on 2026-11-07**. Hermes held it in
*restart-after-turn* mode for the full **1800 s** window, then timed out and drained (`#77184`). The drain
killed the two long-running cron executions that were still in flight, and both were closed **`failed`**
with `RuntimeError: Cron job '…' lost its durable fire claim ownership`:
| job | id | run window (ledger) | closed |
|---|---|---|---|
| *Mnemosyne Health Monitor* | `ea79a2107e36` | 00:00:42.918 → 05:02:22.870 | `failed` |
| *wiki-stocktake-maintenance* | `1a8e90ea2271` | 03:00:31.515 → 05:02:22.174 | `failed` |
systemd restarted the unit at **05:01:00** (`hermes-gateway.service: Scheduled restart job, restart counter
is at 3.`); the replacement process was the one that wrote both terminal `failed` rows at 05:02:22.
**The two runs were collateral, not the cause** — nothing in the ledger shows them failing on their own.
## Timeline (AWST; `journalctl --user -u hermes-gateway` unless stated)
| time | event |
|---|---|
| 04:27:55 | `conduit_push` self-update starts (`0.122.2 → 0.134.4`, [[systems/conduit]]) — `~/.hermes/conduit-self-update.json` |
| ~04:29:52 | a gateway restart is requested — **inferred**, not logged: the after-turn wait that expires at 04:59:52 is 1800 s |
| 04:29:54.774 / .796 | `ERROR cron.scheduler: Job 'ea79a2107e36'/'1a8e90ea2271': Restart-safe cron worker exited with status 1 after adopting this execution, without writing a durable terminal state; whether side effects ran is unknown.` → **incident rows raised** |
| 04:29:5x → 04:59 | the gateway **itself keeps running** — `pid 175269` keeps logging at 04:34, 04:39 … 04:59 |
| 04:32:32 | conduit's ledger marks its update `step: failed`, `failed_step: restarting_gateway`, detail *"The plugin is updated, but the gateway didn't restart. Run hermes gateway status on the host."* |
| 04:59:52.433 | `WARNING gateway.run: Restart after-turn wait timed out after 1800s with 3 still active; proceeding to stop()/drain which may interrupt remaining work (#77184)` |
| 05:00:43.469 | `ERROR cron.scheduler: Job 'Mnemosyne Health Monitor' failed: RuntimeError: … lost its durable fire claim ownership`; admitted turn `cc64fdf07d8c4dbd898d83dc2fa74894` fails |
| 05:00:47.344 | the same for `wiki-stocktake-maintenance`; admitted turn `56c8fe136a1241e7827316299b490ecb` fails |
| 05:00:50.863 | `Force-killed MCP process 175401 (cua-driver) after SIGTERM timeout` |
| 05:01:00 | systemd: *Scheduled restart job, restart counter is at 3* |
| 05:01:41 | gateway back up as `pid 2981474` |
| 05:02:22 | both ledger rows finalised `failed` |
| 05:02:41 → 05:02:59 | job `96eb95ce12f2` (agent-log ERROR watchdog) exits 1 having caught the health monitor's traceback — **it reported the event correctly, ~19 s later** |
| 05:43:03/04 | `hermes-dashboard` stopped and started (the 0.22.5 → 0.23.0 update, [[systems/hermes-agent]]) |
| 11:11:44 → 11:11:55 | the gateway is restarted again to load the reinstalled `hermes-hq-push` 0.9.42 routes ([[systems/hermes-hq-push]]) |
| 15:43:17 / 17:08:52 | the health monitor's two incident rows are closed / resolved |
| 2026-11-08 00:00 | **this audit:** both stocktake rows are still `detected`, `alerted_at: null` |
## Incident-ledger state (live read 2026-11-08 00:0x AWST)
| id | job | state | first seen | alerted_at | closed |
|---|---|---|---|---|---|
| `1a8e90_2f370ef764f5` | wiki-stocktake-maintenance | **`detected` (open)** | 2026-11-07T05:02:21.573 | **null** | — |
| `1a8e90_628d914f1f97` | wiki-stocktake-maintenance | **`detected` (open)** | 2026-11-07T04:29:54.815 | **null** | — |
| `ea79a2_111e67a3d83c` | Mnemosyne Health Monitor | `closed` | 2026-11-07T04:29:54.795 | 04:29:57.581 | 15:43:17 |
| `ea79a2_f317fcd843ae` | Mnemosyne Health Monitor | `resolved` | 2026-11-07T13:36:56.713 | 13:36:59 | 17:08:52 — *"Cron execution unverified; reconcile `…/admissions/df79590101fb3b3164e2261a1018f78473c5`"* |
The day ends with **two open rows on one job, neither alerted** (`alerted_at` null) — the same *red row, no
alert* shape the 2026-11-06 audit recorded for Job Radar, here on a different failure path.
## Root cause and working hypotheses
**Established.** This is a **restart-drain** event, not a scheduler or model fault: the ledger error string,
the 1800 s after-turn timeout, and the systemd restart on the same minute line up. The runtime behaviour is
`#77184` — a requested restart is deferred until active turns finish, and when the 1800 s window expires
the gateway proceeds to `stop()`/drain *"which may interrupt remaining work"*. The two cron runs were that
remaining work.
**Not established — the restart's origin.** The request at ~04:29:52 is:
- **2 s before** both restart-safe cron workers exited status 1 (04:29:54), and
- inside the `conduit_push` self-update window (`0.122.2 → 0.134.4`, 04:27:55 → 04:32:32), whose own final
step is recorded as **`restarting_gateway` = failed**.
*Working hypothesis:* **the conduit self-update requested the restart, could not complete it (the gateway
runs under systemd), recorded failure — and the gateway went on to drain and restart 31 minutes later.**
Every timestamp fits; no log line names the requester, so this is not proven. The alternative (another
component requested it) cannot be excluded. The 0.22.5 → 0.23.0 updater is **excluded**: its own receipt
does not begin until **05:05:13**, after the restart was already pending.
**Second, independent defect — adoption lost the claim instead of resuming it.** Both workers *adopted*
their executions at 04:29:54 and then exited status 1 without a durable terminal state; the adopted runs
were never re-driven, and the claims were only closed by the next process at 05:00:43/47. The ledger
cannot say whether their side effects ran (*"whether side effects ran is unknown"*), so the stocktake's
2026-11-07 edits are known to exist only because an in-day session committed them
([[systems/scheduled-tasks]]).
## Impact
- **Mnemosyne Health Monitor** delivered nothing from its 00:00 slot. Its 11-06 slot had taken
**7 h 47 m 56 s** and its 11-05 slot **12 h 13 m 54 s**, so this slot was already deep in its multi-hour
tail ([[systems/mnemosyne]]).
- **wiki-stocktake-maintenance** lost its 03:00 slot for a **second consecutive day** (`failure_streak: 2`,
`last_status: error`) — no stocktake ran on 2026-11-07.
- **Job Radar was not affected on 11-07:** no slot failed and the day's report landed normally, unlike
2026-11-06 ([[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]).
## Verification evidence
journalctl --user -u hermes-gateway --since '2026-11-07 04:00' --until '2026-11-07 05:10'
/home/hermes/.hermes/cron/executions.db (status/error per row; cron_incidents table)
/home/hermes/.hermes/conduit-self-update.json
/home/hermes/.hermes/cron/jobs.json (failure_streak, last_status)
## Open items
- The two open stocktake rows have **never been reconciled** (`alerted_at` null) — either resolve them or
fix the alerting path that let a `detected` row sit silent.
⚠ **2026-11-09 re-check: still open, and now three.** `1a8e90_628d914f1f97` and `1a8e90_2f370ef764f5`
(both first seen 2026-11-07) are unchanged, and a **third** arrived —
`1a8e90_a9cad99cb150`, `detected` 2026-11-08T04:35:38, `alerted_at: null`, from the 03:01:11 → 04:36:08
failure on `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending…`.
**Unclosed incident rows 2 → 3.** The alert path itself is **not** broken: the *Newsletter Digest* row
`85edf6_0877ca1e0eac` alerted at **07:23:07** and closed as `resolved` at **08:06:54** the same window.
So the gap is specific to the **stocktake's** rows.
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
- Decide whether the conduit self-update should stop requesting a gateway restart it cannot perform, and
whether the 1800 s after-turn wait should refuse to drain long-running cron executions.
## See also
- [[systems/scheduled-tasks]]
- [[systems/hermes-agent]]
- [[systems/conduit]]
- [[systems/mnemosyne]]
- [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]
@@ -0,0 +1,165 @@
---
title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08)
type: incident
status: active
created: 2026-11-09
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-11-20
confidence: high
tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor]
sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json]
---
# The `conduit_push` `pre_tool_call` hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08)
**Summary.** From **2026-11-07 14:14:03** to **2026-11-08 08:21:03** the `conduit_push` plugin's
**sole `pre_tool_call` callback** overran Hermes's **30 s hook budget** and was skipped. Each skip
suppresses the callback for a window, so the *pre-tool* side of tool dispatch degraded from *slow* to
*blocked* — this is the fault that ate the 2026-11-08 morning and stretched that day's cron runs to
**6 h** (*Mnemosyne Health Monitor* 12:00:51 → 18:00:42) and **1 h 13 m** (*FreshRSS Midday* 12:00:50 →
14:10:43). Fixed **2026-11-08 13:35 AWST** by raising `plugins.hook_callback_timeout` **30 → 120**.
## Evidence (live reads, 2026-11-08 23:5x – 2026-11-09 00:1x AWST)
grep -c "callback _pre_tool_call timed out" ~/.hermes/logs/errors.log -> 58
# 2026-11-07: 12 lines 2026-11-08: 46 lines
# first 2026-11-07 14:14:03,075 last 2026-11-08 08:21:03,863
grep -c "skipped after previous timeout" ~/.hermes/logs/*.log -> 89
The exact pair of lines (`~/.hermes/logs/errors.log`):
2026-11-08 08:19:40,942 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call skipped after previous timeout or while still running
2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
**The register (correction to a plausible-but-wrong diagnosis).** `conduit_push/__init__.py:119` is the
**only** `pre_tool_call` registration on the host, and the plugin's caches are already in place —
`session_titles.py:21 CACHE_SECONDS = 30.0`. Timed in-process by the 2026-11-08 session:
**3.1 s cold, 0.00 s warm**. The *"the hook re-opens a 1.87 GB state.db per call, ~9 s each"* theory was
therefore **wrong** — warm calls were free. The timeouts track **load spikes** on the 2-core CT460
(load average **5.89** at 13:27:50; `state.db` carrying a **40 MB WAL** checkpointing on the same spindle),
not per-call database opens.
## Fix applied (2026-11-08 13:35 AWST)
~/.hermes/config.yaml
plugins:
hook_callback_timeout: 120 # was the built-in 30 s default
- **Verified live:** `plugins.hook_callback_timeout = 120`.
- **Verified effective:** no `_pre_tool_call` timeout has been logged since **08:21:03** — re-checked at
the end of the fix window and again at audit time (2026-11-09 00:0x). The **58th** line is still the last.
- **Deliberately NOT done:** rewriting the plugin's caching layer. The caches work; the budget was the
binding constraint. **Next lever is `state.db` checkpoint/retention discipline**, not more caching.
## Consequence — the same signature landed in the cron ledger
The 2026-11-08 03:00 *wiki-stocktake-maintenance* slot (`1a8e90ea2271`, 03:01:11 → 04:36:08) closed
**`failed`** on a **new terminal signature for that job**:
RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model
produced no follow-up text. Send `continue` to let it summarize.
*Working hypothesis, not proven:* a `pre_tool_call` block holding a tool result open is exactly what this
error describes, and it sits inside the same 14 h timeout window. It is **consistent**, not established —
the error string does not name the hook.
## Impact
| item | effect |
|---|---|
| *Mnemosyne Health Monitor* `ea79a2107e36` | 00:01:20 → **08:29:05** (8 h 27 m) and 12:00:51 → **18:00:42** (5 h 59 m 51 s) — its two slowest slots on record, both delivered |
| *wiki-stocktake-maintenance* `1a8e90ea2271` | 03:01:11 → 04:36:08 `failed`; **`failure_streak` 3**; a **third** unalerted `detected` incident row |
| *FreshRSS Midday Briefing* `ecc1fc13655a` | 12:00:50 → **14:10:43** (2 h 09 m 53 s) |
| *Tech & AI Newsletter Digest* `85edf6772bca` | an admission-read `database is locked` incident (`85edf6_0877ca1e0eac`) at **07:23:02**, **resolved** at 08:06:54 when the run completed |
| every interactive tool call | slow, then suppressed — the user-visible symptom that opened the investigation |
## Verification evidence
~/.hermes/logs/errors.log (58 timeout lines; last 2026-11-08 08:21:03)
~/.hermes/config.yaml (plugins.hook_callback_timeout: 120)
~/.hermes/plugins/conduit_push/__init__.py:119,257 (the only pre_tool_call registration)
~/.hermes/plugins/conduit_push/session_titles.py:21 (CACHE_SECONDS = 30.0)
~/.hermes/cron/executions.db, cron_incidents, jobs.json (run windows, streak, incident rows)
## Open items
- **`state.db` is 1.89 GB with 2,901 sessions / 197,682 messages and a live 40 MB WAL** — it is the load
source behind the spikes. Checkpoint and retention policy are still undecided
([[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
- Confirm whether the stocktake's `No reply: … tool result was still pending` failure is really the same
fault; if it recurs while the hook budget is 120 s, this hypothesis is dead.
- Decide whether a skipped `pre_tool_call` should **fail loudly** rather than silently degrade tool dispatch.
## Follow-up (2026-11-10 nightly audit)
- **The fix held; the cascade did not recur.** `errors.log` (1,035 retained lines) contains **no `pre_tool_call` timeout entry at all**. Its only two hook lines are benign skips — `on_session_end` 2026-11-09 22:58:35 and `pre_llm_call` 2026-11-10 00:02:54 (*"skipped after previous timeout or while still running"*).
- **No 11-09 run showed the cascade signature.** Outside the two long-runners documented separately (stocktake **9 h 10 m**, Job Radar Stage 1 **9 h 09 m** — both stalled in *tool iteration*, not in a hook), the longest 11-09 execution was **74 min**, then 40 min and 11 min.
- **2026-11-09 was the first `failed`-free cron day since 2026-11-05** (11-06: 1 failed, 11-07: 3, 11-08: 1, 11-09: **0 of 174**), and unclosed incident rows went 3 → 0.
- **Residual:** the two long-runners above are the same 9-hour-turn environment this bug leaves behind, and one (`wiki-stocktake-maintenance`) is pinned to `free-stack` — a combo recorded broken on 2026-08-14. The job-iteration budget stays in force.
## Recurrence (2026-11-14 nightly audit) — the raised budget did not hold
**Reopened.** The `pre_tool_call` hook overran its **raised 120 s** budget again at **2026-11-14 00:05:40**,
on the wiki-audit job's own tool calls. The 30 -> 120 s fix of 2026-11-08 is therefore **proven
insufficient at current load**, and this page is no longer `resolved`.
| time (AWST) | line |
|---|---|
| 2026-11-14 00:05:40,043 | `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` |
| 2026-11-14 00:05:43 -> 00:06:38 | **10** x `skipped after previous timeout or while still running`, ~6 s apart |
- **Source:** `~/.hermes/logs/errors.log` (single-source count; `agent.log` carries the same lines).
By day: **2026-11-12 = 0 timeouts / 0 skips; 2026-11-13 = 0 / 0; 2026-11-14 = 1 / 10.** It is the
**first `pre_tool_call` timeout in the retained log since the 2026-11-08 cascade** — the 2026-11-08
lines themselves have since **rotated out** of `errors.log`, so this page's original 58-timeout figure
still stands on the 2026-11-09 read and was **not re-verifiable** this window.
- **Cost this time:** this audit's first `skill_view` and `terminal` calls were **refused outright** by the
plugin (first-hand evidence), with dispatch recovering on its own by ~00:06:40 — roughly **one minute**
of hard suppression. It is a **single-event recurrence, not the 18-hour cascade**. **No cron run failed:**
2026-11-13 settled **173 `completed` / 0 `failed`**.
- **Load is the variable, and it is now ~2x the level recorded when the fault was diagnosed:**
`load average **12.08 / 13.02 / 12.33**` on **2 vCPU** (2026-11-08 read: **5.89**). `state.db` has grown
**1.89 GB -> 2.04 GB** (**2,036,908,032 B**; **2,977 sessions / 207,577 messages**, was 2,901 / 197,682),
and the root filesystem is **88 % used**.
- **The earlier open hypothesis is now settled.** *"If it recurs while the hook budget is 120 s, this
hypothesis is dead"* — it recurred, so the load trigger is **confirmed** and there is **no headroom left
in the timeout**: at load ~12 the callback needs >120 s. **Re-raising the budget would only mask the
fault.** The remaining levers are unchanged and now urgent — **`state.db` checkpoint/retention** and
reducing host load (see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
## Recurrence 2 (2026-11-15) — eleven timeouts in two clusters, 10 refusals
Recorded by the **2026-11-17 nightly audit** while backing up the 11-15 window: the **2026-11-15 audit
captured none of this**. Exact `errors.log` counts for 11-15:
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` —
**5** in 01:07:17 → 01:25:43 and **6** in 12:18:15 → 12:48:40 (a two-cluster day, unlike 11-14's one).
- **10** tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
— hour 01: `read_file` ×2 (134.17 s and 0.43 s), `execute_code` ×4, `patch` ×1, `read_file` ×1
(121.03 s); hour 12: `execute_code` ×1. Call-level blocks reach **120.63 s**, and the affected sessions
are `local-989b3642…` (01:xx) and `local-b6a1490f…` (12:xx).
- 11-14, for comparison: **1** timeout and **4** refusals, plus **4** plugin load failures.
- **11-16 and 11-17: 0 timeouts, 0 refusals, 0 load failures.** The episode did not carry over.
The loader half of the same 11-15 storm — **52** failed plugin loads, **519** discarded registrations
(`conduit_push` 474 of them) and **2** `pre_llm_call` timeouts — is now on its own page,
[[incidents/2026-11-15-plugin-load-timeout-storm]]. Its 12:18–12:49 cluster is the useful detail here:
**a hook-timeout cluster can occur with no accompanying loader failure**, so the two share a cause class
rather than a mechanism.
**Effect on the trigger hypothesis recorded above.** Amplitude is not a simple function of load average:
the 11-15 episode ran at a load the audit cannot measure retroactively, while the 11-17 read
(**6.19 / 6.64 / 7.48**) sits *below* the 11-14 figure of 12.08 with **no** recurrence — and a *larger*
`state.db` (2.09 GB, WAL 40 MB, `/` at 88 %). Keep a **load-average threshold out of this page's claim**;
keep **`state.db` size, disk headroom and the gateway's own timing budgets** in it.
## See also
- [[systems/conduit]]
- [[systems/hermes-agent]]
- [[systems/scheduled-tasks]]
- [[systems/mnemosyne]]
- [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
- [[current-state]]
@@ -0,0 +1,89 @@
---
title: "2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op"
type: incident
status: resolved
created: 2026-11-10
updated: 2026-11-11
verified_on: 2026-11-11
confidence: high
tags: [cron, job-radar, stall, silent-failure, handoff, read-loop]
sources: ["executions.db (dca8482e4f76: 2026-11-09T07:00:48+08:00 → 2026-11-09T16:10:12+08:00, status completed, delivery_outcome suppressed; 5fc292ff0f3b: 11:30:38 → 12:10:07 and 14:30:34 → 14:37:27, both completed/suppressed)", "~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md (ends INCOMPLETE: tool budget exhausted; last words \\u201cI stopped retrying because I kept running read_file 5 times without making progress.\\u201d)", "~/.hermes/radar-ref/handoff/ (newest handoff on disk: 2026-11-06; no 2026-11-07/08/09 files)", "cron_incidents (no row raised for this run)"]
---
# 2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op
## Summary
On 2026-11-09 the Job Radar's Stage 1 (`dca8482e4f76`) ran for **9 h 09 m 24 s** (07:00:48 → 16:10:12), ended `INCOMPLETE: tool budget exhausted`, and **never wrote its handoff file**. Because both Stage-2 slots run on their own schedule, **both fired before Stage 1 finished** and each declined silently rather than erroring. Net effect: **the radar landed nothing on 11-09 and raised no alert.**
## Timeline (AWST, 2026-11-09)
| Time | Event |
|---|---|
| 07:00:48 | Stage 1 starts |
| 11:30:38 → 12:10:07 | Stage-2 slot 1 fires — Stage 1 still running → silent no-op (`suppressed`) |
| 14:30:34 → 14:37:27 | Stage-2 slot 2 fires — Stage 1 still running → silent no-op (`suppressed`) |
| ~16:10:1x | Stage 1 writes `INCOMPLETE: tool budget exhausted` |
| 16:10:12 | Stage 1 closes — `completed`, delivery `suppressed` |
| all day | `radar-ref/handoff/2026-11-09.json` is never created |
## Root Cause
Two compounding faults, both previously documented for this job:
1. **Stage 2 never checks that a fresh handoff exists**, so when Stage 1 fails it proceeds and self-suppresses instead of failing loudly. Two chances to notice, zero alerting.
2. **Stage 1 overran without producing output.** Its own final message: *"I stopped retrying because I kept running read_file 5 times without making progress."* This is a **new trigger** for the same silent family seen on 11-02/11-03 (there: iteration cap) — here: an unproductive read loop that consumed the run's budget.
## Evidence
- `executions.db` `dca8482e4f76`: `2026-11-09T07:00:48+08:00` → `2026-11-09T16:10:12+08:00`, `status completed`, `delivery_outcome suppressed`.
- `5fc292ff0f3b` (Stage 2): two rows on 11-09, both **before** Stage 1's finish stamp, both `completed` / `suppressed`.
- `~/.hermes/radar-ref/handoff/` — newest file is **2026-11-06**; nothing for 11-07, 11-08 or 11-09.
- `~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md` ends `INCOMPLETE: tool budget exhausted`.
- **No row in `cron_incidents`** for `dca8482e4f76` on 11-09 — the failure was never raised.
## Impact
- The Job Radar — the estate's job-discovery feed — produced **no output for a third day** (no handoff since 11-06).
- The failure is invisible: the run's ledger entry says `completed`, delivery is `suppressed`, and no incident row exists.
## Fix Applied
**None — documentation only.** No configuration change was made this window.
## Prevention
- **Alert on the artefact, not the exit code.** Stage 2 should hard-fail when the expected `radar-ref/handoff/<date>.json` is missing, rather than proceeding to a suppression. That single check converts both 11-09 slots into loud failures.
- Add a **read-progress guard** to Stage 1: when the same file has been read N times without a state change, stop and emit a partial handoff instead of burning the budget to `INCOMPLETE`.
- Note the pattern across incidents: **completion-time-only alerting hides every failure in this family** (stocktake 11-09, radar 11-09, radar 11-02/03).
## Recovery — 2026-11-11: the handoff landed, and Stage 2 delivered
| Field | 11-09 (this incident) | **11-11** |
|---|---|---|
| Stage 1 | 07:00:48 → 16:10:12 (**9 h 09 m 24 s**), no handoff | **07:01:08 → 07:37:34 = 36 m 26 s**, `completed` |
| Handoff on disk | newest was **2026-11-06** | **`radar-ref/handoff/2026-11-11.json` — 22,682 B, mtime 07:59** |
| Stage 2 (11:30 slot) | fired early, declined silently | **11:30:56 → 11:35:18**, `completed` |
| Stage 2 (14:30 slot) | declined silently | **14:30:55 → 14:34:56**, `completed` / **`delivered`** |
| Incident row | **none raised** | `5fc292_773822455856` closed **`resolved` 2026-11-11T08:03:47.644226** |
- **The first handoff in five days** (`2026-11-06` → `2026-11-11`) and the fastest Stage-1 cycle since
the two-stage split: **36 m 26 s** against 3 h 15 m (10-29, the last full success), 9 h 01 m (11-02),
6 h 28 m (11-03) and 9 h 09 m (11-09).
- **Stage 2's recovery fire** (`source: direct`, 07:35:56 → 08:03:47) consumed the handoff and delivered
at **08:03:47**; its completion is what auto-closed the row opened by 11-10's `interrupted` slot.
- ⚠ **Carried forward from 11-10:** the *scheduled* Stage-2 slot that day ran
**11:30:35 → 22:18:49 = 10 h 48 m 14 s** and closed **`failed` / `interrupted`** (delivery
`delivered`, `for_failure: 1`) — the longest run of either window, and itself a new data point for the
"completion-time-only alerting" family on this page.
- **Job state:** both radar jobs read `last_status: ok`, `failure_streak` 0.
**Status: `resolved`.** Both prevention rules on this page still stand and remain unimplemented: Stage 2
should hard-fail on a missing `handoff/<date>.json` rather than suppressing, and Stage 1 needs a
read-progress guard. Neither was needed on 11-11 because the run finished inside budget.
## Related
- [[systems/scheduled-tasks]]
- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]
- [[runbooks/provider-health]]
@@ -0,0 +1,94 @@
---
title: "2026-11-09 — Stocktake Completes With Zero Edits"
type: incident
status: resolved
created: 2026-11-10
updated: 2026-11-11
verified_on: 2026-11-11
confidence: high
tags: [cron, wiki, stocktake, tool-budget, silent-failure, omni-101, free-stack]
sources: ["executions.db (1a8e90ea2271: 2026-11-09T03:00:43+08:00 → 2026-11-09T12:10:47+08:00, status completed, delivery_outcome suppressed)", "~/.hermes/cron/output/1a8e90ea2271/2026-11-09_12-10-46.md (8,305 bytes; RUN INCOMPLETE, \\u201ctool-iteration budget exhausted during verification\\u201d)", "cron_incidents (1a8e90_628d914f1f97, 1a8e90_2f370ef764f5, 1a8e90_a9cad99cb150 — all closed 2026-11-09T12:10:47.125205+08:00)", "wiki-stocktake-maintenance job 1a8e90ea2271 (provider custom:omni-101, model free-stack)"]
---
# 2026-11-09 — Stocktake Completes With Zero Edits
## Summary
The nightly wiki stocktake (`wiki-stocktake-maintenance`, `1a8e90ea2271`) ran on 2026-11-09 from **03:00:43 to 12:10:47** — **9 h 10 m** — and the harness recorded it **`completed`** with delivery `suppressed`. Its own output file records the opposite: the run ran out of tool-iteration budget inside its verification phase, applied **zero edits**, made **no commit**, and its findings never landed.
**This is the third consecutive day the stocktake's work did not land** — 11-07 died at 2 h 02 m, 11-08 at 1 h 35 m (both `failed`), 11-09 at 9 h 10 m (`completed`). The visible status this time is *worse*, not better: `failure_streak` fell **3 → 0** and `last_status` shows **ok**, while the work stayed undone.
## Timeline (AWST, 2026-11-09)
| Time | Event |
|---|---|
| 03:00:43 | Run starts (`1a8e90ea2271`) |
| ~04:1x | Tool-iteration budget exhausted during verification; output file written with `RUN INCOMPLETE` |
| 12:10:47.125205 | Harness closes the run — `completed`, delivery `suppressed` |
| 12:10:47.125205 | The **three** long-open stocktake incident rows close at the same instant: `1a8e90_628d914f1f97` (11-07T04:29:54), `1a8e90_2f370ef764f5` (11-07T05:02:21), `1a8e90_a9cad99cb150` (11-08T04:35:38) |
## Root Cause
Same family as 11-07/11-08: **the run spends its entire tool budget before reaching the write phase.** The run's own output file records the stop as the tool-iteration budget being exhausted **during verification**, with zero edits applied — the edits it had identified were never written.
Secondary cause (confirmed again this window): the job is pinned to `custom:omni-101` / **`free-stack`** — a combo recorded broken on **2026-08-14** (see [[systems/scheduled-tasks]]).
## Evidence
- `executions.db`: `started_at 2026-11-09T03:00:43+08:00` → `finished_at 2026-11-09T12:10:47+08:00`, `status completed`, `delivery_outcome suppressed`. Previous two runs: 11-08 `failed` (03:01:11 → 04:36:08, 1 h 35 m), 11-07 `failed` (03:00:31 → 05:02:22, 2 h 02 m).
- Output file `/home/hermes/.hermes/cron/output/1a8e90ea2271/2026-11-09_12-10-46.md` (8,305 bytes) ends `RUN INCOMPLETE`, with the tool-iteration budget exhausted inside verification.
- The three stocktake incident rows all carry `closed_at = 2026-11-09T12:10:47.125205+08:00` — i.e. they closed on **completion**, not on a fix. All three have `alerted_at = null`: they never alerted.
- Git: **no stocktake commit on 11-09** — zero edits were applied, so there was nothing to commit and nothing to revert. (On 11-07/11-08 the page was left dirty and then reverted; on 11-09 no edit was ever made.)
## Impact
- Nine consecutive stalled stocktake runs (11-01 … 11-09) — the nightly wiki freshness pass is producing no durable output.
- The stocktake's *top-priority* brief item on 11-09 was the **monotonically worsening failure streak** itself. The job closed its own incident while failing that item, which is the failure mode to watch: **`completed` and `ok` now overstate the stocktake's health.**
- Three incident rows were silently cleared, so the fleet's incident board now reads **0 open** while the underlying problem is unfixed.
## Fix Applied
**None — this is a detection-and-verification record.** The page documents the failure and the misleading status so the next audit starts from truth.
Also recorded this window: the `free-stack` pin on `1a8e90ea2271` — the only deterministic configuration fault in the chain — is **still in place**, unchanged from the 11-08 record.
## Prevention
- **Alerting must fire on write progress, not completion.** This run produced 2 of 3 failure days silently and the third day produced a run that reported success while editing nothing. A check on `commit_count` / `edits_applied` for this job would have caught all three.
- Don't read `status`/`failure_streak` as the stocktake's health. Read whether a commit landed.
## Recovery — 2026-11-11: the pass ran in 1 h 10 m and landed 36 pages
The next scheduled slot **worked**, and worked quickly:
| Field | 11-09 (this incident) | **11-11** |
|---|---|---|
| Window | 03:00:43 → 12:10:47 | **03:00:56.081504 → 04:11:38.724852** |
| Duration | **9 h 10 m** | **1 h 10 m 42 s** |
| Terminal state | `completed` / `suppressed` | `completed` / `suppressed` |
| Edits landed | **0** | **36 stale pages re-verified → 0 stale** |
| Commit | **none** | **`341dda9`** + **`c108eb1`** |
| Freshness report | — | `raw/stocktake/freshness-20261111.txt` (**6,235 B**) |
- `341dda9` re-verified 30+ further pages live and de-flagged them, and applied real content
corrections — the live fallback chain rewritten in [[runbooks/test-openrouter]] (primary back to
`smart-route` / `custom:omni-101`), the CT333 `clawtest` naming claim corrected, and the nous
**404 `insufficient_credits_for_paid_model`** signature replacing the old refresh-token reading in
[[incidents/2026-10-22-nous-portal-refresh-token-invalid]].
- `c108eb1` regenerated the freshness report after the log/index addendum, so the published Quartz build
matches HEAD.
- **Job state:** `last_status: ok`, `failure_streak` 0, `last_run` 2026-11-11T03:00:56.
⚠ The job is **still pinned to `custom:omni-101` / `free-stack`** — the combo recorded broken on
2026-08-14 — and this run succeeded on it anyway. That is an observation, not a repin.
**Status: `resolved`.** The 11-07/11-08/11-09 run of failures is closed by output, not by a cosmetic
`failure_streak` reset. The page's own prevention rule stands: *alert on `commit_count` / `edits_applied`,
never on `status`* — this recovery is exactly why it matters, because 11-09 and 11-11 both read
`completed` / `suppressed` and only one of them did anything.
## Related
- [[systems/scheduled-tasks]]
- [[infrastructure/tailscale]]
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
- [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]
@@ -0,0 +1,118 @@
---
title: Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited
type: incident
status: resolved
created: 2026-11-11
updated: 2026-11-12
verified_on: 2026-11-12
review_after: 2026-12-11
confidence: high
tags: [cron, incident, omniroute, rate-limit, audit]
sources: [live-executions-db, live-cron-incidents, gateway-log, live-jobs-json]
---
# Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited (2026-11-10 23:59 → 2026-11-11 00:12)
**This job is the estate's own observer, and on 2026-11-10 it became the estate's only failure.** Its
23:59 slot never produced a report, so **no page, no log entry and no incident alert describes the day
of 2026-11-10** — an unaudited day is invisible by construction.
## What happened
| Field | Value |
|---|---|
| Job | `cbfe434a3b5e` — **Wiki Audit — Nightly**, `custom:omni-101` / `deepseek-paid-smart` |
| Slot | 2026-11-10 23:59:54.486572 +08:00 (cron `59 23 * * *`) |
| Ended | **2026-11-11 00:12:29.646987 +08:00** — **12 m 35 s** |
| Status | `failed` · `delivery_outcome: suppressed` |
| Error | `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` |
| Output | `~/.hermes/cron/output/cbfe434a3b5e/2026-11-11_00-12-28.md` |
The three log lines from the run's own session `local-1000c1a3…` show the shape of the failure:
ERROR agent.chat_completion_helpers: Streaming failed before delivery: [429]: service failure: tenant tn-5ud77abtl1 quota exceeded
ERROR agent.chat_completion_helpers: Streaming failed before delivery: Error code: 429 -
ERROR agent.conversation_loop: API call failed after 3 retries. HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters
ERROR cron.scheduler: Job 'Wiki Audit — Nightly' failed: RuntimeError: HTTP 429: Service temporarily unavailable
- **The pool skipped every target.** Six candidates were evaluated and **`attempted 0`** — the run never
reached a provider. The only live target's credit balance read **100 %** consumed on connection
`conn:3021bab6`, which is why the router refused to dispatch rather than serving a degraded model.
- **`reset after 6h 24m 31s`** puts the pool's recovery at **≈ 2026-11-11 06:36:57 AWST**. The 07:0x
cron cohort ran clean the next morning, so the window closed on its own.
- The **same tenant** (`tn-5ud77abtl1`) appears in the `quota exceeded` line, so this is an
account-level quota fault on the router, not a single-model lockout.
## Incident row — raised, never alerted, closed `resolved` 2026-11-12T00:25:36
`cbfe43_ed01845310d8` — `job_id cbfe434a3b5e`, `failure_type: rate_limit`, state **`detected`**,
`alerted_at: NULL`, `acked_at: NULL`, `closed_at: NULL`, opened **2026-11-11T00:12:28.578435+08:00**
(`first_seen_at` = `last_seen_at` = that instant — seen exactly once).
- It **was** the only open row in `cron_incidents` — the table held **135** rows at audit time and **137**
by the end of this run. `acked_at` is **NULL** as well as `alerted_at`, so nothing was told and
nothing was acknowledged. The previous open item,
`5fc292_773822455856`, closed `resolved` at **2026-11-11T08:03:47.644226**.
- So the estate's single outstanding incident **was** the audit's own failure, and nothing was told
about it. ⚠ **Superseded 2026-11-12** — the row closed `resolved` at **2026-11-12T00:25:36.133387+08:00**;
*cron_incidents* now holds **0 unclosed of 137**. See *Resolution* below.
## Resolution — 2026-11-12T00:25:36, self-cleared
The row closed **`resolved`** at **2026-11-12T00:25:36.133387+08:00**, at the end of the **next** slot of
the same job: `cbfe434a3b5e` ran **2026-11-11T23:59:54.486572 → 2026-11-12T00:25:36.133387 = 25 m 42 s**
and completed. Its delivery carried **both windows**, so 2026-11-10's day was reconciled a day late
rather than never (the recovery record is the `audit(2026-11-11)` commit series at 2026-11-12 00:1x-00:24).
Nothing was repaired to make this happen: the pool's own reset at ≈ **2026-11-11 06:36:57** restored the
dispatch path, exactly as the `reset after 6h 24m 31s` field predicted, and the 07:0x cohort ran clean.
`cron_incidents` now reads **0 unclosed of 137** rows — the estate is incident-clean as of 2026-11-12.
**The recurrence risk is unchanged.** The same *"all targets were skipped by pre-dispatch filters"*
signature was found again on a **second job and a second pin** by the 2026-11-12 03:00 stocktake
([[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]), and every model-calling job still rides
`custom:omni-101` with no failover for the free-tier cohort. Closing this row proves the guard
self-heals; it is not evidence the guard is gone.
## Impact
- **The 2026-11-10 window was never reviewed.** The nightly audit is the only producer of the per-day
ledger/wiki record; its failure means 11-10's 157 ledger rows were never reconciled on the day.
- **Fleet non-clean `last_status` 1 → 2.** `cbfe434a3b5e` now reads `last_status: error`,
`failure_streak: 1`, joining `066bb5191280` (*Uber Eats Monthly Spend Report*, `delivery_failed`,
streak 0).
- **The review published on 2026-11-11 23:59 carries both windows** — it is the recovery, not a
duplicate.
## Same signature, earlier appearances
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — the *identical* pre-dispatch-filter
string killed the Daily Marketing Job Radar on 2026-10-21 (`attempted 0`, no 503 anywhere in the day).
- 2026-11-05 — `deepseek-paid-smart` against the **CT600** endpoint returned `HTTP 503
ALL_TARGETS_SKIPPED` (`poolSize 5`, `attempted 0`) while `omniroute-1` answered 200 — recorded on
[[systems/scheduled-tasks]].
- This is the **third distinct symptom** of the same router behaviour: `attempted 0` with a healthy node
behind it is the thing to recognise, not the specific HTTP code.
## Fix Applied
**None — this is a detection-and-verification record.** No configuration changed this window. The run's
own recovery was the pool reset at ≈ 06:36:57 on 2026-11-11, and the incident row closed on the next
slot's clean completion (2026-11-12T00:25:36).
## Prevention
- **The observer needs an observer.** A failed nightly audit raises no delivery (suppressed), no alert
(`alerted_at: NULL`) and no page — every layer that would normally surface this is *inside the thing
that failed*. A one-line watchdog on `Wiki Audit — Nightly`'s `last_status != ok` (or on "no audit
commit in >24 h") is the only check that can catch it.
- **Alert on the artefact.** Verifying that a dated `audit(YYYY-MM-DD)` commit exists in this repo would
have caught both this failure and the stocktake's zero-edit days — the same lesson recorded on
[[incidents/2026-11-09-stocktake-completes-with-zero-edits]].
## Related
- [[systems/scheduled-tasks]]
- [[systems/omniroute]]
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]
- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]
@@ -0,0 +1,98 @@
---
title: hermes-hq-push 0.9.47+ — Upgrade Refused by the Plugin Security Scan (dangerous verdict)
type: incident
status: open
created: 2026-11-13
updated: 2026-11-13
verified_on: 2026-11-13
review_after: 2026-12-13
confidence: high
tags: [plugins, security-scan, hq-push, plugin-guard, install, incident]
sources: [live-hermes-plugins-list, live-plugin-manifest, live-config, live-plugin-guard-source, live-gh-pr, live-state-db]
---
# hermes-hq-push 0.9.47+ — Upgrade Refused by the Plugin Security Scan (2026-11-12)
**The scan did its job and there is no way to overrule it.** On 2026-11-12 (~17:38–18:29 AWST) the
attempt to take [[systems/hermes-hq-push]] past its installed **0.9.42** was refused before any file
was written, by the pre-install plugin security scan. The installed copy is untouched, still `enabled`,
and still delivering. The blocking finding is in the plugin's **own** `dashboard-tile` skill, and the
fix is sitting in an open pull request.
## What happened
| Field | Value |
|---|---|
| Attempted | `hermes plugins install mrcharlesiv/hermes-hq-plugins/hermes-hq-push --enable --force` |
| Refusal | `Blocked: Security scan blocked plugin install: Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict.` |
| Findings | **39 total — 1 critical**, the rest low/medium |
| Sole critical | `hermes-hq-push/skills/dashboard-tile/SKILL.md:41` — category *obfuscation*: the `echo '<json>' \| python3 …` one-liner |
| Verdict logic | **any** critical ⇒ `dangerous` = hard block; medium/low alone ⇒ `caution` (installable, warned) |
| Installed after | **unchanged** — `v0.9.42`, tree mtime **2026-11-08 02:19**, `enabled`, state `user` |
## Why no override was available (verified in source, not assumed)
- **The only config key is `plugins.scan_on_install`** (default `true`; currently **not set** in
`~/.hermes/config.yaml`, so scanning is on). Setting it `false` disables the scan for **every** future
install — it does not overrule a verdict, it blinds the check. That is the wrong lever.
- **`--force` lifts `caution` only.** Its own refusal text says so, and the code agrees. `--allow-removed`
is unrelated — it is the blocklist for catalogue removals.
- **No env var, no trusted-source list, no allowlist, no `INSTALL_POLICY`/`trust_level` for plugins.**
Those mechanisms exist only in `tools/skills_guard.py`, reachable from the *skills* path
(`skills.guard_agent_created`). `tools/plugin_guard.py:398` hardcodes `trust_level="community"` and
`should_allow_plugin_install` never reads it.
- **The block is a single unconditional branch** at `tools/plugin_guard.py:406-418`: any verdict outside
`safe`/`caution` returns `False, "Blocked (dangerous verdict, …)"`.
## The fix, and why it is not merged
The critical is a *pattern* match, not a defect: the tile skill embedded its JSON payload in a shell
one-liner, which the scanner reads as obfuscation however benign the intent.
| Field | Value |
|---|---|
| Repo / PR | `mrcharlesiv/hermes-hq-plugins` **#2** — the repo's **only** open PR |
| Author | `Tony0410` (Anthony) |
| Head | fork `Tony0410/hermes-hq-plugins`, branch `fix/tile-skill-heredoc` → base `main` |
| State | **OPEN**, `mergedAt: null`, `mergeable: MERGEABLE`, created 2026-11-12T10:00:01Z (updated 10:01:57Z) |
| Diff | **+7 / −2 in one file** — `hermes-hq-push/skills/dashboard-tile/SKILL.md` (the one-liner becomes a heredoc) |
| Local clone | `/home/hermes/workspace/hq-plugins-fix` (commit `90c9d28`, branch checked out, clean) |
The `gh` CLI on this host is authenticated as **`Tony0410` with read-only access** to
`mrcharlesiv/hermes-hq-plugins`, so the change had to travel fork → PR rather than as a branch push.
## Impact
- **The upgrade path to 0.9.47+ is blocked** until PR #2 merges upstream and the plugin is re-installed.
Nothing else in the estate is affected.
- **No app-side impact:** the installed 0.9.42 keeps delivering to the Hermes HQ iPhone app; the refusal
happened before any file was replaced, so there is no half-installed state to clean up.
- The block is **fail-safe by design** — the failure mode is "cannot upgrade", not "installed something
unchecked".
## Related finding — the "stored hold-and-ask instruction" was never stored
The same evening, the alleged standing instruction to *hold and ask before acting on* plugin
security-scan warnings was searched for exhaustively: a recursive sweep of `~/.hermes`
(**184,716 files**, generated/venv/browser-profile/SQLite trees excluded) for the phrases
`wait for my answer`, `show me what it found`, `security scan flags` and `security-scan` found **zero
matches in any editable text file**. The sentence exists only in `state.db` **conversation history**
(58 rows). **Nothing was removed and no policy was reverted**, because no such policy was ever
persisted. Procedure recorded as the skill `stored-instruction-search-and-removal`.
## Prevention / how this closes
- **Merge PR #2**, then re-run the install. That is the whole fix.
- If a standing exception is ever genuinely wanted, **add an explicit trusted-source or allowlist path to
`tools/plugin_guard.py`** — do **not** set `plugins.scan_on_install: false`. Turning the scan off is
not "allowing this plugin"; it is removing the check from every future install, including the ones
nobody is looking at.
- Expect the same class of false positive again: any plugin shipping a shell one-liner that pipes JSON
into an interpreter will read as *obfuscation*. Author the pattern as a heredoc or a script file.
## Related
- [[systems/hermes-hq-push]]
- [[systems/hermes-agent]]
- [[systems/scheduled-tasks]]
- [[log]]
@@ -0,0 +1,332 @@
---
title: Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
type: incident
status: active
created: 2026-11-15
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
confidence: high
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
---
# Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
> ⚠ **2026-11-20 re-read — the freeze ended by itself at 2026-11-18 19:25 AWST (105 h 52 m total), and every
> interactive session inside the window was captured.** The "capture is dead" reading below is a *quiet estate*,
> not a broken hook. Read the newest re-read section before the earlier readings.
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
(exit 1, incident `ea79a2_601037516abd`, `failure_type: auth`, state **`alerted`**, **unclosed**).
Found by the **2026-11-14 nightly audit**; not caused by anything in that audit's window (it started
the previous morning and simply had not been noticed, because the monitor's own earlier reading of
the same condition was classified informational — see below).
## Re-read 2026-11-16 00:18 AWST — still open, stall 38 h 46 m, and now **three** unclosed alerts
Six hours later, at the next nightly audit, nothing has moved. The freeze is **total**, the monitor has
fired twice more, and the scheduler has minted further unclosed rows for the same fault.
- **The counts are frozen, not merely low.** `working_memory` **1130** / `episodic_memory` **1249** —
**the same two numbers as every earlier read of the stall** (2026-11-15 ~00:4x and 12:03) — so this is
not a quiet stretch or a degraded trickle. Zero writes in **38 h 46 m**.
- **The whole write path stopped, not just turn capture.** Newest rows remain the three
`[MODEL_REFRESH_PROPOSAL]` maintenance writes at **`09:32:58` / `09:32:59` AWST**; the newest
*user-turn* capture is **`2026-11-14 09:04:19 AWST`**. Maintenance outlived the last turn by ~28 min,
then both stopped inside the same minute.
- **`cron_incidents` 1 → 3 unclosed** (138 → **141** rows total). All three are `ea79a2107e36`, all
`failure_type: auth`, all `state: alerted`, and **each has `first_seen_at == last_seen_at`** — a single
observation, no `last_seen_at` accumulation, no escalation:
| Incident id | `first_seen_at` | `alerted_at` |
| --- | --- | --- |
| `ea79a2_601037516abd` | `2026-11-15T00:03:35.815373+08:00` | `2026-11-15T00:03:44.559501+08:00` |
| `ea79a2_5f817708701e` | `2026-11-15T12:03:14.928500+08:00` | `2026-11-15T12:03:20.245081+08:00` |
| `ea79a2_75303493d484` | `2026-11-16T00:03:15.418001+08:00` | `2026-11-16T00:03:22.365399+08:00` |
- ⚠ **New mechanism, evidenced: the incident signature embeds the elapsed-hours float.** The three
signatures differ only by `4.64157453` / `4.634034074444445` / `4.643481075555555` — a value **never
equal twice** — so the recurrence never matches the prior row. One durable fault therefore **mints a
fresh unclosed row every slot**, and the estate's unclosed count grows ~1 per 12 h while no single row
ever escalates. This job alone now has **15** `cron_incidents` rows (12 older `unknown`, these 3
`auth`). **Proposed fix — NOT applied** (it changes the monitor's incident path, so it is Anthony's
call): normalise the value out of the signature — round it, or omit it — so the same condition
**re-opens the existing row** instead of fragmenting.
### The monitor's ≈4.64 h figure is NOT the stall — verified against the gate's own source
Every alert in this sequence prints a near-identical elapsed time while the stall grew from 14 h to
38.7 h. That number is not a measurement of the stall. `mnemosyne_freshness_gate.py`'s
`newest_user_message()` selects the newest `messages` row with `role='user'` whose session
`source NOT IN ('cron','subagent')` (the `NON_CONVERSATIONAL` set, line 52) and returns that row's
`source`; the alert prints **`(now − that message) / 3600` h**. So the figure is **the age of the newest
*conversational* message — not the age of the newest capture.**
The newest conversational messages on this box are **Pulse** turns recorded under `source: desktop`
(`Pulse delivery check pXXXX-1 …`, `Pulse morning brief …`), on a **~6 h cadence**. Each alert's
≈4.64 h lands on the then-newest Pulse turn to within about a minute:
| Alert | Elapsed printed | Now ≈ | Newest conversational message it measured |
| --- | --- | --- | --- |
| 2026-11-15 00:03:35 | `4.64157 h` | 00:00:54 | **2026-11-14 19:22:30** (desktop / Pulse) |
| 2026-11-15 12:03:14 | `4.63403 h` | 12:01:25 | **2026-11-15 07:23:25** (desktop / Pulse) |
| 2026-11-16 00:03:15 | `4.64348 h` | 00:01:33 | **2026-11-15 19:22:59** (desktop / Pulse) |
**Why they all read ≈4.64:** the `0 */12` slots happen to fire ~4.6 h after the most recent Pulse turn.
The near-constancy is a **cadence coincidence** and must **not** be read as "the stall is only ~4.6 h" —
the stall is **38 h 46 m**. Same trap class as the naive-UTC note below: a number shaped like a duration
that is not one.
**Positive proof the capture path is dead rather than merely idle:** the identical class of message *was*
captured before the stall — the 2026-11-14 00:09:12 run logged
`last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min`. Four 2026-11-15 Pulse turns
(`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`) now have **no capture row at all**.
### The consolidator is alive and every cycle is a deliberate no-op
Latest `consolidation_log` row **4696** at `2026-11-15T16:14:14.934306+00:00` (= **2026-11-16 00:14:14
AWST**), `items_consolidated: 0`, `session_id: auto-consolidate-maintenance`; cadence ~31 min
(`15:12:13` → `15:43:08` → `16:14:14` UTC). **New evidence:** the row's `summary_preview` reads
**`No-op consolidation cycle — timestamp updated for watchdog`** — the cycle **touches its own timestamp
to satisfy an age test**, so a zero-item streak can run indefinitely while every age-based gate reads
"fresh". The streak is now **~74 consecutive zero-item cycles**, opening
`2026-11-14T02:04:42+00:00` (10:04:42 AWST). **A zero-item-streak threshold — not an age threshold — is
what would have caught this at 10:04 on 11-14.**
### Adjacent fault in the same runtime — the watchdog lost a slot (2026-11-15 11:31)
`96eb95ce12f2` (**Mnemosyne Watchdog**, every 15 min) **failed** on its `11:31:20.699` slot,
`finished_at 2026-11-15T11:49:10.534610` — **17 m 49.8 s**, exit 1:
```
runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds
```
A **600 s timeout inside a 15-minute slot** overran two slots: the `11:45` slot was **skipped** (no row
at all) and the **next slot, `12:01:25.120952`, completed** (`→ 12:02:02.657231`), so it self-recovered
with one slot lost. This is a **new signature** for this job — the August and October occurrences had the
guard quoting *another* job's error text. Across 11-15 → 11-16 it is the only `failed` row of its 97.
Alert delivered (`deliveries.db` `11:49:06.052341`, `for_failure: 1`, `delivered`). **Adjacency, not
proof of common cause** — but the guard and the capture path share the same runtime, so a guard that
times out at 600 s belongs next to this stall rather than dismissed.
## Re-read 2026-11-17 00:27 AWST — stall **62 h 54 m**, **five** unclosed alerts, zero-item streak **120**
Third nightly read, sixth week of the same numbers changing only their counters. **The freeze is still
total.**
- **Counts identical for the fifth consecutive read:** `working_memory` **1130** / `episodic_memory`
**1249**; newest rows still `2026-11-14 09:32:59` AWST (working) and `09:33:34` AWST (episodic) —
**62 h 54 m** with **zero** writes.
- **`cron_incidents` 3 → 5 unclosed** (141 → **143** rows total). Both new rows are `ea79a2107e36`,
`failure_type: auth`, `state: alerted`, and again `first_seen_at == last_seen_at`:
| Incident id | `first_seen_at` | `alerted_at` |
| --- | --- | --- |
| `ea79a2_915d004a108a` | `2026-11-16T12:02:06.679532+08:00` | `2026-11-16T12:02:12.472265+08:00` |
| `ea79a2_0fab8d43db75` | `2026-11-17T00:03:37.624640+08:00` | `2026-11-17T00:03:46.151392+08:00` |
The signature-fragmentation mechanism below is holding exactly as described: **one fresh row per
firing, no escalation**, so the estate's unclosed count now grows **~2 per day** with nothing accumulating
on any single row.
- **The monitor is on a 5-slot failure streak** (`jobs.json`: `ea79a2107e36`, `last_status: error`,
`failure_streak: 5`). 11-16's ledger holds **2 failed rows of 175, both this job**
(00:01:33 → 00:03:22, 12:01:18 → 12:02:12); 11-17 so far 1 failed of 7 (00:01:39 → 00:03:46).
Every failure is `exit 1` with `STALE MEMORY PIPELINE: capture missed: us…` — the same gate.
- **The consolidator is alive and deliberately doing nothing.** `consolidation_log` latest row **4742**
at `2026-11-16T16:00:41.922076+00:00` (= 00:00:41 AWST), `items_consolidated: 0`, summary *"No-op
consolidation cycle — timestamp updated for watchdog"*. Last non-zero row: **4622** at
`2026-11-14T09:33:34` → **120 consecutive zero-item cycles** (was 74 at the 11-16 read). The cycles
still fire ~every 31 min; there is simply **nothing new to consolidate**, which is the memory-side
reason the stall cannot self-heal.
- **Still no cause, still no fix, still `status: active`.** The 11-16 read's two conclusions stand: the
failure is **total**, not a trickle, and it affects **all** write paths, not just turn capture.
`review_after` 2026-11-18 — this page is re-read at the next audit, and the monitor keeps alerting.
## Re-read 2026-11-20 00:5x AWST — the freeze ended by itself after **105 h 52 m**, and the "capture is dead" reading was wrong
**The capture freeze is over. The alarm is not.** Live reads of `mnemosyne.db` at 2026-11-20 00:0x–00:5x AWST:
| counter | 2026-11-17 00:27 | 2026-11-20 00:5x |
| --- | --- | --- |
| `working_memory` | 1130 | **1131** |
| `episodic_memory` | 1249 | **1250** |
| newest write | `2026-11-14 09:32:59` AWST | **`2026-11-18 19:25:03` AWST** (stored UTC `11:25:03`) |
The break coincides with nothing: no config edit, no restart and no repair is recorded against the capture path
in the window — the only nearby state change is the 11-18 19:18 desktop session itself. The single new
`working_memory` row carries `session_id: hermes_20261118_191808_5c9e08` and its content begins
`[USER] You are helping me migrate contex…`, which identifies the captured turn (the `desktop` session
**`20261118_191808_5c9e08`**, *"Helping migrate context AI assistant"*, 9 messages, started 19:18 AWST).
**Stall duration: 105 h 52 m** (2026-11-14 09:33 → 2026-11-18 19:25 AWST).
⚠ **The correction that matters — the window contained almost no conversation to capture.** `state.db` lists
**three** non-cron sessions since 2026-11-14 00:00 AWST: two `desktop` sessions on **11-14** (**08:35**, 40
messages; **08:52**, 41 messages — both *before* the 09:33 boundary), then **nothing** until the **11-18 19:18**
session, whose turn is the row that ends the freeze. Every interactive session in the window was captured, so
the earlier figures on this page (`0 writes in 38 h 46 m`, `62 h 54 m`) are real but measure **a quiet estate**,
not a broken hook: `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message
whose session source is not `cron`/`subagent`, so **silence and failure read identically**. That ambiguity — not
a broken capture path — is the durable defect on this page.
**The monitor is still failing on that same metric, after the break.** `ea79a2107e36` slots since 11-18:
`11-18 00:01:28` failed · `11-18 12:01:13` failed · **`11-19 00:01:34` `completed`** (the only clean slot since
11-17) · `11-19 12:01:17` failed · `11-20 00:01:22` failed — every failure `Script exited with code 1 / 🚨
Mnemosyne Health Monitor — exception(s): STALE MEMORY PIPELINE: capture …`, i.e. **the same signature firing on
a healthy pipeline** (`failure_streak: 2`; incidents `ea79a2_f2be2db3bfce` at 2026-11-19 12:01:58 and
`ea79a2_aae4d914f3c9` at 2026-11-20 00:03:36, both `alerted`, both `failure_type: auth` — the scheduler's
classifier, not a credential fault). The `11-19 00:01` pass and the `12:01` failure bracket the threshold: the
metric is *age of the newest captured conversational turn*, which only grows while the estate is silent.
**The other half of the original finding is unchanged, and larger.** The consolidator still consolidates zero
items: newest `consolidation_log` row **`2026-11-19T16:20:14.587267+00:00`** (= 2026-11-20 00:20 AWST, so it is
running on its ~31-minute cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle
— timestamp updated for watchdog"`. **199 of the last 200 cycles are zero-item** (was **120** at the 11-17 read
— consistent with ~31 min cadence over the three days). The zero-item-streak threshold remains **proposed, not
applied**.
**Unclosed incidents: 5 → 4** (one of the rows present at the 11-17 read has since closed). Current four, none
closed automatically: `cbfe43_fa12cca1ad4b` (*auth*, 2026-11-18 00:01:42, `alerted_at: None` — the wiki audit's
silent skip, see [[incidents/2026-11-18-wiki-audit-credit-outage]]), `96eb95_98ade422d195` (*script*,
2026-11-18 20:01:15 → last_seen 2026-11-20 00:31:16 — see
[[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]), `ea79a2_f2be2db3bfce` (2026-11-19
12:01:58) and `ea79a2_aae4d914f3c9` (2026-11-20 00:03:36).
**Standing open question, restated:** not *"why did capture stop"* but **"why does the gate treat conversational
silence as a fault?"** Until that is fixed, this page will keep re-opening on quiet days.
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
**2026-11-14 09:32:59 AWST** (the row's own `timestamp` field reads `2026-11-14T09:32:59`).
- Newest `episodic_memory.created_at`: **`2026-11-14 01:33:34`** (09:33:34 AWST). **Both** write paths
stopped inside the same minute.
- Last 8 `working_memory` writes, all on 11-14 (UTC): `01:01:20, 01:02:39, 01:02:40, 01:03:28,
01:04:19, 01:32:58, 01:32:59, 01:32:59` — then **nothing for ~15 hours**.
- Rows by day (UTC `created_at`): **11-09: 11, 11-10: 10, 11-11: 8, 11-12: 37, 11-13: 14, 11-14: 15**.
So the pipeline was *low-volume but alive* before the stall, and the stall is not a normal quiet stretch.
- Totals at audit time: **working 1130 / episodic 1249**, DB **27 MB**.
- Gap: at the 00:03:35 alert the newest capture was **14 h 30 m 36 s** old; at this audit's read
(~00:40) **≈15 h 7 m**.
## The alert (and its mislabelling)
```
🚨 Mnemosyne Health Monitor — exception(s):
STALE MEMORY PIPELINE: capture missed: user message 4.64157453 ago (desktop)
has no working_memory row (last capture 2026-11-14 01:32:59)
```
- `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`,
`first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`.
Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`.
- ⚠ **The `auth` label describes the classifier, not the fault — now evidenced, not just suspected.** The
failure is a *staleness* check and the page records no auth error text. This job's **twelve earlier
`cron_incidents` rows** read `agent` ×3, `timeout` ×1, `unknown` ×8 (`ea79a2_649d8ac46e8a`,
`_f17b45759db0`, `_573d17e10fc1`, `_6fb70b032305`, `_54d6b1f0f58a`, `_747b27556fd0`, `_c3ab2d784ba3`,
`_a4661554a3c2`, `_d93c3acdc4f2`, `_9197df32320b`, `_111e67a3d83c`, `_f317fcd843ae`), and the estate-wide
table is **53 `unknown` / 47 `agent` / 15 `timeout` / 14 `script` / 5 `auth`**. `agent` is also impossible
here: the job runs `no_agent` (plain script). Treat the field as opaque routing metadata.
- ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a
~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own
log line below.
## What the earlier slots actually saw — correction, re-derived 2026-11-15 01:2x AWST
**An earlier revision of this page got this backwards.** It claimed the **12:00:52 AWST** run had
"already seen the same hole and called it benign" (`capture delay -589.5 min`), giving the estate ~12 h
of unalarmed lead time, and blamed a "sign, not magnitude" threshold. **Both claims are withdrawn.** The
`-589.5 min` line is not the 12:00 run's — it sits inside the **00:03:35 run's own output block**.
Verified by line number in `~/.hermes/logs/mnemosyne-health-monitor.log` (**1103 lines**):
| line | content |
| --- | --- |
| 884 | `=== 2026-11-14 00:09:12 AWST health monitor (no_agent) ===` |
| 900 | `ℹ️ last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min` |
| 957 | `=== 2026-11-14 12:01:52 AWST health monitor (no_agent) ===` |
| 973 | `ℹ️ last user msg 2026-11-14 01:03 UTC (ios) capture delay 29.4 min` |
| 1030 | `=== 2026-11-15 00:03:35 AWST health monitor (no_agent) ===` |
| 1046 | `ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min` |
| 1047 | `❌ STALE MEMORY PIPELINE: …` |
Two further corrections to the earlier revision:
- **There was no 18:03 run.** The job's schedule is `0 */12 * * *`; the only 2026-11-14 slots are
00:09:12 and 12:01:52 (both exit 0, delivery `suppressed`). The "exited 0 at 12:00 and 18:03"
sentence is withdrawn — 18:03 belongs to some other job.
- **The detection chain was not broken.** `mnemosyne_freshness_gate.py` fires on **magnitude**
(`delay_min < -CAPTURE_LAG_MIN`, `MNEMOSYNE_CAPTURE_LAG_MIN` default **120** min) — there is no
sign-only test to blame. The 12:00:52 measurement of **+29.4 min** is the delay of the newest message
*that had been captured* (the 09:03 AWST `ios` turn, captured 09:32:59). The gate computes against the
newest **uncaptured** message and only reports a problem when one exists; with the conversation idle
from 09:03 to 19:22, every user message still had a capture row, so there was **nothing to detect**.
The first message to postdate the stall was the **19:22 AWST desktop** turn (`11:22 UTC`, the same
message the 00:03 alert names); the next scheduled slot after it was 00:03:35, which escalated
(`-589.5 min` = 09:32:59 − 19:22).
**Corrected statement of the gap:** the pipeline died at **09:32:59 AWST**; it became *detectable* at
**19:22 AWST** (first uncaptured message) and was *detected* at **00:03:35** by the next slot — a
detection latency of 4 h 41 m after detectability, driven by cadence, not by a broken threshold.
**What was genuinely unread sits in the consolidation counters:** the cycles ran every ~31 min and
returned **0 items for 29 consecutive runs** (streak opening `2026-11-14T02:04:42+00:00` = **10:04:42
AWST**; last productive cycles 4622/4621/4620 at `09:33:34` / `09:33:16` / `09:01:40`). Every run
printed `Last consolidation: … (0 items)` — including the "healthy" 12:00:52 one — but the gate tests
consolidation **age** only (`> 36 h`, default), never the item count. **An empty-but-current cycle is
the loudest signal available and the only one nothing watches.**
## Impact
> **Magnitude superseded 2026-11-16 00:18 AWST: 38 h 46 m uncaptured** (was ~15 h), with **two more
> unclosed alerts** and four 2026-11-15 Pulse turns (`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`)
> added to the uncaptured set. The ~15 h figures below stand as the 2026-11-15 readings.
- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST**
(the last user message the monitor could see), all Telegram traffic, and every cron session in the
window. Recall over that period will answer from nothing.
- The **23:30 AWST sleep/consolidation cycle ran and found 0 items** to consolidate
(`Last consolidation: 2026-11-14T15:30:51 UTC (0 items)`), so consolidation is *not* compensating.
- Nothing else in the estate broke: **2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` /
1 `running`** in `executions.db`, deliveries **4 / 4 `delivered`**, and no other cron row went red. This is a
**memory-path incident only**. The one exception is the monitor's **own** job: it is now the fleet's
**single non-clean row** (`last_status: error`, `failure_streak` 1, previously `ok` / 0) and
**`cron_incidents` went 0 → 1 unclosed** (138 rows total). **2026-11-15 so far: 8 rows — 7
`completed` / 1 `failed`** (this job), 1 delivery `delivered`.
## Not the known config killer
The previously recorded capture-killer on this host is a **stringified list/dict** written into
`~/.hermes/config.yaml` by `hermes config set`. Checked and **clean**:
- `config.yaml` **mtime 2026-11-13 18:03** — the file was not modified during or before the stall began.
- A full YAML walk found **no** scalar that begins with `[` or `{`; `memory.mnemosyne.*` parses as
proper lists (`ignore_patterns: []`, `skip_contexts`, `sync_roles`).
## Open / unverified
- **Root cause: not established.** No capture-path error lines appear in `agent.log` after 09:33 —
the writes simply stop. Candidates to test next: a stuck capture worker or watermark, the
`pulse`/heartbeat traffic that ran 12:03–19:22, session-side `memory_enabled` /
`mnemosyne.profile_isolation` interaction, or the sleep cycle's `consolidation_claimed_at` state.
- ✅ **Resolved — the 00:03 escalation DID deliver.** `deliveries.db` holds exactly one 2026-11-15 row:
**`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the
incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked;
the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]).
- ✅ **Answered 2026-11-16 00:18 — capture did NOT resume.** Two further monitor slots failed on the
same condition (`12:03:14` and `2026-11-16 00:03:15`) and the row counts are unchanged (`working`
1130 / `episodic` 1249). Closed as *no*; the underlying cause remains open.
- **The estate's only automated alarm is understating this incident.** The alert's ≈4.64 h is a cadence
artefact, not a stall duration — anyone reading the alert text alone would conclude the pipeline is
~4.6 h behind when it is 38 h 46 m. A stall-duration figure (newest capture vs now) would fix it.
## Related
- [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence
- [[systems/memory-backend]] — memory backend settings
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] — the still-open secrets-at-rest page for the same database
- [[systems/scheduled-tasks]] — the fleet's execution ledger
@@ -0,0 +1,117 @@
---
title: Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
type: incident
status: unresolved
created: 2026-11-17
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-11-24
confidence: high
tags: [hermes, plugin, conduit, timeout, infrastructure, incident, load]
sources: [live-errors-log, live-config-yaml, live-gateway-log, live-cron-jobs-json]
---
# Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
**Not observed on 2026-11-16 or 2026-11-17 — 0 lines of either signature on both days.** No fix was
applied; the storm stopped as abruptly as it started, and the loader budget that produced it is
unchanged, so recurrence is possible. Found by the **2026-11-17 nightly audit**, which was filling the
11-15 window's gaps; the **2026-11-15 audit recorded none of this**. Every count below is an exact
`errors.log` grep, not an estimate.
## What the log records (2026-11-15)
- **52** × `Failed to load plugin '<name>': load timed out after 10s (import + register() never returned)`
- **519** × `Plugin '<name>' called register_{tool,hook,skill,platform}() after its load timed out; ignored`
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping`
- **2** × `Hook 'pre_llm_call' callback _pre_llm_call timed out after 120s — skipping` (06:05:33, 08:41:41)
- **10** × tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
Across the log's retained window (`errors.log` starts **2026-11-09 13:18**; `.1` / `.2` rotations hold
older lines):
| Day | Plugin load failures | Registrations dropped | `pre_tool_call` timeouts | `pre_llm_call` timeouts | Tool refusals |
| --- | --- | --- | --- | --- | --- |
| 2026-11-12 | 0 | 0 | 0 | 0 | 0 |
| 2026-11-13 | 0 | 0 | 0 | 0 | 0 |
| 2026-11-14 | 4 | 0 | 1 | 1 | 4 |
| **2026-11-15** | **52** | **519** | **11** | **2** | **10** |
| 2026-11-16 | 0 | 0 | 0 | 0 | 0 |
| 2026-11-17 | 0 | 0 | 0 | 0 | 0 |
### By plugin (2026-11-15)
| Plugin | Failed loads | Registrations dropped |
| --- | --- | --- |
| `hermes-hq-push` | 26 | 0 |
| `conduit_push` | 9 | **474** |
| `homeassistant` | 7 | 34 |
| `browser-browser-use` | 3 | 3 |
| `basic` | 2 | 2 |
| `cadu-secrets-vault` | 1 | 4 |
| `telegram-platform` | 1 | 1 |
| `nous` | 1 | 0 |
| `drain` | 1 | 0 |
| `browser-browserbase` | 1 | 1 |
| **Total** | **52** | **519** |
## The dropped registrations are the more serious half
A failed load is a warning line. The `register_*()` lines that follow are **function loss**: the loader
gives each plugin a **10 s** budget for `import + register()`, and when that budget expires the plugin's
late registrations are **discarded, not honoured**. `conduit_push` alone lost **474** registrations across
**eight** load events (01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47) — i.e. for those
processes the Conduit tools, skills and hooks were **not registered at all**, silently. See
[[systems/conduit]].
## Clustering
These arrive in bursts, not as a trickle. 18 distinct registration-drop groups on 11-15; load failures
spread over hours 01, 04–11 and 13:
- **01:05–01:32** — `hermes-hq-push` (01:05, 01:17, 01:32) and `homeassistant` (01:17) fail to load,
`conduit_push` fails at 01:32:39 and drops ~52 registrations. Inside the same 20 minutes
`pre_tool_call` times out **5×** (01:07:17, 01:16:11, 01:19:36, 01:22:39, 01:25:43) and **9** tool calls
are refused in the same session (`local-989b3642…`).
- **04:46–04:47** — the day's largest burst: `browser-browser-use`, `conduit_push` ×2, `hermes-hq-push` ×2
and `homeassistant` all fail within ~55 s; **118** registrations discarded.
- **06:31–06:48, 07:06, 08:16, 08:46, 10:47** — repeated `conduit_push`/`hermes-hq-push` failures, each
dropping ~52–56 registrations.
- **12:18–12:49** — a **second** `pre_tool_call` cluster (6 timeouts + 1 refusal) with **no load failures**
in hours 11–12 bar one at 11:49 — so the hook half of this page's family can fire without a loader
failure alongside it.
## What this is — and is not
- **Not a misconfiguration.** `plugins.hook_callback_timeout: 120` is set in `config.yaml` and **no**
plugin-load-timeout key exists there — the 10 s loader budget is a **code default**. The failures are a
**saturation symptom** on this host, not a wrong setting.
- **Same window as the hook cascade.** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
recurred on 11-15 with the 11 `pre_tool_call` timeouts and 10 refusals counted above; the runtime deps
guard also blew its **600 s** budget that day (see
[[incidents/2026-11-14-mnemosyne-capture-stall]]). Several of the estate's own timing budgets were
exceeded in the same 24 h.
- **Adjacency, not proof.** Nothing ties the three to a single cause. What they share is a host under
sustained IO/scheduler pressure while `state.db` had grown past **2 GB** with a **40 MB** WAL (`/` at
88 %). But load average at the 2026-11-17 audit read **6.19 / 6.64 / 7.48** — *below* the 11-14 reading
of 12.08 — with **no** recurrence of this class on 11-16/11-17, so **load average alone does not
predict it**.
- **Not host restarts.** `gateway.log` has **0** startup/shutdown markers on 11-15, and `uptime` at the
audit was 6 d 2 h (host last up **2026-11-10 ~22:04**) — so the ~18 load events are **in-process plugin
(re)loads**. Plugin registration output in `agent.log` appears at agent-process start (e.g. the
2026-11-16 00:00:00 block), which is why a job starting during saturation can run with plugins missing.
## Open question
Why did `import + register()` exceed 10 s for whole plugins, and does the loader fail **closed or open**
for the caller? Today it fails **open** — the job continues, silently without those tools. A check that
asserts **plugin registration** rather than only job exit status would surface this class; the same idea
applies to the hook callback budget. **Proposed, not applied** (it changes Hermes runtime behaviour, so
it is Anthony's call).
## See also
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — the companion hook-callback cascade
- [[systems/conduit]] — what the 474 dropped `conduit_push` registrations cost
- [[systems/hermes-agent]] — plugin loading, config and timeouts
- [[incidents/2026-11-14-mnemosyne-capture-stall]] — same-window runtime pressure
@@ -0,0 +1,111 @@
---
title: Mnemosyne Watchdog red for 115 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST)
type: incident
status: active
created: 2026-11-20
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
confidence: high
tags: [mnemosyne, memory, conflicts, watchdog, cron, alert-fatigue, incident]
sources: [live-cron-executions-db, live-cron-incidents, live-mnemosyne-db, live-deliveries-db, live-jobs-json]
---
# Mnemosyne Watchdog red for 115 consecutive slots (from 2026-11-18 20:01 AWST)
**Open.** `96eb95ce12f2` (*Mnemosyne Watchdog*, 15-minute cadence) has failed **every slot** since
**2026-11-18 20:01:06 AWST** — **115 consecutive failures** through **2026-11-20 00:31:16 AWST**, the last
recorded before this page was written (it was 114 when the audit began; the streak gained a slot mid-audit).
`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`**.
This is the estate's **only** red cron row with a large streak, and it is a *single durable condition*
re-reported every 15 minutes, not a series of separate faults.
## The failure is a fixed, self-describing threshold trip
```
Script exited with code 1
stdout:
🚨 Many unresolved conflicts: 59 (threshold: 50)
```
`cron_incidents` row **`96eb95_98ade422d195`** — `state: alerted`, `failure_type: script`,
`first_seen_at 2026-11-18T20:01:15.906837+08:00`, `last_seen_at 2026-11-20T00:31:16.326357+08:00`,
`alerted_at 2026-11-19T20:46:23.428143+08:00`, `closed_at: None`, output file
`/home/hermes/.hermes/cron/output/96eb95ce12f2/2026-11-20_00-16-15.md`.
Unlike the health monitor's rows (whose signatures embed a changing float and so fragment into a new row per
slot → [[incidents/2026-11-14-mnemosyne-capture-stall]]), **this one fixes on the constant count `59` and
therefore stays a single row.** The cost of that stability is the mirror image: it never closes, and it
re-alerts forever.
## Alerting works — and that is the problem
`deliveries.db`, window 2026-11-18 → 2026-11-20 00:0x — the watchdog's own failure notices:
| delivered | execution |
|---|---|
| 2026-11-18 20:01:16 | `f9865723b4f64d9fbfca7711ab053056` |
| 2026-11-19 02:16:16 | `7aa35681fd914a1e8db64b8dd0e09c84` |
| 2026-11-19 08:31:17 | `1ccc7e280d234287973edf5e0b35a5b7` |
| 2026-11-19 12:01:59 | `396940d9514b419eb024b83b3c10198d` |
| 2026-11-19 14:31:35 | `0766953f92dc4fc4b0d0404dca51683f` |
| 2026-11-19 20:46:19 | `db010a8294bf4cd78457a265a5fb167c` |
| 2026-11-20 00:03:37 | `64c6b5202463487c906f860bb746138b` |
All `status: delivered`, all `for_failure: 1`. So detection, the failure notice and the push path are **all
working**. The output is an alert roughly every 3–6 h for a threshold breach that has not moved in two days —
which is exactly how a red signal stops meaning anything: while this row is red, a *new* Mnemosyne fault
arriving on the same job is indistinguishable from the standing one.
## What the 59 conflicts actually are
Live read of `~/.hermes/mnemosyne/data/mnemosyne.db` (2026-11-20 00:3x AWST): **223 conflicts total, 59
unresolved.** Unresolved by creation day:
| created | unresolved |
|---|---|
| 2026-10-31 | 11 |
| 2026-11-02 | 2 |
| 2026-11-03 | 12 |
| 2026-11-06 | 1 |
| 2026-11-07 | 2 |
| 2026-11-08 | 1 |
| 2026-11-12 | 13 |
| 2026-11-13 | 3 |
| **2026-11-18** | **14** |
| **total** | **59** |
Two things stand out:
1. **Most of the backlog is old.** 42 of the 59 predate 2026-11-14; the oldest batch is from **2026-10-31**,
which is the same batch recorded on [[systems/mnemosyne]] as *"all created in one batch at 2026-10-31
19:26:39 and sharing a single `fact_a_id`, the extraction-noise signature"*. The documented remediation for
this class is to bulk-resolve them as `false_positive` (see the `mnemosyne-operations` skill) — **not
applied here**.
2. **The threshold has been raised before.** The 2026-10-18 and 2026-11-01 reads on [[systems/mnemosyne]] both
record *"1 unresolved conflict (benign, threshold 10)"*. It now reads **50**. So the tripwire was already
loosened 5×, and the backlog has since grown past even that. **UNVERIFIED:** when the threshold was changed
from 10 to 50, and by whom; basis: two dated page reads vs the live alert text — the config line itself was
not opened in this window.
## The 2026-11-18 batch coincides with the migration session
The 14 conflicts created on **2026-11-18** are the newest batch, and 2026-11-18 is also the day of the only
interactive session in this window — `20261118_191808_5c9e08` (desktop, 9 messages, title *"Helping migrate
context AI assistant"*), whose turns were captured into `working_memory` at **19:25:03 AWST** and summarised
episodically at **19:55:20 AWST**. Consistent with the standing assessment that most conflicts in this bank are
LLM-extraction artefacts rather than real contradictions, but **UNVERIFIED:** that the 14 rows are extraction
noise from that session's turns specifically; basis: same-day coincidence plus the page's prior noise analysis —
the fact pairs were not inspected.
## Not applied / open
- **Nothing was resolved, reclassified or deleted.** No conflict row was touched; no threshold was changed.
- The watchdog's cadence, signature construction and alert path were left as-is.
- Open question: should a *standing* threshold breach alert on state change rather than on every slot?
Same family as the zero-item-consolidation streak problem on
[[incidents/2026-11-14-mnemosyne-capture-stall]] — both are conditions that re-report continuously because
the gate tests a level, not a change.
→ [[systems/mnemosyne]], [[systems/scheduled-tasks]]
@@ -0,0 +1,89 @@
---
title: Wiki Audit failed two consecutive nights on a chained-provider credit/auth fault (2026-11-18, 2026-11-19) — 11-17 and 11-18 unaudited
type: incident
status: active
created: 2026-11-20
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
confidence: high
tags: [cron, wiki-audit, incident, model-routing, credentials, credits, router, silent-failure]
sources: [live-cron-executions-db, live-cron-incidents, live-jobs-json, live-deliveries-db, live-wiki-log]
---
# Wiki Audit failed two consecutive nights (2026-11-18, 2026-11-19) — 11-17 and 11-18 were never audited
**Open.** `cbfe434a3b5e` (**Wiki Audit — Nightly**, schedule `0 0 * * *`) failed on **two consecutive slots**,
so the audit that should have reviewed **2026-11-17** and the audit that should have reviewed **2026-11-18**
never ran. The **2026-11-20 00:00** slot is the first to fire since; at the time of writing its execution row
reads `running` (started `2026-11-20T00:00:08`).
This run therefore audits **three windows** — the standard last 24 h plus the two that were skipped.
## Evidence (live probes 2026-11-20 00:1x–00:4x AWST)
| slot | started | finished / last seen | duration | status |
|---|---|---|---|---|
| 2026-11-18 00:00 | `2026-11-18T00:00:08` | `2026-11-18T00:01:43` (incident `first_seen_at` `00:01:42.342149+08:00`) | ~1 m 35 s | `failed` |
| 2026-11-19 00:00 | `2026-11-19T00:00:16` | `2026-11-19T00:03:02` (incident `last_seen_at` `00:03:01.420319+08:00`) | ~2 m 46 s | `failed` |
| 2026-11-20 00:00 | `2026-11-20T00:00:08` | — | — | `running` |
Both failures carry the **same 314-character `RuntimeError`** — a concatenation of every target the router tried:
```
RuntimeError: command-code/deepseek/deepseek-v4-flash: model — [400]: You have insufficient credits to
make this request. Please purchase more credits to continue using the service. (HTTP 400);
openai-compatible-chat-conn:52258193/deepseek-v4-flash-0731: auth — [401]: invalid auth token (HTTP 401);
openai-compati…
```
The tail is cut off at the 314-char cap, mid-way through a further `openai-compatible-…` target, so the
**final** target's verdict is not recoverable from this field. The shape matters more than the wording: this is
not one bad pin — the **whole fallback chain** was refused, first on **credit exhaustion** (`command-code`),
then on an **invalid auth token** (the second target), then at least one more target whose error is truncated away.
- `cron/jobs.json`: `last_status: error`, `failure_streak: 2` — clean 0 at the 2026-11-17 00:27 run.
- `cron_incidents`: row **`cbfe43_fa12cca1ad4b`** — `state: detected`, `failure_type: auth`,
`first_seen_at 2026-11-18T00:01:42.342149+08:00`, `last_seen_at 2026-11-19T00:03:01.420319+08:00`,
**`alerted_at: None`**, `acked_at: None`, `closed_at: None`, output file
`/home/hermes/.hermes/cron/output/cbfe434a3b5e/2026-11-19_00-03-01.md`.
- `deliveries.db`, window since 2026-11-18: **24 rows — 23 `delivered` (9 of them `for_failure: 1`) and one
`failed`** (`2026-11-19 08:34:13`, a *content* delivery: `live adapter send failed: Timed out; live adapter
delivery to telegram`, consistent with *Gmail Action Digest* `08eb86f02d37` reading `delivery_failed`).
**No row is attributable to either failed audit slot** — both carry `delivery_outcome: suppressed` on the
execution row (as does the clean `2026-11-17` slot), i.e. this job's alerts are not wired to leave the box.
So the outage produced **no push to Anthony** and the red cron row was the only signal.
## Why this matters beyond one job
The wiki audit is this estate's own feedback loop: its output is the `log.md` entry and the page edits that keep
`current-state.md`, `systems/mnemosyne.md` and `systems/scheduled-tasks.md` honest. Two silent skips mean the wiki
stood still for two nights while the estate did not — including the window in which the **Mnemosyne watchdog went
red for 114 consecutive slots** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]. The gap is
visible in `index.md`: its newest feed entry was still **2026-11-16** on the morning of 11-20.
## Precedent — and a repeating pattern
Same class as [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] (**HTTP 429**, *"all targets were
skipped by pre-dispatch filters"*, 6 targets evaluated / **0 attempted**) and
[[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. Two observations worth carrying forward:
1. **The 11-10 failure self-cleared on the next slot** — so a single red night for this job is not necessarily a
standing fault. This one did **not** self-clear: two slots, identical error, ~1.5 and ~2.7 minutes each.
2. **`alerted_at: NULL` on both rows.** The 11-10 row was also left unalerted. Two independent failures of this
job have now produced incident rows that never alerted, which points at the job's failure-*delivery* path
rather than at the alert throttle.
## Not applied / open
- **Nothing was changed**: the job's pin, provider and schedule are untouched, no credential was rotated, no
cron entry was edited.
- No re-run was forced; the 2026-11-20 00:00 slot was left to fire on schedule (it did, and its result is the
entry appended to `log.md` the same night).
- **UNVERIFIED:** whether the credit exhaustion is on the `command-code` account specifically or the router's
shared pool — the concatenated error does not name which target a *successful* dispatch would have used;
basis: the truncated 314-char error string.
- **UNVERIFIED:** whether this job's failures are configured `suppressed`, or whether its failure path is simply
not wired to delivery; basis: absence of delivery rows, not a read of the job's delivery config.
→ [[systems/scheduled-tasks]], [[systems/omniroute]]
+94 -13
View File
File diff suppressed because one or more lines are too long
+39 -8
View File
@@ -3,8 +3,8 @@ title: Backups
type: infrastructure type: infrastructure
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-22 updated: 2026-11-07
verified_on: 2026-10-22 verified_on: 2026-11-07
confidence: medium confidence: medium
tags: [infrastructure, backups] tags: [infrastructure, backups]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -54,22 +54,53 @@ How Hermes's own configuration/state gets backed up, and what's missing.
local snapshot and no off-box copy** — the Gitea mirror below has been dead since 2026-08-08. local snapshot and no off-box copy** — the Gitea mirror below has been dead since 2026-08-08.
→ [[systems/hermes-agent]] → [[systems/hermes-agent]]
## ⚠ Scheduled config backup — the Gitea mirror has been dead since 2026-08-08 ## ✅ Scheduled config backup — RESOLVED 2026-11-07 (was dead since 2026-08-08)
`e27da6c34bbb` (**Hermes Config Backup to Gitea**, daily `0 11 * * *`, `hermes-config-backup.sh`) **commits `e27da6c34bbb` (**Hermes Config Backup to Gitea**, daily `0 11 * * *`, `hermes-config-backup.sh`) **commits
locally every night and fails to push every night — while its cron row still reads `last_status: ok` and its locally every night and fails to push every night — while its cron row still reads `last_status: ok` and its
execution row reads `completed`.** Last commit that reached Gitea: **2026-08-07 11:00:47**; the local repo is execution row reads `completed`.** Last commit that reached Gitea: **2026-08-07 11:00:47**; the local repo is
**76 commits ahead of `origin/main`**. Root cause is a credential: `origin` is **89 commits ahead of `origin/main`** (was 76 on 2026-10-22 — **13 more nights of local-only commits** since). Root cause is a credential: `origin` is
`https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` — HTTPS, no userinfo, no credential `https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` — HTTPS, no userinfo, no credential
helper — so the non-interactive push dies on helper — so the non-interactive push dies on
`fatal: could not read Username … No such device or address` → `✗ Push to Gitea FAILED` → `fatal: could not read Username … No such device or address` → `✗ Push to Gitea FAILED` →
`--- Backup complete: FAIL ---`. Full evidence and the proposed fix: `--- Backup complete: FAIL ---`. Full evidence and the proposed fix:
[[incidents/2026-08-08-gitea-config-backup-push-dead]]. [[incidents/2026-08-08-gitea-config-backup-push-dead]].
**Consequence for this page: everything above describes a topology that has not carried `~/.hermes` off the box > **RESOLVED 2026-11-07.** The 92-commit backlog is drained and pushed to Gitea, and the script has been
since 2026-08-07.** The 2026-07-22 full zip and the wiki/authoring repos remain the only recent off-box copies. > hardened so the cron row can no longer report a false green. Full record:
Telegram failure notices *have* been firing daily (last: 2026-10-21 11:00:14, `message_id 8449`); the AgentMail > [[incidents/2026-08-08-gitea-config-backup-push-dead#Resolution 2026-11-07]].
notification step failed.
**What was wrong (four independent causes):** (1) no Gitea credential — `origin` was HTTPS and
`~/.git-credentials` held only a `github.com` entry, so the non-interactive push could not authenticate;
(2) **`backup.sh` had no `exit` statement at all** — it always returned 0, so the scheduler's
`success = (returncode == 0)` recorded `ok` on every failed night (the scheduler was correct; the script
was the liar); (3) a 1.76 GB `checkpoints/` backlog sat in the unpushed range, 13 blobs over 100 MB;
(4) both secondary legs were broken — the `github` remote did not exist, and its fallback pushed a
branch's full ancestry containing a 7.45 GB tarball blob that GitHub's 100 MB limit could never accept.
**What was done:** host-scoped Gitea credential stored via `git credential approve`; `checkpoints/`
stripped from the unpushed range with `git filter-repo --partial` (push set ~2 GB → 264.8 MB, pre-rewrite
history preserved under tag `backup/pre-rewrite-20261107-032658`); the script rewritten to exit non-zero on
failure, stamp `.last_backup_ok` only after remote verification, decide the push by **remote-vs-local**
rather than by working-tree changes, guard every `ls-remote` with `|| true` so an unreachable remote still
reaches the alert path, push GitHub as a **parentless orphan snapshot**, and report Telegram/AgentMail
delivery honestly (`curl --fail` + `"ok":true`; real `agentmail` CLI).
**Re-verified 2026-11-07 — live and green.** Real run exit **0** in 47 s; `git rev-parse HEAD` and
`git ls-remote origin refs/heads/main` both `a50a5170c1c0e2af5235fa63a95a732e2d780703`;
`rev-list --count origin/main..main` = **0**; Gitea API confirms `main` at `a50a5170c1c0`, updated
`2026-11-07T05:10:46+08:00`; `.last_backup_ok` refreshed (was stale from 2026-11-06 11:00). GitHub mirror
now carries a parentless orphan snapshot (`faed9fb5c677`, 0 parents, 30,086 files, largest blob 23.5 MB) —
no data lost, since GitHub's tree was a strict subset of local HEAD. The AgentMail and Telegram alert
paths were each exercised directly and report honestly.
**Consequence for this page: the topology above carries `~/.hermes` off the box again.** The one full
`hermes backup` zip on disk is still `/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, 22 Jul) —
**now 108 days old**, and no new full zip has been taken. `crontab -l -u hermes` is still empty and there
are still **0 user systemd timers**, so nothing schedules the full zip; only `e5ed215fc91c`
(**Mnemosyne DB Backup**, weekly `0 3 * * 0`) and `e27da6c34bbb` (config backup, daily) exist in the cron
fleet, and neither takes a full `hermes backup`. The nightly config backup is a *config* mirror, not a
full-home backup — the gap on the full zip is unchanged.
## Relevant runbooks ## Relevant runbooks
- [[runbooks/backup-wiki]] - [[runbooks/backup-wiki]]
+10 -2
View File
@@ -3,8 +3,8 @@ title: Credentials Map
type: infrastructure type: infrastructure
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-11-11
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: medium confidence: medium
tags: [credentials, security, infrastructure] tags: [credentials, security, infrastructure]
sources: [] sources: []
@@ -45,6 +45,14 @@ Document where credentials are stored and what service they belong to.
> No `connectors:` block is configured in `config.yaml` at all. > No `connectors:` block is configured in `config.yaml` at all.
> `~/.hermes/auth.json` also exists and holds OAuth tokens — treat as a credential store. > `~/.hermes/auth.json` also exists and holds OAuth tokens — treat as a credential store.
> **Re-verified 2026-11-11 — both 2026-10-20 corrections still hold.** `config.yaml` still has **no**
> `imap`/`smtp`/`email`/`connectors`/`gmail`/`mattermost` keys; the email credentials still live in
> `~/.hermes/.env` (mode `0600`, keys `EMAIL_ADDRESS`, `EMAIL_IMAP_HOST`, `EMAIL_SMTP_HOST`,
> `EMAIL_ALLOWED_USERS`, `EMAIL_HOME_ADDRESS`, `EMAIL_ALLOW_ALL_USERS`, `EMAIL_POLL_INTERVAL` and
> more). `tools:` still contains only `tool_search`. `discord:` and `telegram:` are top-level
> blocks, and `platforms:` still lists `mattermost` (so Mattermost remains a plugin, not a `tools:`
> entry). `~/.hermes/auth.json` exists and holds OAuth material.
## Rules ## Rules
- Credentials never written to this wiki - Credentials never written to this wiki
- API vault is first-stop for any key lookup - API vault is first-stop for any key lookup
+88 -14
View File
@@ -3,8 +3,8 @@ title: Docker Services
type: infrastructure type: infrastructure
status: mixed status: mixed
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-11 updated: 2026-11-07
verified_on: 2026-10-11 verified_on: 2026-11-07
confidence: high confidence: high
tags: [docker, homelab, infrastructure] tags: [docker, homelab, infrastructure]
sources: [raw/configs/hermes-config-sanitized.txt, live-probe] sources: [raw/configs/hermes-config-sanitized.txt, live-probe]
@@ -16,32 +16,101 @@ sources: [raw/configs/hermes-config-sanitized.txt, live-probe]
Containerized services supporting Hermes and the homelab. Containerized services supporting Hermes and the homelab.
## Status ## Status
Docker runs on **three hosts**, and CT460's socket is not readable by the agent user. This page previously claimed "Docker runtime not available" — that is only true for CT460's socket from the `hermes` user. Real state, verified 2026-10-11: Docker runs on **four hosts**, and CT460's socket is not readable by the agent user. This page previously claimed "Docker runtime not available" — that is only true for CT460's socket from the `hermes` user. Real state, **re-verified 2026-11-01**:
| Host | Access | Containers running | | Host | Access | Containers running |
|---|---|---| |---|---|---|
| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **~50** — main Docker host | | CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **65** running / 72 total — main Docker host (**re-verified 2026-11-15**), **plus 0–2 transient `worldmonitor-seeder-*` containers** driven by systemd timers (see the note below) |
| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | 9 (steel-browser, gotify, readlater, kokoro-tts, edge-tts, onecli-postgres, …) | | CT201 `runtipi` (`100.96.244.39`) | `pct exec 201 -- docker ps` via PVE | **20** running / 26 total — Runtipi stack: `audiobookshelf`, `calibre-web`, `code-server`, `cyberchef`, `duplicati`, `filebrowser`, `flaresolverr`, `gotify`, `homarr`, `homebox`, `homepage-manager`, `it-tools`, `linkwarden`, `linkwarden-db`, `mealie-1`, `ntfy`, `runtipi`, `runtipi-db`, `syncthing`, `transmission` (**re-verified 2026-11-07** — set-wise diff against live `docker ps`: no additions, no removals) |
| CT460 `hermes` | `sudo docker` — socket is `root:docker srw-rw----`, `hermes` user not in group `docker`; **no passwordless sudo in cron** | `ignis` (confirmed running; inspect requires root) | | CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** running / 9 total (**re-verified 2026-11-07**) — `nostalgic_keldysh`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29**; `thirsty_payne` → `nostalgic_leavitt` (2026-11-02) → `flamboyant_mahavira` (2026-11-04) → **`nostalgic_keldysh`** (2026-11-06) — all `ghcr.io/github/github-mcp-server`, the live one created 2026-11-05T22:15:57Z. ⚠ **Corrected 2026-11-07:** the enumerated list still opened with `flamboyant_mahavira`, which is **not running** — live `docker ps` on CT333 returns **`nostalgic_keldysh`** (created 2026-11-05 22:15:57, `Up 29 hours`), `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1` (8 running), plus `qdrant` **exited** (9 total) |
| CT1001 `docker-01` (`100.121.12.31`, **on `m93p`**) | `pct exec 1001 -- docker ps` via `m93p` | **1** running / 9 total — `sshwifty`. **Added 2026-10-31** — a fourth Docker host, previously undocumented |
| CT460 `hermes` | `pct exec 460 -- docker ps` **as root** from the PVE host — socket is `root:docker srw-rw----`, the `hermes` user is not in group `docker` (its own `docker ps` → `permission denied while trying to connect to the docker API`) | `ignis` — **running** (`Up 5 days`, `ignis-subpath:0.8.9`, `0.0.0.0:8080->8080/tcp`), re-verified 2026-11-05 |
## CT200 live inventory (verified 2026-10-11, ~50 containers) ## CT200 live inventory (re-verified 2026-11-15, 65 running / 72 total, excluding 1 transient `worldmonitor-seeder-*` container)
`guacamole-guacamole-1`, `telegram-bot-api`, `ytdlp-web-player`, `homarr`, `overtchat-searxng`, `searxng`, `khoj-search-1`, `prowlarr`, `overtchat-app`, `postgres-newsku`, `guacamole-postgres-1`, `jellyfin`, `libre-closet`, `apprise-api`, `tt-rss-web-nginx-1`, `tt-rss-updater-1`, `tt-rss-app-1`, `tt-rss-db-1`, `audiobookshelf`, `qbittorrent`, `open-webui`, `readlater`, `readlater-extraction-worker`, `worldmonitor-redis`, `redis-cache`, `nextstep-db`, `nextstep-scheduler-1`, `nextstep-app`, `portainer`, `flaresolverr`, `sshwifty`, `8mblocal`, `save-restricted-bot`, `kokoro-tts`, `edge-tts`, `filebrowser`, `filebrowser1`, `worldmonitor-redis-rest`, `worldmonitor`, `overtchat-redis`, `chromadb`, **`ignis-ignis-1`**, `nc_app_talk_bot_ai`, `excalidraw-frontend`, `nc_app_context_agent`, `portkey-gateway-web-1`, `guacamole-guacd-1`, `lobe-chat`, `mycompose-saas-1`. `8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, **`phonon-demo`**, **`ptts-openai-server`**, **`ptts-ws-server`**, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`.
**Watch this one:** `mycompose-db-1` is in `Restarting (1)` — a crash-looping database. Flagged for follow-up; root cause not investigated on 2026-10-11. **Re-verified 2026-11-15 00:4x AWST — 61 → 64 running (excluding the seeder), 68 → 72 total, and the whole delta is the TTS stack.** Live `docker ps` on CT200 returns **65 running / 72 total**, which is **64 non-seeder running + 1** transient `worldmonitor-seeder-scheduled`. A set-wise diff of the live names against this page's 2026-11-07 list found **three additions and no removals**, and the arithmetic closes exactly: **61 non-seeder → 64 non-seeder** is the **`phonon-demo`**, **`ptts-openai-server`** and **`ptts-ws-server`** trio — all created 2026-11-14, documented on [[systems/phonon-tts]]. Earlier in this pass a generic `total − prior total` subtraction suggested a fourth, unnamed addition; that was a **comparison artefact** (the 11-07 row's 61 was an explicit *non-seeder* count, today's 65 was a raw `running` count including the seeder) and is **withdrawn** — the name-level diff is exact and accounts for every added container. This pass verified **counts, the TTS trio, and the absence of removals**; it is not a full CT200 audit.
**Re-verified 2026-11-06 — the 2026-11-05 "the seeder does not exist" finding was WRONG, and the cause is a
sampling artefact, not a removal.** A set-wise diff against live `docker ps --format '{{.Names}}'` on CT200
shows **no container live but absent from the list above**, and **no listed container has disappeared** — the
61-container running set above still matches live exactly, set-wise. The `worldmonitor-seeder-*` containers are
**transient and systemd-timer driven**, so they are only visible while a seed run is in flight: at the
2026-11-06 03:1x probe **both** were present (`worldmonitor-seeder-scheduled`, created 02:45:14, `Up 32 minutes
(unhealthy)`, and `worldmonitor-seeder-fast`, created 03:17:15, `Up 22 seconds (health: starting)`); at the
2026-11-05 05:1x probe **neither** was, because no run was in flight — hence the zero `seeder` matches. The
drivers are **systemd timers on CT200, not cron**: `worldmonitor-seed.timer` (`OnUnitInactiveSec=4h`,
`RandomizedDelaySec=10min` → `worldmonitor-seed.service` → `docker run --rm --name
worldmonitor-seeder-scheduled … scripts/run-seeders.sh`) and `worldmonitor-seed-fast.timer`
(`OnUnitInactiveSec=15min`, `RandomizedDelaySec=1min` → `worldmonitor-seed-fast.service` → `--name
worldmonitor-seeder-fast`). Both services were `ActiveState=activating` at the probe. The **corrected** claim
is therefore: **the seeders exist and run on a 4h + 15min schedule; a point-in-time `docker ps` can miss them
entirely, so "absent" must never be read as "removed".** The **corrected on record** note: the earlier
`worldmonitor-seeder-fast` → `worldmonitor-seeder-scheduled` rename narrative was also wrong — there is **one
script with two timer-driven invocations** under two container names, not a rename. Live CT200 also shows
`worldmonitor` (`Up 6 days`, **healthy**), `worldmonitor-redis` (`Up 6 days`) and `worldmonitor-redis-rest`
(`Up 6 days`).
**Re-verified 2026-11-05 — 61 → 61 running, seeder entry removed.** A set-wise diff against live
`docker ps --format '{{.Names}}'` on CT200 shows **no container live but absent from the list above**, and
**no listed container has disappeared**. ~~The `worldmonitor-seeder-scheduled` entry recorded on 2026-11-04 as
"unhealthy" **does not exist on any host** — verified with `docker ps -a --filter name=seeder` across CT200,
CT201, CT333 and CT460, all of which return **zero** matches.~~ **⚠ SUPERSEDED 2026-11-06 — see the
correction above: this was a sampling artefact. The seeders are transient systemd-timer-driven containers,
not absent.** ~~There is **no seeder container and no worldmonitor-seeder cron job** anywhere in the estate —
the 2026-11-04 "watch item" was a transient container that has since been removed.~~ **⚠ SUPERSEDED
2026-11-06: the seeders DO exist; the driver is a systemd timer pair, not a cron job.** ~~**Correction on
record:** the earlier `worldmonitor-seeder-fast` → `worldmonitor-seeder-scheduled` rename narrative did not
reflect a durable redeploy; the seeder is simply absent now.~~ **⚠ SUPERSEDED 2026-11-06: one script, two
timer-driven container names — not a rename.**
**CT333 container set — `nostalgic_leavitt` → `flamboyant_mahavira` (2026-11-05) → `nostalgic_keldysh` (2026-11-06).**
Live CT333 `docker ps` returns 8 running containers, and the name that had replaced `thirsty_payne` on 2026-11-04
and then `flamboyant_mahavira` on 2026-11-05 has **itself been replaced again**:
**`nostalgic_keldysh`** (image `ghcr.io/github/github-mcp-server`, **created 2026-11-05T22:15:57Z**, `Up 5 hours`
at the 2026-11-06 probe). The count is unchanged at 8 and the other seven names are identical
(`community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`,
`onecli-postgres-1`). A ninth container, `qdrant`
(`qdrant/qdrant`), exists but is **stopped** (`Exited (143) 4 months ago`) — it is not in the running count and
is not listed above. **Docker generates these random names for compose-less containers, so repeated renames are
normal — the reliable check is always the live set, never the recorded name.**
**CT460 `ignis` access path corrected (2026-11-04).** The old row said "no passwordless sudo in cron", which
implied the container list could not be read at all. It can: **`pct exec 460 -- docker ps` from the PVE host
runs as root and works** (`ignis|ignis-subpath:0.8.9|Up 4 days|0.0.0.0:8080->8080/tcp`). What does *not* work is
the `hermes` user's own `docker ps` inside CT460 — `permission denied while trying to connect to the docker API
at unix:///var/run/docker.sock` — because the socket is `root:docker srw-rw----` and `hermes` is not in group
`docker`. Use the root-via-PVE path.
**Re-verified 2026-11-01:** the 2026-10-11 inventory listed **49** containers; the live count is now **61**.
**Corrected 2026-11-01:** the inventory list had three **misspelled** `overtchat-*` entries (written
`overchat-*`) and one **stale** entry, `worldmonitor-seeder-fast`, which is **not running** — live
`docker ps -a --filter name=worldmonitor` shows only `worldmonitor`, `worldmonitor-redis` and
`worldmonitor-redis-rest` (all `Up 35 hours`). The list above is regenerated verbatim from live
`docker ps --format '{{.Names}}'`. **CT201's Tailscale IP was also wrong** — it read
`100.116.116.116`; live `pct exec 201 -- tailscale ip -4` returns **`100.96.244.39`** (which is
also what `[[infrastructure/hosts]]` already recorded). **CT333's first container was renamed** —
`adoring_jemison` → **`thirsty_payne`**.
**12 containers were added** since: `emulatorjs`, `excalidraw-backend`, `khoj-database-1`, `khoj-sandbox-1`,
`khoj-server-1`, `mycompose-db-1`, `onlyoffice-docserver`, `quickdrop`, `react-resume-client-1`,
`secret-santa-app`, `tgvideo-bot`, `watchtower`. **No container listed on the 2026-10-11 page has
disappeared** — the inventory is a strict superset.
**✅ `mycompose-db-1` crash-loop is FIXED.** The 2026-10-11 page flagged it as `Restarting (1)` — a
crash-looping database. Live `docker ps -a --filter name=mycompose` now shows **both** `mycompose-db-1`
and `mycompose-saas-1` as **`Up 35 hours`** (checked 2026-11-01). No longer a watch item.
**Note the duplicate Ignis:** `ignis-ignis-1` on CT200 is a separate deployment from CT460's `ignis`. See [[systems/ignis]]. **Note the duplicate Ignis:** `ignis-ignis-1` on CT200 is a separate deployment from CT460's `ignis`. See [[systems/ignis]].
## CT201 runtipi — Tailscale routes (verified 2026-10-11) ## CT201 runtipi — Tailscale routes (verified 2026-10-11; Steel routes removed 2026-10-29)
| Host | Path | Target | | Host | Path | Target |
|---|---|---| |---|---|---|
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet) | `/` | `http://127.0.0.1:8085` | | `runtipi.kangaroo-eel.ts.net:8443` (tailnet) | `/` | `http://127.0.0.1:8085` |
| | `/mgr` | `http://127.0.0.1:8084` | | | `/mgr` | `http://127.0.0.1:8084` |
| `runtipi.kangaroo-eel.ts.net` (tailnet) | `/` | `http://127.0.0.1:8085` | | `runtipi.kangaroo-eel.ts.net` (tailnet) | `/` | `http://127.0.0.1:8085` |
| | `/mgr` | `http://127.0.0.1:8084` | | | `/mgr` | `http://127.0.0.1:8084` |
| | `/steel` | `http://127.0.0.1:5173` |
| | `/steel-api` | `http://127.0.0.1:3000` |
**Correction:** this page previously said "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there" and listed CT201's `:8443` → `localhost:8082` for Homepage. Live `tailscale serve status` on CT201 shows the target is `127.0.0.1:8085` on both `:8443` and the default host, plus a `/steel` (5173) and `/steel-api` (3000) pair. The `/steel-api` route does not strip the path prefix — use the container IP directly for API calls. **Correction:** this page previously said "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there" and listed CT201's `:8443` → `localhost:8082` for Homepage. Live `tailscale serve status` on CT201 shows the target is `127.0.0.1:8085` on both `:8443` and the default host. That table also carried a `/steel` (5173) and `/steel-api` (3000) pair — **both routes were deleted on 2026-10-29** with `tailscale serve --set-path <path> off` when Steel was removed, so CT201's remaining routes are only `/` and `/mgr` on **both** `:443` and `:8443`. (The `/steel-api` route never stripped the path prefix — moot now that it is gone.)
## Confirmed Compose Files ## Confirmed Compose Files
@@ -50,7 +119,12 @@ Docker runs on **three hosts**, and CT460's socket is not readable by the agent
| Hermes Agent | `/home/hermes/.hermes/hermes-agent/docker-compose.yml` | hermes, hermes-dashboard | | Hermes Agent | `/home/hermes/.hermes/hermes-agent/docker-compose.yml` | hermes, hermes-dashboard |
| Hermes WebUI | `/home/hermes/hermes-webui/docker-compose.yml` | webui | | Hermes WebUI | `/home/hermes/hermes-webui/docker-compose.yml` | webui |
| ReadLater | `/home/hermes/workspace/readlater/docker-compose.yml` | readlater, edge-tts, kokoro-tts | | ReadLater | `/home/hermes/workspace/readlater/docker-compose.yml` | readlater, edge-tts, kokoro-tts |
| Steel Browser | `/root/steel-browser/docker-compose.yml` (CT201) | steel-browser-api, steel-browser-ui | | Ignis | `/home/hermes/docker/ignis/docker-compose.yml` (CT460) | ignis, ignis-data, ignis-obsidian-app |
**Steel Browser — removed 2026-10-29.** The CT201 compose file and Dockerfile are no longer live: the project
dir `/root/steel-browser` was deleted along with its containers, images and network. Both files are archived at
`/root/_archived-steel/` on CT201 (with `/root/_archived-steel/steel-cache-archive-20261029.tgz`), and the
images remain re-pullable from ghcr.io. → [[comparisons/steel-vs-camoufox]]
## Service Definitions (from compose files) ## Service Definitions (from compose files)
+8 -2
View File
@@ -3,8 +3,8 @@ title: Domains and Tunnels
type: infrastructure type: infrastructure
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-20 updated: 2026-11-11
verified_on: 2026-10-20 verified_on: 2026-11-11
confidence: high confidence: high
tags: [infrastructure, network, cloudflare] tags: [infrastructure, network, cloudflare]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -59,6 +59,12 @@ The catch-all remains `service: http_status:404`. The ct207 `config.yml` still c
## Full ct207 ingress list (context, not all Hermes-related) ## Full ct207 ingress list (context, not all Hermes-related)
The tunnel also fronts: `pve.martinwa.org` (Proxmox UI), `docker.martinwa.org` (Portainer), `runtipi.martinwa.org`, `homar.martinwa.org`, `nextcloud.martinwa.org`, `readlater.martinwa.org`, `velvet.martinwa.org`, `overtchat.martinwa.org`, `freellmapi.martinwa.org`, `omnideb.martinwa.org`, `newstarter.martinwa.org`, and SSH endpoints for debianvm/pve/webserver. These belong to the wider homelab, not Hermes specifically — recorded here only so an ingress diff is possible later. The tunnel also fronts: `pve.martinwa.org` (Proxmox UI), `docker.martinwa.org` (Portainer), `runtipi.martinwa.org`, `homar.martinwa.org`, `nextcloud.martinwa.org`, `readlater.martinwa.org`, `velvet.martinwa.org`, `overtchat.martinwa.org`, `freellmapi.martinwa.org`, `omnideb.martinwa.org`, `newstarter.martinwa.org`, and SSH endpoints for debianvm/pve/webserver. These belong to the wider homelab, not Hermes specifically — recorded here only so an ingress diff is possible later.
> **Full live re-read of ct207 `/etc/cloudflared/config.yml` on 2026-11-11 — the list is unchanged
> from the 2026-10-20 diff.** All ten additions/corrections recorded above are present and no new
> hostname has appeared; the catch-all is still `service: http_status:404`. ct207's LAN address is
> `192.168.178.132`, consistent with the `ssh-webserver.martinwa.org` → `100.82.40.23` entry.
> `mcp-portal.martinwa.org` remains **absent** from the ingress list — still unknown, still flagged.
## History ## History
- ct207's `/etc/cloudflared/` directory has ~25 dated `config.yml.bak*` files going back to 2026-06-18, including several during ct460's own setup (`config.yml.before-hermes-ct460-20260717203530`, `...before-hermes-desktop-20260718`, `...before-hermesagent-20260718085557`) — evidence the ingress list has been actively iterated on throughout Hermes's bring-up. - ct207's `/etc/cloudflared/` directory has ~25 dated `config.yml.bak*` files going back to 2026-06-18, including several during ct460's own setup (`config.yml.before-hermes-ct460-20260717203530`, `...before-hermes-desktop-20260718`, `...before-hermesagent-20260718085557`) — evidence the ingress list has been actively iterated on throughout Hermes's bring-up.
+16 -11
View File
@@ -3,8 +3,8 @@ title: Known Hosts and Paths
type: infrastructure type: infrastructure
status: partial status: partial
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-26 updated: 2026-11-07
verified_on: 2026-10-26 verified_on: 2026-11-07
confidence: medium confidence: medium
tags: [infrastructure, hosts, containers, paths] tags: [infrastructure, hosts, containers, paths]
sources: [] sources: []
@@ -20,11 +20,11 @@ sources: []
| litellm | LiteLLM proxy | `http://litellm:4000/v1` | Yes (from config) | | litellm | LiteLLM proxy | `http://litellm:4000/v1` | Yes (from config) |
| `custom:Omniroute:20128` | OmniRoute proxy (CT600 node `omniroute`) | `http://omniroute:20128/v1` → `100.88.81.19:20128` | Yes (live probe 2026-10-26: 6,480 models) | | `custom:Omniroute:20128` | OmniRoute proxy (CT600 node `omniroute`) | `http://omniroute:20128/v1` → `100.88.81.19:20128` | Yes (live probe 2026-10-26: 6,480 models) |
| `custom:omni-101` | OmniRoute proxy (CT101 node `omniroute-1`) — **the live default provider** | `http://100.78.63.20:20128` | Yes (live probe 2026-10-26) | | `custom:omni-101` | OmniRoute proxy (CT101 node `omniroute-1`) — **the live default provider** | `http://100.78.63.20:20128` | Yes (live probe 2026-10-26) |
| CT201 | Steel Browser, Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.166` (LAN) | Yes (live probe) | | CT201 | Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.77` (LAN) | Yes (live probe 2026-10-30 — LAN address corrected from `192.168.178.166`) |
| CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net`, `100.118.5.51` (Tailscale) | Yes | | CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net`, `100.118.5.51` (Tailscale) | Yes |
| PVE `pve` | Proxmox VE hypervisor hosting CT460 and the other `pve` guests | `100.96.100.82` (Tailscale), `pve.kangaroo-eel.ts.net` | Yes (live `pct list` 2026-10-26) | | PVE `pve` | Proxmox VE hypervisor hosting CT460 and the other `pve` guests | `100.96.100.82` (Tailscale), `pve.kangaroo-eel.ts.net` | Yes (live `pct list` 2026-11-07 — 17 LXCs + 4 VMs) |
| PVE `m93p` | Second, unclustered Proxmox VE 9.2.10 hypervisor | `100.104.51.98` (Tailscale) | Yes (live `pct list` 2026-10-26) | | PVE `m93p` | Second, unclustered Proxmox VE 9.2.10 hypervisor | `100.104.51.98` (Tailscale) | Yes (live `pct list` 2026-11-07 — 20 LXCs + 2 VMs) |
| CT450 (hermesagent) | Hermes Agent **v0.22.3** (2026.9.24), local `5b147ab036d6` — same build as CT460, hermes-webui, hermes-ui (Vite) | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes | | CT450 (hermesagent) | Hermes Agent **v0.22.5** (2026.9.24), local `bf0203a794a0` — **same build as CT460**; hermes-webui, hermes-ui (Vite). ⚠ a stale **root-level** `/usr/local/bin/hermes` on CT450 still reports **v0.19.0** — the live agent is the `hermes`-user install under `~/.hermes/hermes-agent` | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes (live probe 2026-10-30 — version corrected from `v0.22.3`, build from `5b147ab036d6` to `bf0203a794a0`) |
## Project Paths ## Project Paths
@@ -34,8 +34,8 @@ sources: []
| `/home/hermes/workspace` | Workspace | default terminal cwd | | `/home/hermes/workspace` | Workspace | default terminal cwd |
| `/home/hermes/.local/bin/hermes` | Hermes binary | | | `/home/hermes/.local/bin/hermes` | Hermes binary | |
| `/home/hermes/.hermes/config.yaml` | Config | `_config_version: 51` | | `/home/hermes/.hermes/config.yaml` | Config | `_config_version: 51` |
| `/home/hermes/.hermes/skills/` | Skills dir | 302 skills across 69 categories (`.archive/` excluded) | | `/home/hermes/.hermes/skills/` | Skills dir | **302 skills across 68 categories** (`.archive/` excluded — 46 archived; re-verified 2026-11-07) |
| `/home/hermes/.hermes/cron/` | Cron jobs | 28 jobs — 25 enabled / 3 disabled | | `/home/hermes/.hermes/cron/` | Cron jobs | **29 jobs — 26 enabled / 3 disabled** (re-verified 2026-10-31 00:3x). ⚠ Superseded: the 2026-10-30 03:3x reading was 25 / 4 — the 4th disabled job, `ea79a2107e36` Mnemosyne Health Monitor, was auto-paused 2026-10-30 01:04 after a gateway-connection failure, but **that pause lasted only until 10:02:48**, when its long-running slot closed `completed`, incident `ea79a2_d93c3acdc4f2` auto-closed `resolved`, and the job returned to `enabled: true` / `last_status: ok` / streak 0. **CT460 itself was then dark for 5 h 30 m on 2026-10-30** (PVE `pve` rebooted; host boot 16:24:00, CT460 16:28:00, scheduler resumed 16:37:21 — ≈39 cron slots lost, 0 failures) → [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]] |
| `/home/hermes/.hermes/mnemosyne/` | Mnemosyne DB | backups/, data/, logs/, models/ | | `/home/hermes/.hermes/mnemosyne/` | Mnemosyne DB | backups/, data/, logs/, models/ |
| `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | cua-driver | v0.9.0 | | `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | cua-driver | v0.9.0 |
| `~/.camofox/` | Camofox profiles | managed persistence | | `~/.camofox/` | Camofox profiles | managed persistence |
@@ -59,15 +59,20 @@ sources: []
- `/home/hermes/workspace/readlater/docker-compose.yml` - `/home/hermes/workspace/readlater/docker-compose.yml`
## Homelab References ## Homelab References
- **PVE host `pve`:** `pve.kangaroo-eel.ts.net` (100.96.100.82) — **verified** via live `pct list` 2026-10-26 (17 LXCs) - **PVE host `pve`:** `pve.kangaroo-eel.ts.net` (100.96.100.82) — **verified** via live `pct list` 2026-11-07 (**17 LXCs** + 4 VMs; boot 2026-10-30 16:24:00)
- **PVE host `m93p`:** 100.104.51.98, Proxmox VE 9.2.10 — **verified** via live `pct list` 2026-10-26 (19 LXCs). Guest IDs collide between the two hosts - **PVE host `m93p`:** 100.104.51.98, Proxmox VE 9.2.10 — **verified** via live `pct list` 2026-11-07 (**20 LXCs** + 2 VMs; boot 2026-11-05 21:19:52). ⚠ **Corrected 2026-11-07:** the previous figure of 19 LXCs omitted **CT107 (`mem0`)**, which is running — see [[infrastructure/proxmox]]. Guest IDs collide between the two hosts
- **CT600 (`omniroute`):** OmniRoute server, `100.88.81.19` — **running and live**; `omniroute` is the hostname that resolves to it - **CT600 (`omniroute`):** OmniRoute server, `100.88.81.19` — **running and live**; `omniroute` is the hostname that resolves to it
- **CT601 (`omni`):** the old CT601 OmniRoute guest **no longer exists** on either hypervisor (`pct config 601` → no such config). The `omni` Tailscale peer (100.93.204.4) is its orphaned node, offline 69d - **CT601 (`omni`):** the old CT601 OmniRoute guest **no longer exists** on either hypervisor (`pct config 601` → no such config on both `pve` and `m93p`). The orphaned `omni` Tailscale peer (`100.93.204.4`) has since **expired out of the tailnet entirely** — as of the 2026-10-30 live probe the tailnet contains **no** peer named `omni` and no peer at `100.93.204.4`, so the old "offline 69d" description no longer applies. The only OmniRoute nodes left are CT600 (`omniroute`) and CT101 (`omniroute-1`).
- **CT101 (`omniroute`, m93p):** second OmniRoute node, `100.78.63.20` — running - **CT101 (`omniroute`, m93p):** second OmniRoute node, `100.78.63.20` — running
## Unverified ## Unverified
- Individual LXCs are verified only when probed directly; the two `pct list` snapshots above are the current authority for what is running - Individual LXCs are verified only when probed directly; the two `pct list` snapshots above are the current authority for what is running
- Guests not present in those `pct list` outputs are **not** running on either hypervisor — do not assume existence from older session notes - Guests not present in those `pct list` outputs are **not** running on either hypervisor — do not assume existence from older session notes
> **Note (2026-10-29):** CT201 no longer hosts the Steel Browser — the containers, images, network, project dir
> and Tailscale Serve routes were all removed when Steel was deleted from the estate. CT333's separate,
> orphaned Steel copy was removed at the same time. → [[comparisons/steel-vs-camoufox]]
## Related ## Related
- [[docker-services]] - [[docker-services]]
- [[infrastructure/tailscale]]
+49 -15
View File
@@ -3,11 +3,11 @@ title: Network Map
type: infrastructure type: infrastructure
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-14 updated: 2026-11-07
verified_on: 2026-10-14 verified_on: 2026-11-07
confidence: high confidence: high
tags: [infrastructure, network] tags: [infrastructure, network]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt, live-probe:ct460-ss+tailscale-serve-status-2026-11-05]
--- ---
# Network Map # Network Map
@@ -22,11 +22,13 @@ Where Hermes and its dependencies run, and how traffic reaches them.
- **This container (hermes, ct460):** `192.168.178.129` (DHCP, eth0) - **This container (hermes, ct460):** `192.168.178.129` (DHCP, eth0)
### Tailscale (`kangaroo-eel.ts.net`) ### Tailscale (`kangaroo-eel.ts.net`)
_Peer IPs below re-checked 2026-11-07 against `tailscale status` on ct460; `webserver`, `clawtest`, `hermesagent`, `docker`, `omniroute` all matched the values listed. Full 52-row node table with live states lives in [[infrastructure/tailscale]]._
- **pve:** `100.96.100.82` - **pve:** `100.96.100.82`
- **m93p:** `100.104.51.98` — **second, unclustered Proxmox hypervisor (PVE 9.2.0), discovered 2026-10-14.** See [[infrastructure/proxmox]]. Guest IDs collide with `pve`'s, so always qualify a guest with its host. - **m93p:** `100.104.51.98` — **second, unclustered Proxmox hypervisor (PVE 9.2.10), discovered 2026-10-14.** ⚠ Corrected 2026-11-07: this line still read *"9.2.0"* although both hosts run the identical `pve-manager 9.2.10` build (live `dpkg -l pve-manager` on both). See [[infrastructure/proxmox]]. Guest IDs collide with `pve`'s, so always qualify a guest with its host.
- **hermes (ct460):** `100.118.5.51` - **hermes (ct460):** `100.118.5.51`
- **hermesagent (ct450, the original/legacy Hermes agent dashboard):** `100.98.205.112` - **hermesagent (ct450, the original/legacy Hermes agent dashboard):** `100.98.205.112`
- **omniroute (ct600):** resolves via `omniroute.kangaroo-eel.ts.net` → `100.88.81.19` - **omniroute (ct600):** resolves via `omniroute.kangaroo-eel.ts.net` → `100.88.81.19`
- **omniroute-1 (third OmniRoute node):** `100.78.63.20` (`omniroute-1.kangaroo-eel.ts.net`) — added 2026-11-05.
- **nanobot (ct333, "clawtest"):** `100.122.189.81` - **nanobot (ct333, "clawtest"):** `100.122.189.81`
- **webserver (ct207, runs the Cloudflare Tunnel):** `100.82.40.23` - **webserver (ct207, runs the Cloudflare Tunnel):** `100.82.40.23`
- **debianvm (VM 101):** `100.98.95.65` - **debianvm (VM 101):** `100.98.95.65`
@@ -42,19 +44,51 @@ Where Hermes and its dependencies run, and how traffic reaches them.
| Port | Bind | Process | Purpose | | Port | Bind | Process | Purpose |
|---|---|---|---| |---|---|---|---|
| 22 | `*` | sshd | SSH | | 22 | `*` | sshd | SSH |
| 80 / [::]:80 | `*` | nginx | reverse proxy, default vhost | | 80 | `*` | nginx | reverse proxy, default vhost |
| 19119 | `*` | nginx → 127.0.0.1:9119 | public Hermes webui path (tunneled) | | 443 / 8443 / 9443 | `100.118.5.51` (tailscale) | tailscaled | Tailscale Serve/Funnel listeners — route table below |
| 9119 | `0.0.0.0` | hermes (webui) | Hermes Web UI backend | | 19119 | `*` | nginx → 127.0.0.1:9119 | public Hermes webui path (Cloudflare-tunneled) |
| 8644 / 8642 | `*` | hermes | additional Hermes agent ports | | 9119 | `0.0.0.0` | `hermes-webui/server.py` | Hermes Web UI backend (Serve `/`) |
| 5174 | `*` | node | Hermes desktop companion (tunneled as hermes-desktop.martinwa.org) | | 8644 / 8642 | `*` | hermes (gateway) | additional Hermes agent ports |
| 9377 | `*` | node | Stratos — AI Command Centre | | 9900 | `0.0.0.0` | hermes (gateway) | gateway aux port (up 2026-11-05) |
| 8787 | `127.0.0.1`* | python | Hermes gateway health endpoint (see [[runbooks/update-hermes-safely]]) | | 5174 | — | *(no listener)* | desktop companion port listed up to 2026-10-14; **not listening 2026-11-05** |
| 9099 / 8790 | `*` | python3 | auxiliary Hermes processes (unverified purpose) | | 5173 | `0.0.0.0` | node (vite) | content-spark-hub dev server (Serve `:9443` `/`) |
| 5901 / [::]:5901 | `*` | x11vnc | VNC on Xvfb `:99`, see [[systems/headless-server]] | | 5175 | `127.0.0.1` | `python3 -m http.server` (`hermes-ui/app/dist`) | hermes-ui static build (Serve `/ui`) |
| 6080 / 6081 | `127.0.0.1` | websockify | noVNC web bridges for two VNC displays | | 9377 | `*` | node — `/home/hermes/services/camofox-browser` | **Camofox browser REST** (`{"ok":true,"engine":"camoufox"}`). Corrected 2026-11-05 — this port had been recorded as "Stratos — AI Command Centre". |
| 9222 | `0.0.0.0` | node `camoufox-ws-server.mjs` | Camofox Playwright/CDP WebSocket endpoint |
| 7777 | `0.0.0.0` | node `@openchamber/web serve` | OpenChamber web |
| 6767 | `0.0.0.0` | Paseo Daemon | Paseo daemon HTTP (loopback helper also on 42013) |
| 8080 | `*` | Ignis | Ignis browser-native Obsidian, see [[systems/ignis]] (Serve `/ignis`) |
| 8085 | `0.0.0.0` | `agentmail-webhook.py` | AgentMail webhook receiver; funnel-exposed (`/webhook`) |
| 8787 | `*` | `hermes-webui/server.py` | hermes-webui backend (Serve `/webui`); used as the health-probe target in [[runbooks/update-hermes-safely]] |
| 9099 | `0.0.0.0` | `job_dashboard_server.py` | jobs dashboard + static routes (Serve `/jobs`) |
| 9120 | `0.0.0.0` | `serve.py` (repo `public/`) | **Agent Estate Wiki site** (Serve `/wiki`) — added 2026-11-05 |
| 5901 | `*` | x11vnc | VNC on Xvfb `:99`, see [[systems/headless-server]] |
| 6080 / 6081 | `127.0.0.1` | websockify → 5900 / 5901 | noVNC bridges for the Camofox and desktop displays |
| 25 | `127.0.0.1` / `[::1]` | postfix | local mail transport | | 25 | `127.0.0.1` / `[::1]` | postfix | local mail transport |
\* `curl` target used in [[runbooks/update-hermes-safely]] health check; binding not independently re-verified in this pass. Loopback-only helpers (not routable): `49347` hermes gateway, `42013` Paseo, `42647` + `38187` opencode.
## Tailscale Serve / Funnel (verified 2026-11-07)
`tailscale serve status` on ct460 reports 14 proxies: 12 under `https://hermes.kangaroo-eel.ts.net` (tailnet only), one tailnet route on `:9443`, and one Funnel route on `:8443` (public).
| Route | Target | Note |
|---|---|---|
| `/` | `127.0.0.1:9119` | Hermes Web UI |
| `/ui` | `127.0.0.1:5175` | hermes-ui static build |
| `/jobs` | `127.0.0.1:9099` | jobs dashboard |
| `/wiki` | `100.118.5.51:9120` | Agent Estate Wiki |
| `/webui` | `127.0.0.1:8787` | hermes-webui |
| `/ignis` | `127.0.0.1:8080` | Ignis |
| `/gitea` | `100.82.100.128:3000` | Gitea — **CT221 on `m93p`**, not ct207 (⚠ **corrected 2026-11-07**: this line wrongly said *"ct207 tailnet IP"*; live `pct exec 221 -- tailscale ip -4` → `100.82.100.128`, and ct207's own IP is `100.82.40.23`). The target answers **HTTP 200**. |
| `/desktop` | `127.0.0.1:6081` | desktop VNC (noVNC) |
| `/vnc-camofox` | `127.0.0.1:6080` | Camofox VNC |
| `/evening-grid.html` | `127.0.0.1:9099/evening-grid.html` | jobs dashboard page |
| `/agent-estate` | `127.0.0.1:8769` | **DEAD — nothing listening on 8769 (re-confirmed 2026-11-07)** |
| `/synergy-prep` | `127.0.0.1:9123` | **DEAD — nothing listening on 9123 (re-confirmed 2026-11-07)** |
- `:9443` (tailnet only) — `/` → `127.0.0.1:5173`.
- Funnel `:8443` (public) — `/webhook` → `127.0.0.1:8085/webhook` (AgentMail webhook receiver).
## Relevant runbooks ## Relevant runbooks
- [[runbooks/diagnose-docker]] - [[runbooks/diagnose-docker]]
+102 -37
View File
@@ -3,8 +3,8 @@ title: Proxmox VE
type: infrastructure type: infrastructure
status: active status: active
created: 2026-07-26 created: 2026-07-26
updated: 2026-10-14 updated: 2026-11-07
verified_on: 2026-10-14 verified_on: 2026-11-07
confidence: high confidence: high
tags: [proxmox, virtualization, infrastructure, homelab] tags: [proxmox, virtualization, infrastructure, homelab]
--- ---
@@ -17,28 +17,31 @@ Hypervisor hosting all VMs and LXC containers for the homelab.
## Details ## Details
- **Host:** `pve.kangaroo-eel.ts.net` (Tailscale: `100.96.100.82`) - **Host:** `pve.kangaroo-eel.ts.net` (Tailscale: `100.96.100.82`)
- **Node name:** `pve` - **Node name:** `pve`
- **Version:** Proxmox VE 8.x - **Version:** Proxmox VE **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, running kernel `7.0.14-5-pve`) — **corrected 2026-10-31**; this page previously said *"Proxmox VE 8.x"*, which was wrong. `m93p` runs the **same** `pve-manager 9.2.10` build, so the two hosts are no longer version-skewed (see the `m93p` section).
- **Primary user:** `Anthony@pam` (full admin) - **Primary user:** `Anthony@pam` (full admin)
- **Additional users:** `root@pam`, `glance@pve`, `prom-exporter@pve`, `homepage@pve` - **Additional users:** `root@pam`, `glance@pve`, `prom-exporter@pve`, `homepage@pve`
## Live LXC Inventory (verified 2026-10-14 via `pct list`) ## Live LXC Inventory (verified 2026-11-07 via `pct list`)
Re-verified 2026-11-07 03:2x AWST (stocktake): **live `pct list` on `pve` still returns exactly the 17 LXCs in the table below** (CT255 `mattermost` `running`, CT209 `cloudreve` and CT555 `elementsynapse` `stopped`, CT600 `omniroute` `running`, and **no CT601 on either hypervisor**), and **`qm list` on `pve` still returns exactly the 4 VMs** (100 `homeassistant`, 101 `guest-vps`, 110 `omarchy`, 282 `ubuntu-server-xfce` — all running; **no 310, no 403, no 401**). Both `pve-manager` packages are still **9.2.10** on **both** hosts. The **one correction** was on the `m93p` side: see the `m93p` section below — **CT107 (`mem0`) was missing**, so the m93p guest count was one short.
Re-verified 2026-11-01 03:5x AWST (stocktake): **`pct list` on `pve` and `qm list` on `pve` both still match the tables below exactly** — 17 LXCs and 4 VMs on `pve`, with CT255 `mattermost` still `running`, CT209 `cloudreve` and CT555 `elementsynapse` still `stopped`, and **no CT601 on either hypervisor** (`/etc/pve/lxc/601.conf` absent). **`m93p` re-confirmed**: `pve-manager/9.2.10` (same build as `pve`), `qm list` returns **VM 103 `cachyos-m93p` stopped** and **VM 106 `openmausbot-vm` running**, and `pct exec 1001 -- docker ps` returns `sshwifty` (CT1001). Storage percentages were refreshed against live `pvesm status` on both hosts (see the Storage section). **Note:** `m93p` SSH over the tailnet timed out on the first two attempts and succeeded on the third — a flaky path, not a host outage (`tailscale ping` → pong throughout).
> **Two Proxmox hosts exist — always qualify a guest with its host.** `pvecm nodes` is empty on > **Two Proxmox hosts exist — always qualify a guest with its host.** `pvecm nodes` is empty on
> both, so they are independent and unclustered. Guest IDs also collide — **CT221 (gitea) and > both, so they are independent and unclustered. Guest IDs also collide — **CT221 (gitea) and
> CT245 (openweb) carry the same VMID on both hosts** — so an unqualified "CT221" is ambiguous. > CT245 (openweb) carry the same VMID on both hosts** — so an unqualified "CT221" is ambiguous.
> `m93p` (`100.104.51.98`, PVE 9.2.0) was **discovered 2026-10-14** and is documented in its own > `m93p` (`100.104.51.98`, PVE 9.2.10) was **discovered 2026-10-14** and is documented in its own
> section below. It hosts several services the wiki had placed on `pve`. > section below. It hosts several services the wiki had placed on `pve`.
| VMID | Name | Status | Purpose | | VMID | Name | Status | Purpose |
|---|---|---|---| |---|---|---|---|
| 103 | cliproxyapi | running | Cliproxy API | | 103 | cliproxyapi | running | Cliproxy API |
| 200 | docker | running | Main Docker host — 50+ containers (CT460's docker.sock is NOT accessible to the agent user; use CT200 for fleet inspection) | | 200 | docker | running | Main Docker host — 60+ containers (CT460's docker.sock is NOT accessible to the agent user; use CT200 for fleet inspection) |
| 201 | runtipi | running | Runtipi — Homepage, ntfy, etc. | | 201 | runtipi | running | Runtipi — Homepage, ntfy, etc. |
| 205 | ollama | running | Ollama LLM inference | | 205 | ollama | running | Ollama LLM inference |
| 207 | webserver | running | Web server | | 207 | webserver | running | Web server |
| 209 | cloudreve | **stopped** | Cloudreve (deprecated) | | 209 | cloudreve | **stopped** | Cloudreve (deprecated) |
| 215 | n8n | running | N8n automation | | 215 | n8n | running | N8n automation |
| 255 | mattermost | **stopped** | Mattermost — unreachable since ~2026-09-20 (Tailscale peer offline, **29d** as of 2026-10-19) | | 255 | mattermost | **running** | Mattermost — **back online 2026-10-30/31** after a ~40-day outage; `systemctl is-active mattermost` → `active`, listening on `:8065`, gateway reconnected 2026-10-30 16:36:17 |
| 270 | nextcloud | running | Nextcloud (apache2 active) | | 270 | nextcloud | running | Nextcloud (apache2 active) |
| 299 | sftpgo | running | SFTPGo file server | | 299 | sftpgo | running | SFTPGo file server |
| 333 | clawtest | running | ClawTest — Nanobot, Zeroclaw | | 333 | clawtest | running | ClawTest — Nanobot, Zeroclaw |
@@ -46,9 +49,16 @@ Hypervisor hosting all VMs and LXC containers for the homelab.
| 460 | hermes | running | Hermes CT460 (production) | | 460 | hermes | running | Hermes CT460 (production) |
| 465 | qwenpaw | running | QwenPaw + Paseo daemon (active) | | 465 | qwenpaw | running | QwenPaw + Paseo daemon (active) |
| 475 | agentzero | running | AgentZero (`100.106.54.94`) — **not previously recorded** | | 475 | agentzero | running | AgentZero (`100.106.54.94`) — **not previously recorded** |
| 555 | elementsynapse | **stopped** | Element Synapse (deprecated) — also retired from Tailscale (offline 81d) | | 555 | elementsynapse | **stopped** | Element Synapse (deprecated) — also retired from Tailscale (offline 102d) |
| 600 | omniroute | **running** | OmniRoute — live server on `:20128` (see [[systems/omniroute]]) | | 600 | omniroute | **running** | OmniRoute — live server on `:20128` (see [[systems/omniroute]]) |
| 601 | omni | **stopped** | Retired successor to CT600 — stopped, `onboot=0`, offline 54d |
**CT601 (`omni`) no longer exists.** The 2026-10-14 inventory listed it as `**stopped**`; live
`ls /etc/pve/lxc/601.conf` on **both** hypervisors returns **no such file**, and the orphaned `omni`
Tailscale peer (`100.93.204.4`) has **expired out of the tailnet entirely** — the live
`tailscale status --json` contains no peer named `omni` and no peer at that IP. The guest was
**deleted**, not merely halted. The only OmniRoute nodes left are **CT600** (`omniroute`,
`100.88.81.19`) and **CT101** (`omniroute-1`, `100.78.63.20`, on `m93p`) — both verified running with
`omniroute` listening on `0.0.0.0:20128`. → [[systems/omniroute]], [[infrastructure/hosts]]
### ⚠ Four entries moved off this host (corrected 2026-10-14) ### ⚠ Four entries moved off this host (corrected 2026-10-14)
@@ -66,19 +76,26 @@ error forward.
| ~~CT245 openweb~~ | `m93p` | **CT245** — same ID, different host | | ~~CT245 openweb~~ | `m93p` | **CT245** — same ID, different host |
| ~~CT260 freshrss~~ | `m93p` | **CT1260** | | ~~CT260 freshrss~~ | `m93p` | **CT1260** |
## Live QEMU Inventory (verified 2026-10-14 via `qm list`) ## Live QEMU Inventory (verified 2026-10-31 via `qm list`)
| VMID | Name | Status | Notes | | VMID | Name | Status | Notes |
|---|---|---|---| |---|---|---|---|
| 100 | homeassistant | running | Home Assistant (13GB RAM) | | 100 | homeassistant | running | Home Assistant (**5100 MB** RAM — the page previously said 13 GB) |
| 110 | omarchy | running | Omarchy (13GB RAM, 106GB disk) — **new, not previously recorded** | | 101 | guest-vps | running | Guest VPS (6144 MB RAM, 48 GB disk) — **was missing from this table entirely** |
| 282 | ubuntu-server-xfce | **running** | Ubuntu desktop — recorded **stopped** until 2026-10-14; live `qm list` shows running | | 110 | omarchy | running | Omarchy (13000 MB RAM, 106 GB disk) |
| 310 | cachyos | **stopped** | CachyOS | | 282 | ubuntu-server-xfce | **running** | Ubuntu desktop (5000 MB RAM, 133 GB disk) — recorded **stopped** until 2026-10-14; live `qm list` shows running |
| 310 | cachyos | **absent** | CachyOS — **no longer present in `qm list`** on `pve`; the only CachyOS guest is `cachyos-m93p` (VM 103) on `m93p`, which is stopped |
| 403 | open-claw | — | OpenClaw agent; **not present in `qm list`** — it runs elsewhere (Open-Claw VM, see [[infrastructure/hosts]]) | | 403 | open-claw | — | OpenClaw agent; **not present in `qm list`** — it runs elsewhere (Open-Claw VM, see [[infrastructure/hosts]]) |
| 401 | hermes-old | — | Legacy Hermes VM; **not present in `qm list`** — appears decommissioned/removed | | 401 | hermes-old | — | Legacy Hermes VM; **not present in `qm list`** — appears decommissioned/removed |
**Not in either list:** VM 403 (`open-claw`) and VM 401 (`hermes-old`) are referenced by the wiki but no longer appear in `qm list`. They have either been migrated off this host or deleted. Status unconfirmed — verify before relying on either entry. **Not in either list:** VM 403 (`open-claw`) and VM 401 (`hermes-old`) are referenced by the wiki but no longer appear in `qm list`. They have either been migrated off this host or deleted. Status unconfirmed — verify before relying on either entry.
**Re-verified 2026-10-31:** live `qm list` on `pve` returns exactly **4 VMs — 100 homeassistant, 101
guest-vps, 110 omarchy, 282 ubuntu-server-xfce, all running**. Two corrections to the 2026-10-14 table:
**VM 101 (`guest-vps`) was missing** despite running, and **VM 310 (`cachyos`) no longer exists** on this
host (it had been recorded as `stopped`; the live `qm list` has no 310 at all). Home Assistant's memory was
also corrected from *"13GB"* to the live **5100 MB**.
## Host `m93p` (100.104.51.98) — second Proxmox hypervisor, discovered 2026-10-14 ## Host `m93p` (100.104.51.98) — second Proxmox hypervisor, discovered 2026-10-14
Identified because a tailnet-wide scan for `:8006` returned only `m93p` and `pve`. It had been Identified because a tailnet-wide scan for `:8006` returned only `m93p` and `pve`. It had been
@@ -86,17 +103,20 @@ sitting in the Tailscale node list as an unremarkable `online` peer, but no wiki
as a hypervisor. as a hypervisor.
- **Hostname:** `m93p` · **Tailscale:** `100.104.51.98` - **Hostname:** `m93p` · **Tailscale:** `100.104.51.98`
- **PVE version:** **9.2.0** (kernel `7.0.14-12-pve`, `pve-manager 9.2.10`) — materially newer than `pve`'s 8.x - **PVE version:** **9.2.10** (`pve-manager/9.2.10/43df2e01f27a1a19`, kernel `7.0.14-12-pve`) — **corrected 2026-10-31**. The previous entry read *"9.2.0"* and claimed `m93p` was *"materially newer than pve's 8.x"*. Both hosts now run the **identical `pve-manager 9.2.10` build**; only the kernel differs (`7.0.14-12-pve` vs `7.0.14-5-pve`). The hosts are **version-identical, not version-skewed**.
- **Uptime:** **up 1 day, 7:18** at 2026-11-07 04:38 AWST (live `uptime -s` → **boot 2026-11-05 21:19:52**; `uptime` → `up 1 day, 7:18`, load average **0.76 / 1.08 / 1.24** — normal). ⚠ **Corrected 2026-11-07:** the previous entry read *"up 1 day, 12:02 at 2026-11-01 03:56 AWST — the host rebooted ~2026-10-30 16:0x"*. The **reboot itself is still real** (`pve` booted **2026-10-30 16:24:00** per `uptime -s`, and the CT1264 openmausbot outage below follows from it), but the **uptime figure and the implied boot date were wrong** — the host has since rebooted again on **2026-11-05 21:19:52**, so it was up only **1 day, 7:18** when probed, not 12 h. The `pve` host's own boot time is **2026-10-30 16:24:00** (up **7 days, 12:14** at probe time)
- **Not clustered with `pve`:** `pvecm nodes` empty, no `/etc/pve/corosync.conf` - **Not clustered with `pve`:** `pvecm nodes` empty, no `/etc/pve/corosync.conf`
### LXC Inventory (verified 2026-10-14 via `pct list` on m93p) ### LXC Inventory (verified 2026-10-31 via `pct list` on m93p)
| VMID | Name | Status | Notes | | VMID | Name | Status | Notes |
|---|---|---|---| |---|---|---|---|
| 100 | obsidian-livesync | running | Obsidian LiveSync — ID does not collide; `pve` has no CT100 | | 100 | obsidian-livesync | running | Obsidian LiveSync — ID does not collide; `pve` has no CT100 |
| 101 | omniroute | running | **An OmniRoute instance distinct from `pve`'s CT600/CT601 — see [[systems/omniroute]]; identity/role unconfirmed** | | 101 | omniroute | running | **Second OmniRoute node**, `100.78.63.20` (`omniroute-1` Tailscale peer), `omniroute` listening on `:20128` — identity/role **now confirmed**, see [[systems/omniroute]] |
| 102 | vaultwarden | running | Vaultwarden password manager | | 102 | vaultwarden | running | Vaultwarden password manager |
| 104 | rclone | running | Rclone | | 104 | rclone | running | Rclone |
| 105 | conduit | running | **Conduit** — iPhone calendar / health / location / contacts bridge (see [[systems/messaging-integrations]]). **Added 2026-10-31** — this guest was missing from the 2026-10-14 table despite running |
| 107 | mem0 | running | **mem0 MCP server** (`mem0-mcp.service` → `:8890`), Tailscale `100.73.94.53` (`mem0` peer) — **added 2026-11-07**, this guest was missing from the table despite running |
| 221 | gitea | running | **This is the host behind the wiki's `gitea` node — the Gitea remote lives here, not on `pve`** | | 221 | gitea | running | **This is the host behind the wiki's `gitea` node — the Gitea remote lives here, not on `pve`** |
| 245 | openweb | running | Open WebUI — `pve` has no live CT245; the guest this page previously recorded on `pve` is this one | | 245 | openweb | running | Open WebUI — `pve` has no live CT245; the guest this page previously recorded on `pve` is this one |
| 500 | litellm | running | LiteLLM proxy | | 500 | litellm | running | LiteLLM proxy |
@@ -105,47 +125,92 @@ as a hypervisor.
| 1006 | freellmapi | running | FreeLLM API | | 1006 | freellmapi | running | FreeLLM API |
| 1007 | homebridge | running | Homebridge | | 1007 | homebridge | running | Homebridge |
| 1008 | rustdesk | running | RustDesk remote access | | 1008 | rustdesk | running | RustDesk remote access |
| 1241 | ntfy | running | Ntfy push notifications — wiki's `CT241` | | 1241 | ntfy | running | ntfy push notifications — wiki's `CT241` |
| 1260 | freshrss | running | FreshRSS feed reader — wiki's `CT260` | | 1260 | freshrss | running | FreshRSS feed reader — wiki's `CT260` |
| 1261 | paseo | running | Paseo coding-agent daemon | | 1261 | paseo | running | Paseo coding-agent daemon |
| 1262 | mailflow | running | MailFlow | | 1262 | mailflow | running | MailFlow |
| 1263 | social-dashboard | running | Social dashboard | | 1263 | social-dashboard | running | Social dashboard |
| 1264 | openmausbot | **stopped** | OpenMausBot — **stopped**, which is why the `openmausbot` Tailscale peer is offline 9d. **Added 2026-10-31** — was missing from the 2026-10-14 table. The companion **VM106 `openmausbot-vm` is running** and holds the separate `openmausbot-vm` peer (`100.77.107.30`) |
### QEMU Inventory (verified 2026-10-14 via `qm list` on m93p) **Re-verified 2026-11-07 03:2x AWST (stocktake):** live `pct list` on `m93p` returns **20 LXCs — 18 running,
2 stopped (CT998 `nightbridge` and CT1264 `openmausbot`)**. ⚠ **The count was one short**: the previous
figure of 19 omitted **CT107 (`mem0`)**, a live running guest confirmed from inside the container
(`pct exec 107 -- tailscale ip -4` → `100.73.94.53`, `systemctl is-active tailscaled` → `active`,
`mem0-mcp.service` → `active`). CT107 is now in the table above.
**Re-verified 2026-10-31:** the earlier reading of **19 LXCs — 18 running, 1 stopped
(CT998 nightbridge) plus CT1264 openmausbot stopped** was itself already one short for the same reason
(CT107 `mem0` was not counted). The 2026-10-14 table listed only **17**, omitting
two live guests: **CT105 (`conduit`)** and **CT1264 (`openmausbot`)**. Note that **VMIDs 103, 220 and 555
do not exist on `m93p`** (`/etc/pve/lxc/{103,220,555}.conf` all absent) — CT103 (cliproxyapi),
CT220 (caddy) and CT555 (elementsynapse) are on **`pve`** only, and `pve`'s CT555 is **stopped** while
`m93p` has no 555 at all. Guest IDs still collide across hosts, so always qualify with the host.
### QEMU Inventory (verified 2026-10-31 via `qm list` on m93p)
| VMID | Name | Status | Notes | | VMID | Name | Status | Notes |
|---|---|---|---| |---|---|---|---|
| 103 | cachyos-m93p | **stopped** | CachyOS | | 103 | cachyos-m93p | **stopped** | CachyOS (6144 MB RAM, 80 GB disk) |
| 106 | openmausbot-vm | running | OpenMausBot VM (8192 MB RAM, 30 GB disk) — **Added 2026-10-31**; was missing from the 2026-10-14 table. Holds the `openmausbot-vm` Tailscale peer (`100.77.107.30`) |
**Reachability note:** guest service ports on `m93p` were **not** reachable from CT460 during this **Reachability note:** guest service ports on `m93p` were **not** reachable from CT460 during this
pass (they bind to their own interfaces and are not Tailscale-Served). Only hypervisor `:8006` on pass (they bind to their own interfaces and are not Tailscale-Served). Only hypervisor `:8006` on
`100.104.51.98` is open. Reach them via the host or a Tailscale Serve route before probing services. `100.104.51.98` is open. Reach them via the host or a Tailscale Serve route before probing services.
Re-confirmed 2026-10-31: ports 3000, 4000, 5432, 8766, 8080 and 9000 were all closed from CT460.
**⚠ Follow-up:** `m93p`'s guest list is recorded here from a single `pct list` / `qm list` pass. **✅ Follow-up closed 2026-10-31:** `m93p`'s `omniroute` (CT101) is a **second live OmniRoute node**,
The pages [[systems/omniroute]], [[infrastructure/hosts]], [[systems/model-providers]] and not a duplicate of CT600/CT601. Live `pct config 101` on `m93p` confirms `hostname: omniroute`, and
[[infrastructure/docker-services]] still describe single-host topology and should be re-verified `omniroute` is listening on `0.0.0.0:20128` inside it; the `omniroute-1` Tailscale peer
against `m93p` — in particular whether `m93p`'s `omniroute` (CT101) is a fourth OmniRoute node or (`100.78.63.20`) resolves to it. **CT601 no longer exists on either host** — `/etc/pve/lxc/601.conf` is
a duplicate of CT600/CT601. absent from `pve` and `m93p`, and the orphaned `omni` peer (`100.93.204.4`) has expired out of the
tailnet. → [[systems/omniroute]], [[infrastructure/hosts]]
## Storage (live, 2026-10-14 03:0x AWST — `pvesm status` + `df -h /mnt/ssd0`, host `pve`) ## Storage (live, 2026-11-01 03:5x AWST — `pvesm status` on both hosts)
### Host `pve` (100.96.100.82)
| Datastore | Type | Status | Total | Used | Used % | | Datastore | Type | Status | Total | Used | Used % |
|---|---|---|---|---|---| |---|---|---|---|---|---|
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 808 GiB | **88.17 %** (`pvesm`) | | `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 717 GiB | **78.29 %** (`pvesm`) |
| `SSD1` | lvmthin | active | 976 GiB | 815 GiB | 87.47 % | | `SSD1` | lvmthin | active | 931 GiB | 610 GiB | **65.46 %** |
| `fred-qnap` | cifs | active | 13.7 TiB | 10.3 TiB | 77.22 % | | `fred-qnap` | **nfs** | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
| `local` | dir | active | 67.7 GiB | 55.1 GiB | 81.68 % | | `local` | dir | active | 67.7 GiB | 55.3 GiB | **81.62 %** |
| `local-lvm` | lvmthin | active | 141 GiB | 19.0 GiB | 13.44 % | | `local-lvm` | lvmthin | active | 141 GiB | 20.1 GiB | **14.25 %** |
| `iMacHDD` | cifs | **inactive** | — | — | — |
**`ssd0` improved from the 93 % figure: now 88.17 % by `pvesm`, ~63 GiB free.** The earlier 93 % **`ssd0` has improved further: 88.17 % (2026-10-14) → 76.43 % (2026-11-01) → 78.29 % (2026-11-07)**, ~152 GiB free. `fred-qnap` is now
came from `df`, which excludes reserved blocks; `pvesm` divides by the full 960 GiB. Both describe **84.21 %** and its type changed from **`cifs` to `nfs`** (13.4 TiB total, 11.3 TiB used) — the share was
the same ~808 GiB used. **Use the `pvesm` number when comparing across hosts** — it is the only remounted over NFS rather than SMB.
one that means the same thing on `m93p` too. `iMacHDD` remains inactive
(`storage 'iMacHDD' is not online`) — check the CIFS mount before using it as a target. **Storage re-verified 2026-11-07 03:2x AWST** against live `pvesm status` on both hosts — every
percentage on both tables below was refreshed.
**`iMacHDD` no longer appears in `pvesm status` on `pve`** — the 2026-10-14 table listed it as an
inactive `cifs` entry. The iMac shares now live on **`m93p`** as `cifs-anthonyimac` and `cifs-imachdd`
(both active, 76.21 %). Storage was **reassigned from `pve` to `m93p`** rather than the mount failing.
### Host `m93p` (100.104.51.98)
| Datastore | Type | Status | Total | Used | Used % |
|---|---|---|---|---|---|
| `cifs-anthonyimac` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
| `cifs-imachdd` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
| `local` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
| `local-hdd` | dir | active | 93.9 GiB | 71.6 GiB | 76.21 % |
| `nfs-proxmox-31` | dir | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
| `nfs-proxmox-54` | dir | active | 13.4 TiB | 11.3 TiB | **84.21 %** |
| `nfs-ssd0-39` | dir | active | 916 GiB | 717 GiB | **78.29 %** |
| `omb-internal` | dir | active | 93.9 GiB | 60.7 GiB | **64.66 %** |
**`m93p` storage was not documented before 2026-10-31.** Note that `m93p` exposes the **same backing
volumes as `pve`** under different names — `nfs-proxmox-31`/`nfs-proxmox-54` mirror `pve`'s `fred-qnap`
(both 84.21 %), and `nfs-ssd0-39` mirrors `pve`'s `ssd0` (both 78.29 %). The iMac shares that were
`iMacHDD` on `pve` are `cifs-anthonyimac` / `cifs-imachdd` on `m93p`.
**Mount points on the host** (`ls -d /mnt/*`): `/mnt/ssd0`, `/mnt/pve`, `/mnt/vm`, `/mnt/rootfs`, plus per-guest mounts `/mnt/ct200`, `/mnt/vm110-esp-ro`, `/mnt/vm282`, `/mnt/vm283`, `/mnt/vm400`, `/mnt/vm402`, `/mnt/esp110`, `/mnt/test-nfs`, `/mnt/tmp` and retired ones (`/mnt/old-openclaw-ro`, `/mnt/openclaw-old`, `/mnt/vm400-backup`, `/mnt/vm400-export`, `/mnt/vm402-fix`). **Mount points on the host** (`ls -d /mnt/*`): `/mnt/ssd0`, `/mnt/pve`, `/mnt/vm`, `/mnt/rootfs`, plus per-guest mounts `/mnt/ct200`, `/mnt/vm110-esp-ro`, `/mnt/vm282`, `/mnt/vm283`, `/mnt/vm400`, `/mnt/vm402`, `/mnt/esp110`, `/mnt/test-nfs`, `/mnt/tmp` and retired ones (`/mnt/old-openclaw-ro`, `/mnt/openclaw-old`, `/mnt/vm400-backup`, `/mnt/vm400-export`, `/mnt/vm402-fix`).
**Use the `pvesm` number when comparing across hosts** — it is the only one that means the same thing on
both hypervisors.
## API Tokens ## API Tokens
- `homepage@pve!homepage` — PVEAuditor role, privsep=0 (used by Homepage dashboard) - `homepage@pve!homepage` — PVEAuditor role, privsep=0 (used by Homepage dashboard)
+62 -43
View File
@@ -3,8 +3,8 @@ title: Tailscale
type: infrastructure type: infrastructure
status: active status: active
created: 2026-07-26 created: 2026-07-26
updated: 2026-10-26 updated: 2026-11-12
verified_on: 2026-10-26 verified_on: 2026-11-12
confidence: high confidence: high
tags: [tailscale, vpn, networking, infrastructure] tags: [tailscale, vpn, networking, infrastructure]
--- ---
@@ -19,68 +19,83 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access
- **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`) - **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`)
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`) - **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
## Tailscale Node Inventory (verified 2026-10-25) ## Tailscale Node Inventory (verified 2026-11-12)
Full node list from `tailscale status` — **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age. Re-verified 2026-11-12 03:1x–03:4x AWST (stocktake): table rebuilt from `tailscale status --json` — **three peers flipped offline in the peer record while all three are in fact reachable, one genuine offline age/stamp correction, and 10 offline ages drifted +2d. No IP changes, no missing table rows, no new peers.** ⚠ **`mailflow` (100.115.60.92, CT1262 m93p), `qwenpaw` (100.110.72.126, CT465 pve) and `springroll` (100.100.153.125) all read `Online: false` in the peer record** (`LastSeen` 2026-11-10 13:59Z / 2026-11-11 12:20Z / 2026-11-11 08:30Z), yet **`tailscale ping` returns pong on every one** — mailflow via a direct path (`192.168.178.142:41641`, 2 ms), qwenpaw direct (`192.168.178.186:41641`, 7 ms) and springroll via **DERP(sin)** (210 ms, relay rather than direct) — and live `pct list` shows **CT1262 and CT465 running**. All three rows therefore stay **online**; the peer record is stale, exactly the m93p pattern this page has already recorded three times. ⚠ **`omarchymac` was genuinely stale in the *other* direction:** the 2026-11-10 row read *"offline 2d — last seen 2026-11-08 00:45Z"*, while the live record reads `LastSeen 2026-11-07T06:41:48Z` = **offline 4d** — the peer was last seen *earlier* than the page claimed and `tailscale ping` → **timeout**, so it is a real offline, not a stale record. The **9 named offline ages** were refreshed (+2d each): cloudreve 108d → **110d**; conduit 65d → **66d**; elementsynapse 111d → **113d**; freshrss-new 84d → **86d**; octoagent 8d → **10d**; omarchybook 19d → **21d**; omarchymac 2d → **4d**; openmausbot 19d → **21d**; ubuntu-server-xfce 81d → **82d**; and the six `codex-environment-*` ephemeral peers 2d → **4d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (58 rows = 57 live IPv4 peers + `hermes` itself; `tailscale status --json` reports **80 peers total — 57 with a `100.` IPv4, 23 IPv6-only**, unchanged). Container states cross-checked against live `pct list` **and `qm list`** on **both** hypervisors: `pve` (100.96.100.82) **17 LXCs + 4 VMs** (VM100 homeassistant, VM101 guest-vps, VM110 omarchy, VM282 ubuntu-server-xfce), `m93p` (100.104.51.98) **20 LXCs + 2 VMs** (VM103 cachyos-m93p stopped, VM106 openmausbot-vm running) — counts unchanged. **One new guest not previously on this page: CT1007 `homebridge` on `m93p`, running but with `tailscaled` `inactive` inside it** (`pct exec 1007 -- systemctl is-active tailscaled` → `inactive`), so it is a running guest that does not join the tailnet — the same class as CT105 `conduit` and CT1264 `openmausbot`. **Both DEAD Serve targets re-confirmed dead:** `ss -ltn` still shows **no listener on 8769** (`/agent-estate`) and **none on 9123** (`/synergy-prep`). **CT201's Serve table re-probed** (`pct exec 201 -- tailscale serve status`) and is unchanged: `/` → `8085` and `/mgr` → `8084` on both `:443` and `:8443`. **Re-probed and confirmed this page's stale incident:** CT299 `sftpgo` is online and healthy → [[incidents/2026-09-02-sftpgo-outage]].
Re-verified 2026-11-10 00:4x AWST (nightly audit): `tailscale status --json` re-read — **6 new peers added, 8 offline ages drifted, 1 state transition back *online* (`octop`); no IP changes and no missing table rows.** ⚠ **Six peers that had never been listed now sit in the tailnet** — `codex-environment` (`100.111.126.78`), `codex-environment-1b88dc` (`100.76.42.76`), `codex-environment-8dadbb` (`100.65.200.27`), `codex-environment-955866` (`100.81.119.118`), `codex-environment-9fbd61` (`100.106.215.121`) and `codex-environment-d3edbd` (`100.74.3.2`) — every one created **2026-11-07 05:09–05:56Z**, every one **`Online: false`** with `LastSeen` **2026-11-07 11:09–11:56Z**. **None is a guest on either hypervisor** (live `pct list` and `qm list` on both `pve` and `m93p` list no such guest), so they are **orphaned/ephemeral tailnet peers**, not estate hosts: six records registered inside 47 minutes and all went silent within ~6 h, the signature of a throwaway CI/dev environment rather than infrastructure. **State transition:** `octop` (`100.92.163.122`) went **online → offline 1d** (`LastSeen 2026-11-08T13:13:04Z`) after returning for a single day on 11-07. The **9 offline ages** were refreshed (+1d each): cloudreve 105d → **106d**; conduit 61d → **62d**; elementsynapse 108d → **109d**; freshrss-new 81d → **82d**; octoagent 5d → **6d**; omarchybook 16d → **17d**; openmausbot 16d → **17d**; ubuntu-server-xfce 77d → **78d**; plus **`omarchymac`, which was not aged but re-stated** — the 2026-11-07 row read *"offline 5d — last seen 2026-11-01 11:13Z"* while the live record reads **`LastSeen 2026-11-08T00:45:17Z` = offline 2d**, so **both the age and the last-seen stamp on that row were stale** and are corrected here. Node count **52 → 58 non-funnel nodes** (57 live IPv4 peers + `hermes` itself). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none** once the six additions land; the **~23 IPv6-only `funnel-ingress-node` peers** remain deliberately excluded (`tailscale status --json`: **80 peers total — 57 with a `100.` IPv4, 23 IPv6-only**). Container counts re-confirmed live: `pve` (100.96.100.82) **17 LXCs + 4 VMs**, `m93p` (100.104.51.98) **20 LXCs + 2 VMs** — unchanged. **Not re-probed this pass** (so the 2026-11-07 findings stand un-contradicted rather than re-verified): the `:443` Serve route table (12 paths), the two DEAD CT460 targets (8769 `/agent-estate`, 9123 `/synergy-prep`), and per-guest tailnet identity via `pct exec <vmid> -- tailscale ip -4`.
Re-verified 2026-11-07 03:1x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 9 offline-age drifts, no IP changes and no missing peers.** **`octop` came back online**: the peer record reads `Online: true` with `LastSeen 0001-01-01` and `tailscale ping 100.92.163.122` → **`pong from octop via DERP(syd) in 98ms`**, so the row moves offline → online. It remains **not an LXC or VM on either hypervisor** (live `pct list` / `qm list` on both `pve` and `m93p` list no such guest) — an **orphaned tailnet peer**, so its return is not an estate change. The **9 offline ages** were refreshed (+1d each): cloudreve 104d → **105d**; conduit 60d → **61d**; elementsynapse 107d → **108d**; freshrss-new 80d → **81d**; octoagent 4d → **5d**; omarchybook 15d → **16d**; omarchymac 4d → **5d**; openmausbot 15d → **16d**; ubuntu-server-xfce 76d → **77d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) **17 guests** and `m93p` (100.104.51.98) **20 guests + 2 VMs** — ⚠ **the m93p guest count on this page was one short**: live `pct list` on `m93p` returns **20 LXCs** (the extra is **CT107 `mem0`**, `100.73.94.53`, `mem0-mcp.service` active — confirmed from inside the guest), not 19; CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT998 `nightbridge` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). The route table still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; **both DEAD targets re-confirmed dead** — `ss -ltn` still shows **no listener on 8769** (`/agent-estate`) and **none on 9123** (`/synergy-prep`) — while every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` answers **HTTP 200**.
Re-verified 2026-11-06 03:0x–03:2x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 8 offline-age drifts, no IP changes and no missing peers.** **`anthonys-mac-mini` came back online**: the peer record reads `Online: true` and `tailscale ping 100.76.60.37` → `pong from anthonys-mac-mini via DERP(syd) in 103ms`, so the row moves offline → online (client device, not an estate host). The **8 offline ages** were refreshed (+1d each): cloudreve 103d → **104d**; elementsynapse 106d → **107d**; freshrss-new 79d → **80d**; octoagent 3d → **4d**; octop 1d → **2d**; omarchybook 14d → **15d**; omarchymac 3d → **4d**; openmausbot 14d → **15d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). `m93p` re-pinged → `pong from m93p via 192.168.178.250:41641 in 3ms` (online, peer record agrees), and `octop` re-pinged → **timeout** (genuinely offline, matching its `LastSeen`). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) 17 guests and `m93p` (100.104.51.98) 19 guests + 2 VMs — CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). ⚠ **Two CT460 Serve routes are dead but still present in `tailscale serve status`**: `/agent-estate` → `127.0.0.1:8769` and `/synergy-prep` → `127.0.0.1:9123` have **no listener** (live `ss -ltn` probe) and therefore serve 502 — both rows are now marked DEAD here, matching [[infrastructure/network-map]] which had already recorded them as dead on 2026-11-05. The route table itself still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` is reachable.
Full node list from `tailscale status` — **58 non-funnel nodes** (57 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
Re-verified 2026-11-04 03:0x–05:5x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 missing peers added, 3 state transitions, 8 offline-age drifts.** **Two live peers had never been listed** and are now in the table: **`mem0`** (`100.73.94.53`, CT107 on `m93p` — confirmed by `pct exec 107 -- tailscale ip -4`) and **`obsidian-livesync`** (`100.116.244.84`, CT100 on `m93p` — confirmed by `pct exec 100 -- tailscale ip -4`). Node count corrected to **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). **Three genuine transitions, each confirmed by a ping rather than the peer record alone:** **`octoagent`** (100.117.10.68) went **online → offline 2d** (`LastSeen 2026-11-01T13:48:20Z`, `tailscale ping` → timeout) and **`octop`** (100.92.163.122) went **online → offline <1d** (`LastSeen 2026-11-03T09:03:02Z`, ping → timeout); **neither is an LXC or VM on either hypervisor** (`pct list` / `qm list` on both `pve` and `m93p` list no such guest) — they are **orphaned tailnet peers**, so their loss is not an estate outage. **`sftpgo`** (CT299) went **offline → online** during this audit: it had been **frozen for ~24 h by a wedged `vzdump` task** whose NFS write path to `fred-qnap` had hung; killing the task unfroze the guest and `pct exec 299 -- systemctl is-active sftpgo` → **`active`** with `LISTEN *:8080` → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. **`omarchymac`** aged from `offline <1d` (last seen 2026-10-31 13:20Z) to **offline 2d** (`LastSeen` is actually `2026-11-01T11:13:06Z`). The **8 offline ages** were refreshed (cloudreve 99d → **102d**; conduit 55d → **58d**; elementsynapse 102d → **105d**; freshrss-new 75d → **78d**; omarchybook 10d → **13d**; openmausbot 10d → **13d**; ubuntu-server-xfce 72d → **75d**; omarchymac <1d → **2d**). **`m93p`'s peer record reads `Online: true` again** — the stale-record run of 10-29 → 11-01 (four consecutive false reads) has cleared, and `tailscale ping 100.104.51.98` → pong. Container identities re-confirmed from **inside** the guests (`pct exec <vmid> -- tailscale ip -4` on both hypervisors): CT107 `mem0`, CT100 `obsidian-livesync`, CT255 `mattermost` (**running**, `systemctl is-active mattermost` → `active`, `curl http://100.98.116.14:8065/api/v4/system/ping` → **HTTP 200**), CT333 `clawtest` (**running**), CT207 `webserver` (**running**, `pct exec 207 -- tailscale ip -4` → `100.82.40.23`), CT299 `sftpgo` (**running**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
Re-verified 2026-10-31 05:4x AWST (stocktake): table rebuilt from `tailscale status --json` — **one state transition and seven offline-age drifts, no IP changes and no missing peers.** **`mattermost` came back online**: the peer record reads `Online: true` for the first time since the outage, and the guest behind it is genuinely healthy — live `pct exec 255 -- systemctl is-active mattermost` → **`active`**, `ss -ltnp` shows `mattermost` listening on `*:8065`, `pct exec 255 -- tailscale ip -4` → `100.98.116.14`, and the gateway's own `gateway.log` records **`✓ mattermost connected` at 2026-10-30 16:36:17** (after an earlier successful reconnect at 07:19:37 the same day). CT255 is therefore **running** on `pve` — the 40-day outage is over. The **7 offline ages** were refreshed (cloudreve 97d → **98d**; conduit 53d → **55d**; elementsynapse 100d → **102d**; freshrss-new 73d → **74d**; omarchybook 8d → **9d**; openmausbot 8d → **9d**; ubuntu-server-xfce 70d → **71d**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Two container annotations were added for peers that had none: `openmausbot` (**CT1264, m93p — stopped**) and `openmausbot-vm` (**VM106, m93p**), both confirmed from live `pct list` / `qm list` on `m93p`. The `m93p` row's "peer record stale" note was updated — its peer record read `Online: false` on 10-29 and 10-30 and reads **`Online: true`** today, the third consecutive day the record disagreed with the (reachable) host. CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
Re-verified 2026-11-01 03:0x–04:0x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition and 5 offline-age drifts, no IP changes and no missing peers.** **`omarchymac` went offline**: the peer record reads `Online: false` with `LastSeen 2026-10-31T13:20:00Z` and `tailscale ping 100.126.173.115` times out, so this is a **genuine** transition (the JSON `OS` field reads `linux`, not a Mac). **`m93p`'s peer record reads `Online: false` again** — the **4th consecutive** stale record — while `tailscale ping 100.104.51.98` → `pong from m93p via DERP(syd) in 98ms` and SSH over the tailnet returned `up 1 day, 12:02`, so the row stays **online** and the peer record remains non-authoritative for hypervisor reachability. The **5 offline ages** were refreshed (cloudreve 98d → **99d**; freshrss-new 74d → **75d**; omarchybook 9d → **10d**; openmausbot 9d → **10d**; ubuntu-server-xfce 71d → **72d**). **`conduit` (55d) and `elementsynapse` (102d) were already correct** and are unchanged. **Every peer IP in the table still matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Container identity re-confirmed from inside the guests: `pct exec 201 -- tailscale ip -4` → `100.96.244.39` (CT201, runtipi), `pct exec 1001 -- docker ps` → `sshwifty` (CT1001 on `m93p`), CT209 `cloudreve` and CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT255 `mattermost` **running** with `systemctl is-active mattermost` → `active` and `LISTEN *:8065`. Both OmniRoute nodes re-probed: `100.88.81.19` (CT600) and `100.78.63.20` (CT101) both return **401** on `/v1/models` (alive, auth required).
Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong). Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong).
Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`. Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`.
Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**. Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**.
Re-verified 2026-10-21 03:0x: the 2026-10-19 table had drifted again — **`octop` (100.92.163.122), `springroll` (100.100.153.125), `openmausbot` and `openmausbot-vm` were missing from the table body** (only mentioned in prose), `omarchymac` had gone offline (online → idle/offline 3d), and every offline age was stale. Table rebuilt from `tailscale status --json`, keyed on DNS name rather than `HostName` (the JSON `HostName` for `freshrss` is `ct260`, and both `omniroute` nodes share the hostname `omniroute`). Container states cross-checked against live `pct list` on `pve` (100.96.100.82) and `m93p` (100.104.51.98): CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running. Re-verified 2026-10-21 03:0x: the 2026-10-19 table had drifted again — **`octop` (100.92.163.122), `springroll` (100.100.153.125), `openmausbot` and `openmausbot-vm` were missing from the table body** (only mentioned in prose), `omarchymac` had gone offline (online → idle/offline 3d), and every offline age was stale. Table rebuilt from `tailscale status --json`, keyed on DNS name rather than `HostName` (the JSON `HostName` for `freshrss` is `ct260`, and both `omniroute` nodes share the hostname `omniroute`). Container states cross-checked against live `pct list` on `pve` (100.96.100.82) and `m93p` (100.104.51.98): CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running.
Re-verified 2026-10-30 03:3x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 IP changes** (anthonys-mac-mini: 100.74.17.15 -> 100.76.60.37; omarchymac: 100.84.180.69 -> 100.126.173.115) and **4 nodes dropped out of the tailnet entirely** (anthonys-imac (100.112.123.124), cachyos (100.124.208.15), omarchymac-1 (100.126.173.115), omni (100.93.204.4)). `omarchymac` is now `100.126.173.115` (the peer formerly listed as `omarchymac-1`; the old `omarchymac` at `100.84.180.69` no longer matches any peer) and `anthonys-mac-mini` moved to `100.76.60.37` — both same-named nodes, new addresses. The four dropped peers are Tailscale expired/offline records (`anthonys-imac`, `cachyos`, `omni`, and the old `omarchymac` address), not newly-installed hardware. **`m93p` reads `Online: false` in the peer record for the third consecutive audit, yet the host is plainly up — `tailscale ping 100.104.51.98` → `pong from m93p via 192.168.178.118:41641 in 0s` and SSH returned `m93p / up 12 days, 11:02` — so the peer record is not authoritative for hypervisor reachability and the row records it as online. Node count corrected to **50 non-funnel nodes** (49 live IPv4 peers + `hermes` itself); `tailscale status --json` reports **72 peers total — 49 with a `100.` IPv4, 23 IPv6-only** (the `funnel-ingress-node` set). **Live IPv4 peers absent from the table: none.** Container identity re-confirmed from inside guests via `pct exec <vmid> -- tailscale ip -4` on `pve` (100.96.100.82) and `m93p` (100.104.51.98); CT600 is confirmed **running** and CT601 confirmed **absent from both hypervisors**. Serve tables re-probed per host: CT460's `:443` route set matches the table below exactly (12 paths), and CT201 (`pct exec 201 -- tailscale serve status`) serves only `/` → `127.0.0.1:8085` and `/mgr` → `127.0.0.1:8084` on **both** `:443` and `:8443`, with no `/steel` or `/steel-api` routes remaining.
Re-verified 2026-10-29 03:0x AWST (stocktake): State column rebuilt from `tailscale status --json` — **11 rows changed**. Two were **state transitions**, not age drift: **`sftpgo` (CT299, pve) came back online** (the 2026-10-28 audit had recorded it as `offline <1d`; live `tailscale ping 100.88.223.70` → `pong from sftpgo via 192.168.178.92:41641 in 0s`, `pct exec 299 -- systemctl is-active sftpgo` → `active`, `pct exec 299 -- tailscale ip -4` → `100.88.223.70`), and **`m93p` is online despite a stale peer record** — `tailscale status --json` still reports `Online: false` with `LastSeen 2026-10-28T18:40:00Z`, but `tailscale ping 100.104.51.98` → `pong from m93p via 192.168.178.118:41641 in 1ms` and SSH over the tailnet returned `m93p / up 11 days`. This is the **second** occurrence of the m93p stale-peer-record pattern (first seen 2026-10-26) — the peer record is not authoritative for hypervisor reachability. The remaining **9 rows were offline-age drift of exactly +1 day** (anthonys-imac 34d → 35d; cachyos 54d → 55d; cloudreve 95d → 96d — CT209 stopped; elementsynapse 98d → 99d — CT555 stopped; freshrss-new 71d → 72d; mattermost 38d → 39d — CT255 stopped; omarchybook 6d → 7d; omarchymac 10d → 11d; omni 71d → 72d — CT601 removed from both hypervisors; openmausbot 6d → 7d). Node count unchanged at **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself); `tailscale status --json` reports **76 peers total — 53 with a `100.` IPv4, 23 IPv6-only** (the `funnel-ingress-node` set). **Nodes in the table that are no longer in the tailnet: none. Live IPv4 peers absent from the table: none. IP mismatches: none.** Container identity re-confirmed from **inside** every LXC via `pct exec <vmid> -- tailscale ip -4` across **both** hypervisors (`pve` 100.96.100.82, `m93p` 100.104.51.98) — 34 guests enumerated; every peer IP mapped to its CT as listed, and the three stopped guests (`CT209` cloudreve, `CT255` mattermost, `CT555` elementsynapse) and the three running-but-unconnected guests (`CT105` conduit, `CT1264` openmausbot, `CT100` obsidian-livesync) confirmed. CT601 remains absent from both hypervisors.
Re-verified 2026-10-28 00:2x (nightly audit): State column rebuilt from `tailscale status --json` — **14 rows changed** (anthonys-imac: idle/offline 32d -> offline 34d; cachyos: idle/offline 52d -> offline 54d; cloudreve: idle/offline 93d — CT209 stopped -> offline 95d — CT209 stopped; conduit: idle/offline 49d -> offline 52d; elementsynapse: idle/offline 96d — CT555 stopped -> offline 98d — CT555 stopped; freshrss-new: idle/offline 69d -> offline 71d; m93p: online -> offline <1d; mattermost: idle/offline 36d — CT255 stopped -> offline 38d — CT255 stopped; omarchybook: idle/offline 4d -> offline 6d; omarchymac: idle/offline 8d -> offline 10d; omni: idle/offline 69d — CT601 removed from both hypervisors -> offline 71d — CT601 removed from both hypervisors; openmausbot: idle/offline 4d -> offline 6d; sftpgo: online — CT299 (pve) -> offline <1d — CT299 (pve); ubuntu-server-xfce: idle/offline 66d -> offline 68d). Header count corrected to **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself). Nodes in the table that are no longer in the tailnet: **none**. Live IPv4 peers absent from the table: **none**. **State transitions inside the audit window (not just age drift): `m93p` online → offline <1d, `sftpgo` (CT299) online → offline <1d.**
Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale status --json` — **`springroll-browser` (100.109.221.91) was missing entirely** and has been added. **`omarchybook` (100.107.9.100) and `openmausbot` (100.91.214.91) have both gone offline** (online → idle/offline 3d) and `openmausbot` is no longer a live peer despite its m93p LXC (CT1264) still running `tailscale` — the node itself has stopped connecting. Every offline age was refreshed (e.g. `mattermost` 31d → **35d**, `omni` 64d → **68d**, `elementsynapse` 91d → **95d**, `ubuntu-server-xfce` 60d → **64d**, `omarchymac` 3d → **7d**). `m93p` (100.104.51.98) now reads **idle/offline 13m ago** — the peer record is stale, but the host is plainly reachable (both `pct list` probes and the SSH session to CT201/CT460 during this audit succeeded), so treat it as a Tailscale-side liveness artefact, not a hypervisor outage. `homeassistant` still advertises `ExitNodeOption: true` but `ExitNode: false` — it **offers** exit-node capability; it is not currently serving as one (the old "active; offers exit node" wording overstated the live state). Container states cross-checked again against live `pct list` on both hypervisors, and each peer's container identity confirmed by running `tailscale ip -4` **inside** the candidate LXC on both hosts — the authoritative mapping, since peer names and LXC names do not always match. Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale status --json` — **`springroll-browser` (100.109.221.91) was missing entirely** and has been added. **`omarchybook` (100.107.9.100) and `openmausbot` (100.91.214.91) have both gone offline** (online → idle/offline 3d) and `openmausbot` is no longer a live peer despite its m93p LXC (CT1264) still running `tailscale` — the node itself has stopped connecting. Every offline age was refreshed (e.g. `mattermost` 31d → **35d**, `omni` 64d → **68d**, `elementsynapse` 91d → **95d**, `ubuntu-server-xfce` 60d → **64d**, `omarchymac` 3d → **7d**). `m93p` (100.104.51.98) now reads **idle/offline 13m ago** — the peer record is stale, but the host is plainly reachable (both `pct list` probes and the SSH session to CT201/CT460 during this audit succeeded), so treat it as a Tailscale-side liveness artefact, not a hypervisor outage. `homeassistant` still advertises `ExitNodeOption: true` but `ExitNode: false` — it **offers** exit-node capability; it is not currently serving as one (the old "active; offers exit node" wording overstated the live state). Container states cross-checked again against live `pct list` on both hypervisors, and each peer's container identity confirmed by running `tailscale ip -4` **inside** the candidate LXC on both hosts — the authoritative mapping, since peer names and LXC names do not always match.
| Node | IP | State | | Node | IP | State |
|---|---|---| |---|---|---|
| hermes | 100.118.5.51 | **self** — CT460 (pve) | | hermes | 100.118.5.51 | **self** — CT460 (pve) |
| agentzero | 100.106.54.94 | online — CT475 (pve) | | agentzero | 100.106.54.94 | online — CT475 (pve) |
| anthonys-imac | 100.112.123.124 | idle/offline 32d |
| anthonys-ipad | 100.111.198.120 | online | | anthonys-ipad | 100.111.198.120 | online |
| anthonys-iphone | 100.100.116.67 | online | | anthonys-iphone | 100.100.116.67 | online |
| anthonys-mac-mini | 100.74.17.15 | online | | anthonys-mac-mini | 100.76.60.37 | online — **returned 2026-11-06** (peer record `Online: true`, `tailscale ping` → `pong from anthonys-mac-mini (100.76.60.37) via DERP(syd) in 103ms`; client device, not an estate host) |
| cachyos | 100.124.208.15 | idle/offline 52d |
| clawtest | 100.122.189.81 | online — CT333 (pve) | | clawtest | 100.122.189.81 | online — CT333 (pve) |
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) | | cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
| cloudreve | 100.111.187.7 | idle/offline 93d — CT209 stopped | | cloudreve | 100.111.187.7 | offline 110d — CT209 stopped |
| conduit | 100.81.79.87 | idle/offline 49d | | codex-environment | 100.111.126.78 | offline 4d — last seen 2026-11-07 11:09Z; **created 2026-11-07 05:5xZ**, six-record burst; **not an LXC/VM on either hypervisor** (orphaned/ephemeral tailnet peer; added 2026-11-10, last-seen stamps corrected 2026-11-10) |
| codex-environment-1b88dc | 100.76.42.76 | offline 4d — last seen 2026-11-07 11:12Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| codex-environment-8dadbb | 100.65.200.27 | offline 4d — last seen 2026-11-07 11:46Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| codex-environment-955866 | 100.81.119.118 | offline 4d — last seen 2026-11-07 11:49Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| codex-environment-9fbd61 | 100.106.215.121 | offline 4d — last seen 2026-11-07 11:51Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| codex-environment-d3edbd | 100.74.3.2 | offline 4d — last seen 2026-11-07 11:56Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| conduit | 100.81.79.87 | offline 66d — last seen 2026-09-06 01:32Z |
| docker | 100.96.150.69 | online — CT200 (pve) | | docker | 100.96.150.69 | online — CT200 (pve) |
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) | | docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
| elementsynapse | 100.75.250.37 | idle/offline 96d — CT555 stopped | | elementsynapse | 100.75.250.37 | offline 113d — CT555 stopped |
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) | | freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
| freshrss | 100.83.60.93 | online — CT1260 (m93p) | | freshrss | 100.83.60.93 | online — CT1260 (m93p) |
| freshrss-new | 100.110.180.115 | idle/offline 69d | | freshrss-new | 100.110.180.115 | offline 86d — last seen 2026-08-17 17:04Z |
| gitea | 100.82.100.128 | online — CT221 (m93p) | | gitea | 100.82.100.128 | online — CT221 (m93p) |
| guest-vps | 100.117.14.84 | online | | guest-vps | 100.117.14.84 | online — VM101 (pve) |
| hermesagent | 100.98.205.112 | online — CT450 (pve) | | hermesagent | 100.98.205.112 | online — CT450 (pve) |
| homeassistant | 100.68.139.5 | online | | homeassistant | 100.68.139.5 | online — VM100 (pve) |
| litellm | 100.79.29.64 | online — CT500 (m93p) | | litellm | 100.79.29.64 | online — CT500 (m93p) |
| m93p | 100.104.51.98 | online | | m93p | 100.104.51.98 | online — hypervisor (peer record reads `Online: true`; `tailscale ping` → pong) |
| mailflow | 100.115.60.92 | online — CT1262 (m93p) | | mailflow | 100.115.60.92 | online — CT1262 (m93p) — ⚠ peer record reads `Online: false` (`LastSeen 2026-11-10T13:59:56Z`, offline 1d), but `tailscale ping` → **`pong from mailflow (100.115.60.92) via 192.168.178.142:41641 in 2ms`** and live `pct list` on `m93p` shows **CT1262 running** — stale peer record, not an outage (re-probed 2026-11-12) |
| mattermost | 100.98.116.14 | idle/offline 36d — CT255 stopped | | mattermost | 100.98.116.14 | online — CT255 (pve) |
| n8n | 100.84.124.30 | online | | mem0 | 100.73.94.53 | online — CT107 (m93p) |
| n8n | 100.84.124.30 | online — CT215 (pve) |
| nextcloud | 100.90.102.19 | online — CT270 (pve) | | nextcloud | 100.90.102.19 | online — CT270 (pve) |
| ntfy | 100.67.153.116 | online — CT1241 (m93p) | | ntfy | 100.67.153.116 | online — CT1241 (m93p) |
| octoagent | 100.117.10.68 | online | | obsidian-livesync | 100.116.244.84 | online — CT100 (m93p) |
| octop | 100.92.163.122 | online | | octoagent | 100.117.10.68 | offline 10d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
| octop | 100.92.163.122 | **online** (re-verified 2026-11-12 00:4x AWST — peer record `Online: true`; still **not an LXC/VM on either hypervisor** — orphaned tailnet peer) |
| ollama | 100.108.44.119 | online — CT205 (pve) | | ollama | 100.108.44.119 | online — CT205 (pve) |
| omarchy | 100.73.227.103 | online | | omarchy | 100.73.227.103 | online — VM110 (pve) |
| omarchybook | 100.107.9.100 | idle/offline 4d | | omarchybook | 100.107.9.100 | offline 21d — last seen 2026-10-21 15:55Z |
| omarchymac | 100.84.180.69 | idle/offline 8d | | omarchymac | 100.126.173.115 | offline 4d — last seen 2026-11-07 06:41Z (⚠ **was stale at 2d / 2026-11-08 00:45Z**; corrected 2026-11-12 — the peer was last seen *earlier* than the 2026-11-10 row claimed and `tailscale ping` → timeout, so this is a genuine offline, not a stale record) |
| omarchymac-1 | 100.126.173.115 | online |
| omni | 100.93.204.4 | idle/offline 69d — CT601 removed from both hypervisors |
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute | | omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) | | omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
| openmausbot | 100.91.214.91 | idle/offline 4d | | openmausbot | 100.91.214.91 | offline 21d — CT1264 (m93p) stopped; last seen 2026-10-21 15:50Z |
| openmausbot-vm | 100.77.107.30 | online | | openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) |
| openweb | 100.69.230.13 | online — CT245 (m93p) | | openweb | 100.69.230.13 | online — CT245 (m93p) |
| paseo | 100.72.188.93 | online — CT1261 (m93p) | | paseo | 100.72.188.93 | online — CT1261 (m93p) |
| pve | 100.96.100.82 | online | | pve | 100.96.100.82 | online |
| qwenpaw | 100.110.72.126 | online — CT465 (pve) | | qwenpaw | 100.110.72.126 | online — CT465 (pve) — ⚠ peer record reads `Online: false` (`LastSeen 2026-11-11T12:20:00Z`), but `tailscale ping` → **`pong from qwenpaw (100.110.72.126) via 192.168.178.186:41641 in 7ms`** and live `pct list` on `pve` shows **CT465 running** — stale peer record, not an outage (re-probed 2026-11-12) |
| rclone | 100.107.116.17 | online — CT104 (m93p) | | rclone | 100.107.116.17 | online — CT104 (m93p) |
| runtipi | 100.96.244.39 | online — CT201 (pve) | | runtipi | 100.96.244.39 | online — CT201 (pve) |
| rustdesk | 100.95.227.57 | online — CT1008 (m93p) | | rustdesk | 100.95.227.57 | online — CT1008 (m93p) |
| sftpgo | 100.88.223.70 | online — CT299 (pve) | | sftpgo | 100.88.223.70 | online — CT299 (pve) — **recovered 2026-11-04 (freeze 1) and again 2026-11-05 (freeze 2, manual `ssd0` run)**: a wedged `vzdump` had frozen the guest for ~24 h; killing the task unfroze it (see [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]) — **re-confirmed online 2026-11-12** (`tailscale ping` → `pong from sftpgo (100.88.223.70) via 192.168.178.92:41641 in 0s`, `pct list` → CT299 running, healthz → 401) |
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) | | social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
| springroll | 100.100.153.125 | online | | springroll | 100.100.153.125 | online — ⚠ peer record reads `Online: false` (`LastSeen 2026-11-11T08:30:00Z`), but `tailscale ping` → **`pong from springroll (100.100.153.125) via DERP(sin) in 210ms`** (answers via relay, not direct); not an LXC/VM on either hypervisor (client device) — re-probed 2026-11-12 |
| springroll-browser | 100.109.221.91 | online | | springroll-browser | 100.109.221.91 | online |
| ubuntu-server-xfce | 100.101.26.104 | idle/offline 66d | | ubuntu-server-xfce | 100.101.26.104 | offline 82d — last seen 2026-08-20 19:36Z — VM282 (pve) running |
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) | | vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
| webserver | 100.82.40.23 | online — CT207 (pve) | | webserver | 100.82.40.23 | online — CT207 (pve) |
@@ -92,16 +107,16 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| Path | Target | Notes | | Path | Target | Notes |
|---|---|---| |---|---|---|
| `/` | `http://127.0.0.1:9119` | Hermes WebUI | | `/` | `http://127.0.0.1:9119` | Hermes WebUI |
| `/ui` | `http://127.0.0.1:5175/` | Hermes UI (Vite) | | `/ui` | `http://127.0.0.1:5175` | Hermes UI (Vite) |
| `/jobs` | `http://127.0.0.1:9099` | Jobs Dashboard | | `/jobs` | `http://127.0.0.1:9099` | Jobs Dashboard |
| `/wiki` | `http://100.118.5.51:9120` | Agent Estate Wiki (Quartz) | | `/wiki` | `http://100.118.5.51:9120` | Agent Estate Wiki (Quartz) |
| `/ignis` | `http://127.0.0.1:8080` | Ignis (browser-native Obsidian) | | `/ignis` | `http://127.0.0.1:8080` | Ignis (browser-native Obsidian) |
| `/webui` | `http://127.0.0.1:8787/` | Hermes WebUI (Standalone) | | `/webui` | `http://127.0.0.1:8787` | Hermes WebUI (Standalone) |
| `/gitea` | `http://100.82.100.128:3000` | Gitea | | `/gitea` | `http://100.82.100.128:3000` | Gitea |
| `/desktop` | `http://127.0.0.1:6081` | CUA noVNC | | `/desktop` | `http://127.0.0.1:6081` | CUA noVNC |
| `/vnc-camofox` | `http://127.0.0.1:6080` | Camofox VNC | | `/vnc-camofox` | `http://127.0.0.1:6080` | Camofox VNC |
| `/synergy-prep` | `http://127.0.0.1:9123/` | Synergy prep | | `/synergy-prep` | `http://127.0.0.1:9123` | Synergy prep — ⚠ **DEAD: nothing listening on 9123** (live `ss -ltn` probe 2026-11-06, **re-confirmed 2026-11-07 and 2026-11-12**; the route still exists in `tailscale serve status` but serves 502) |
| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) | | `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) — ⚠ **DEAD: nothing listening on 8769** (live `ss -ltn` probe 2026-11-06, **re-confirmed 2026-11-07 and 2026-11-12**; route still present in `tailscale serve status` but serves 502) |
| `/evening-grid.html` | `http://127.0.0.1:9099/evening-grid.html` | Evening grid | | `/evening-grid.html` | `http://127.0.0.1:9099/evening-grid.html` | Evening grid |
Re-verified against `tailscale serve status` on **2026-10-26** — the CT460 route table above matched live exactly (**12 paths on :443**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; no drift). Re-verified against `tailscale serve status` on **2026-10-26** — the CT460 route table above matched live exactly (**12 paths on :443**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; no drift).
@@ -115,19 +130,23 @@ Re-verified against `tailscale serve status` on **2026-10-26** — the CT460 rou
Re-verified 2026-10-26 (stocktake): the node table was rebuilt again from `tailscale status --json` — **`octoagent` (100.117.10.68) and `omarchymac-1` (100.126.173.115) were both missing entirely** and have been added; `m93p` (100.104.51.98) now reads **online** (it was "idle/offline 13m ago" — a stale Tailscale peer record, and the host was plainly reachable throughout); every offline age was refreshed (`mattermost` 35d → **36d**, `omni` 68d → **69d**, `elementsynapse` 95d → **96d**, `ubuntu-server-xfce` 64d → **66d**, `omarchymac` 7d → **8d**, `omarchybook` 3d → **4d**, `openmausbot` 3d → **4d**). The `omni` row now reads **"CT601 removed from both hypervisors"** rather than "CT601 stopped" — live `pct list` on both `pve` and `m93p` shows **no CT601 at all**, so the old guest is gone, not merely halted. Container identity re-confirmed from **inside** each LXC via `pct exec <vmid> -- tailscale ip -4` on both hypervisors; every peer whose LXC has no Tailscale IP was checked against `pct list` (`CT105`/`CT1264` run but do not connect — `conduit` and `openmausbot` offline; `CT209`/`CT255`/`CT555` are `stopped`). Serve tables re-probed per host with `pct exec 201 -- tailscale serve status`. Re-verified 2026-10-26 (stocktake): the node table was rebuilt again from `tailscale status --json` — **`octoagent` (100.117.10.68) and `omarchymac-1` (100.126.173.115) were both missing entirely** and have been added; `m93p` (100.104.51.98) now reads **online** (it was "idle/offline 13m ago" — a stale Tailscale peer record, and the host was plainly reachable throughout); every offline age was refreshed (`mattermost` 35d → **36d**, `omni` 68d → **69d**, `elementsynapse` 95d → **96d**, `ubuntu-server-xfce` 64d → **66d**, `omarchymac` 7d → **8d**, `omarchybook` 3d → **4d**, `openmausbot` 3d → **4d**). The `omni` row now reads **"CT601 removed from both hypervisors"** rather than "CT601 stopped" — live `pct list` on both `pve` and `m93p` shows **no CT601 at all**, so the old guest is gone, not merely halted. Container identity re-confirmed from **inside** each LXC via `pct exec <vmid> -- tailscale ip -4` on both hypervisors; every peer whose LXC has no Tailscale IP was checked against `pct list` (`CT105`/`CT1264` run but do not connect — `conduit` and `openmausbot` offline; `CT209`/`CT255`/`CT555` are `stopped`). Serve tables re-probed per host with `pct exec 201 -- tailscale serve status`.
## CT201 Serve Routes ## CT201 Serve Routes
Verified live on 2026-10-26 via `pct exec 201 -- tailscale serve status` on `pve` (100.96.100.82). The previous version of this table was wrong in two ways and has been corrected. Re-verified live on 2026-10-30 via `pct exec 201 -- tailscale serve status` on `pve` (100.96.100.82). The previous version of this table was wrong in two ways and has been corrected.
| Host / Port | Path | Target | Notes | | Host / Port | Path | Target | Notes |
|---|---|---|---| |---|---|---|---|
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Homepage asset proxy front door — **502** (re-probed 2026-10-26; upstream 8082 has no listener) | | `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Homepage asset proxy front door — **502** (re-probed 2026-10-30; upstream 8082 has no listener — `curl http://127.0.0.1:8082/` → exit code 7 / `http=000`) |
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` container — **200** | | `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` container — **200** (re-probed 2026-10-30) |
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Same asset proxy — **502** (re-probed 2026-10-26) | | `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Same asset proxy — **502** (re-probed 2026-10-30) |
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` — **200** | | `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` — **200** (re-probed 2026-10-30) |
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/steel` | `http://127.0.0.1:5173` | Steel Browser UI — **200** on :443 (re-probed 2026-10-26). ⚠ **`/steel` is NOT served on :8443** — it 502s there |
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/steel-api` | `http://127.0.0.1:3000` | Steel Browser API (path prefix not stripped — use direct IP for API calls) | **Steel routes removed 2026-10-29.** This table previously carried two more rows on `:443` — `/steel` →
`http://127.0.0.1:5173` and `/steel-api` → `http://127.0.0.1:3000`. Both were deleted with
`tailscale serve --set-path <path> off` when Steel was removed from CT201, so CT201's only remaining routes
are `/` → `127.0.0.1:8085` and `/mgr` → `http://127.0.0.1:8084`, on **both** `:443` and `:8443`.
→ [[comparisons/steel-vs-camoufox]]
Two corrections to the pre-2026-10-25 table: Two corrections to the pre-2026-10-25 table:
1. **`:8443` is served by `tailscaled` on CT201, but so is `:443`.** `ss -ltnp` on CT201 shows `tailscaled` bound to `100.96.244.39:443`, and the serve table above lists routes on **both** ports. The old note claiming "Port 443 occupied by Traefik on CT201" is **false as of 2026-10-25** — no Traefik container or process exists on CT201 (`docker ps -a` and `ps aux` both came back empty for `traefik`), and Tailscale owns 443 there. The Homebox/steel path prefix is not stripped on `/steel-api`. 1. **`:8443` is served by `tailscaled` on CT201, but so is `:443`.** `ss -ltnp` on CT201 shows `tailscaled` bound to `100.96.244.39:443`, and the serve table above lists routes on **both** ports. The old note claiming "Port 443 occupied by Traefik on CT201" is **false as of 2026-10-25** — no Traefik **process** runs on CT201 (`ps aux` returns nothing for `traefik`), and Tailscale owns 443 there. *(Evidence corrected 2026-10-30: the original wording said `docker ps -a` "came back empty for `traefik`" — it does not. Live `docker ps -a` lists `runtipi-reverse-proxy | traefik:v3.2 | Exited (255) 2 months ago`. The container is **stopped**, not absent; the conclusion that Traefik does not occupy 443 stands.)*
2. The old table routed `:8443` straight to `http://localhost:8082`. That port has **no listener**; the live target is `8085` (the `asset_proxy.py` front door), which itself proxies to the dead 8082 and returns 502. → [[systems/homepage-dashboard]] 2. The old table routed `:8443` straight to `http://localhost:8082`. That port has **no listener**; the live target is `8085` (the `asset_proxy.py` front door), which itself proxies to the dead 8082 and returns 502. → [[systems/homepage-dashboard]]
## Atomic Restore Script ## Atomic Restore Script
@@ -152,7 +171,7 @@ tailscale serve --set-path /jobs off
``` ```
## Notes ## Notes
- **CT201 now serves Tailscale on BOTH `:443` and `:8443`** (verified 2026-10-25). The old "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there" claim was **wrong**: no Traefik container or process exists on CT201, and `tailscaled` owns 443 there (`ss -ltnp` shows it bound to `100.96.244.39:443`). The CT201 Serve Routes table above now records both ports. - **CT201 now serves Tailscale on BOTH `:443` and `:8443`** (verified 2026-10-30). The old "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there" claim was **wrong**: no Traefik **process** runs on CT201, and `tailscaled` owns 443 there (`ss -ltnp` shows it bound to `100.96.244.39:443`). The CT201 Serve Routes table above now records both ports. *(Evidence corrected 2026-10-30: the original wording said "no Traefik container or process exists on CT201" — the container does exist, `runtipi-reverse-proxy | traefik:v3.2`, but it is **stopped** — `Exited (255) 2 months ago`. The conclusion that Tailscale owns 443 is unchanged.)*
- CT460 has no conflict — Tailscale uses 443 directly, with the public webhook Funnel on `:8443`. - CT460 has no conflict — Tailscale uses 443 directly, with the public webhook Funnel on `:8443`.
- Routes must be re-added in order (restore script handles this). - Routes must be re-added in order (restore script handles this).
- After every rebuild, verify that `tailscale serve status` contains both the `:443` Serve routes and the `:8443/webhook` Funnel route. - After every rebuild, verify that `tailscale serve status` contains both the `:443` Serve routes and the `:8443/webhook` Funnel route.
+1280 -772
View File
File diff suppressed because one or more lines are too long
+33 -6
View File
@@ -3,8 +3,8 @@ title: Retained Research Answers
type: query type: query
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-12 updated: 2026-11-04
verified_on: 2026-10-12 verified_on: 2026-11-04
review_after: 2027-01-12 review_after: 2027-01-12
confidence: medium confidence: medium
tags: [query, research, vnc, remote-desktop] tags: [query, research, vnc, remote-desktop]
@@ -18,11 +18,32 @@ Research answers worth keeping: what was asked, the answer, the evidence it rest
> The 2026-07-22 stub was filled on 2026-10-12 with the first retained answer (asked in the 2026-10-11 > The 2026-07-22 stub was filled on 2026-10-12 with the first retained answer (asked in the 2026-10-11
> interactive session `talaria-ios`, 41 messages). > interactive session `talaria-ios`, 41 messages).
>
> **2026-11-04 re-verification:** the answer's *framing* was wrong — it was written for "TigerVNC + Xtigervnc
> servers" that the estate does not run. The client-side advice remains valid for the one VNC server that does
> exist (x11vnc on CT460). See the correction note under the answer.
## Fastest VNC client (asked 2026-10-11) ## Fastest VNC client (asked 2026-10-11)
**Answer for the CT460/CT200 TigerVNC + Xtigervnc servers:** use the **TigerVNC native viewer** with **Answer for the CT460 VNC server:** use the **TigerVNC native viewer** with AutoSelect and the defaults —
AutoSelect and the defaults — free, and the fastest standard-RFB client. Lowest bandwidth is **KasmVNC** free, and the fastest standard-RFB client.
> ⚠ **Corrected 2026-11-04 — the original question was framed against servers the estate does not run.**
> This answer was written on 2026-10-12 for "the CT460/CT200 TigerVNC + Xtigervnc servers". **Neither exists.**
> Live state, re-verified 2026-11-04:
> - **CT460** runs **x11vnc**, not TigerVNC and not Xtigervnc: `x11vnc -display :99 -forever -shared -rfbport 5901
> -rfbportv6 5901 -noxdamage -quiet -nopw` (PID 1290295), fronted by **websockify** → noVNC on
> `127.0.0.1:6080 → 5900` (a **dead** target — no listener on 5900) and `127.0.0.1:6081 → 5901` (**live**).
> The display itself is `Xvfb :99 -screen 0 1600x1200x24 -ac -nolisten tcp`.
> - **CT200** runs **no VNC server at all**. Its only X server is `Xvfb :495478600 -screen 0 800x680x24`, started
> 2026-10-30 — that is the Camofox browser's own headless Xvfb, and the Camofox VNC plugin tree lives on
> **CT460** (`/home/hermes/services/camofox-browser/plugins/vnc/`), not on CT200. No `x11vnc`, no `Xtigervnc`,
> no `vncserver`, and nothing listening on 5900/5901/6080/6081.
>
> **The advice itself still stands**, because x11vnc speaks standard RFB and TigerVNC's viewer is the right
> client for it — but it now applies to **one** host, and the page's "the estate's own VNC/Xtigervnc setup" pointer
> is wrong. The real setup is documented at [[systems/headless-server]], which already described x11vnc correctly
> back on 2026-10-20; this query page was the outlier. Lowest bandwidth is **KasmVNC**
(WebP, ~30 % less) but that is a *server* swap, not a client. **RealVNC Connect** is not a faster decoder, it is (WebP, ~30 % less) but that is a *server* swap, not a client. **RealVNC Connect** is not a faster decoder, it is
a forked protocol (RFB 6, audio, no port-forwarding) and needs a RealVNC server. **TurboVNC** remains the a forked protocol (RFB 6, audio, no port-forwarding) and needs a RealVNC server. **TurboVNC** remains the
reference for lossless 3D/CAD. Genuinely low latency means leaving RFB entirely: **Parsec** or reference for lossless 3D/CAD. Genuinely low latency means leaving RFB entirely: **Parsec** or
@@ -66,8 +87,14 @@ software (a thin client with no SIMD headroom decodes slowly regardless of viewe
**Does NOT establish:** exact throughput multipliers on current hardware (the benchmark is 2012); no **Does NOT establish:** exact throughput multipliers on current hardware (the benchmark is 2012); no
end-to-end benchmark was run against Anthony's own hosts. If speed becomes a real complaint, measure the end-to-end benchmark was run against Anthony's own hosts. If speed becomes a real complaint, measure the
existing server before changing clients. existing server before changing clients. **It also does not establish anything about TigerVNC or Xtigervnc
*server*-side behaviour on this estate, because the estate runs neither** — x11vnc's own encoder settings
(`-compress`, `-quality`, `-encodings`) are the live lever on CT460, not the TigerVNC/TurboVNC encoder options
the report discusses. The compression-level and local-cursor levers above map onto x11vnc as
`-compress <level>` and the `x11vnc` cursor-shape options, but the specific numbers were tuned for a different
server and should be re-measured before being trusted on this one.
## Related ## Related
- [[systems/hermes-agent]] — the estate's own VNC/Xtigervnc setup lives under [[systems/headless-server]] - [[systems/headless-server]] — **the estate's actual VNC setup** (x11vnc on `Xvfb :99`, noVNC via websockify) — re-verified 2026-11-04
- [[systems/hermes-agent]] — the agent host this answer was researched for
- [[log]] — audit entries - [[log]] — audit entries
+121
View File
@@ -0,0 +1,121 @@
# Wiki freshness audit — 2026-10-27
Threshold: >21 days since `updated`
## STALE (0)
## FRESH (104)
- [ 7d] CONTRIBUTING.md
- [ 1d] agents/hermes-chaos.md
- [ 1d] agents/hermes-production.md
- [ 7d] agents/index.md
- [ 16d] agents/nanobot.md
- [ 7d] agents/open-claw.md
- [ 16d] agents/qwenpaw.md
- [ 7d] agents/zeroclaw.md
- [ 7d] comparisons/camoufox-vs-chrome.md
- [ 7d] comparisons/hermes-vs-nanobot.md
- [ 7d] comparisons/memory-backends.md
- [ 7d] comparisons/model-routers.md
- [ 14d] comparisons/steel-vs-camoufox.md
- [ 7d] concepts/agent-memory.md
- [ 14d] concepts/browser-automation.md
- [ 7d] concepts/model-routing.md
- [ 7d] concepts/persistent-shells.md
- [ 7d] concepts/tool-calling.md
- [ 0d] current-state.md
- [ 7d] decisions/index.md
- [ 7d] decisions/keep-openrouter-as-fallback.md
- [ 7d] decisions/persistent-shell-enabled.md
- [ 7d] decisions/rollback-steel-to-camofox.md
- [ 7d] experiments/index.md
- [ 3d] incidents/2026-07-29-mnemosyne-relay.md
- [ 7d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 3d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 5d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 4d] incidents/2026-08-15-skills-index-drift.md
- [ 15d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 2d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 7d] incidents/2026-08-29-gmail-read-state.md
- [ 6d] incidents/2026-09-02-sftpgo-outage.md
- [ 16d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 21d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 1d] incidents/2026-10-03-ct450-hermes-update.md
- [ 1d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 21d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 3d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 18d] incidents/2026-10-09-memory-recall-relevance.md
- [ 11d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 0d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 8d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 0d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 5d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 2d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 1d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 1d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 1d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 0d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 5d] infrastructure/backups.md
- [ 7d] infrastructure/credentials-map.md
- [ 16d] infrastructure/docker-services.md
- [ 7d] infrastructure/domains-and-tunnels.md
- [ 1d] infrastructure/hosts.md
- [ 13d] infrastructure/network-map.md
- [ 13d] infrastructure/proxmox.md
- [ 1d] infrastructure/tailscale.md
- [ 15d] queries/retained-research-answers.md
- [ 2d] reports/communities-wa-social-media-audit-2026.md
- [ 6d] reports/uber-eats/2026-07.md
- [ 6d] reports/uber-eats/2026-08.md
- [ 6d] reports/uber-eats/2026-09.md
- [ 6d] reports/uber-eats/index.md
- [ 7d] runbooks/backup-wiki.md
- [ 7d] runbooks/diagnose-docker.md
- [ 7d] runbooks/gateway-resource-resilience.md
- [ 0d] runbooks/job-radar.md
- [ 7d] runbooks/multiplexer-setup.md
- [ 1d] runbooks/provider-health.md
- [ 7d] runbooks/recover-docker-service.md
- [ 7d] runbooks/restart-browser.md
- [ 7d] runbooks/restart-hermes.md
- [ 7d] runbooks/restore-openrouter.md
- [ 7d] runbooks/test-openrouter.md
- [ 7d] runbooks/troubleshoot-tool-loop.md
- [ 1d] runbooks/update-hermes-safely.md
- [ 14d] systems/browser-backend.md
- [ 7d] systems/cloudflare-tunnel.md
- [ 1d] systems/conduit.md
- [ 1d] systems/current-profile.md
- [ 7d] systems/guanaco.md
- [ 7d] systems/headless-server.md
- [ 1d] systems/hermes-agent.md
- [ 7d] systems/hermes-ui.md
- [ 15d] systems/home-assistant.md
- [ 2d] systems/homepage-dashboard.md
- [ 16d] systems/ignis.md
- [ 1d] systems/memory-backend.md
- [ 1d] systems/messaging-integrations.md
- [ 3d] systems/mnemosyne.md
- [ 1d] systems/model-providers.md
- [ 15d] systems/nanobot.md
- [ 7d] systems/nextcloud.md
- [ 7d] systems/obsidian.md
- [ 5d] systems/omniroute.md
- [ 7d] systems/opencode-go.md
- [ 7d] systems/openrouter.md
- [ 16d] systems/paseo.md
- [ 8d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 7d] systems/terminal-backend.md
- [ 7d] systems/tool-search.md
## NO updated DATE (4)
- SCHEMA.md
- experiments/active/example.md
- index.md
- log.md
TOTAL: 0 stale, 104 fresh, 4 no-date of 108 pages
+122
View File
@@ -0,0 +1,122 @@
# Wiki freshness audit — 2026-10-28
Threshold: >21 days since `updated`
## STALE (2)
- [ 22d] incidents/2026-10-01-cliproxy-connection-auth-expired.md (updated 2026-10-06)
- [ 22d] incidents/2026-10-05-dead-model-pin-repin.md (updated 2026-10-06)
## FRESH (103)
- [ 8d] CONTRIBUTING.md
- [ 2d] agents/hermes-chaos.md
- [ 2d] agents/hermes-production.md
- [ 8d] agents/index.md
- [ 17d] agents/nanobot.md
- [ 8d] agents/open-claw.md
- [ 17d] agents/qwenpaw.md
- [ 8d] agents/zeroclaw.md
- [ 8d] comparisons/camoufox-vs-chrome.md
- [ 8d] comparisons/hermes-vs-nanobot.md
- [ 8d] comparisons/memory-backends.md
- [ 8d] comparisons/model-routers.md
- [ 15d] comparisons/steel-vs-camoufox.md
- [ 8d] concepts/agent-memory.md
- [ 15d] concepts/browser-automation.md
- [ 8d] concepts/model-routing.md
- [ 8d] concepts/persistent-shells.md
- [ 8d] concepts/tool-calling.md
- [ 0d] current-state.md
- [ 8d] decisions/index.md
- [ 8d] decisions/keep-openrouter-as-fallback.md
- [ 8d] decisions/persistent-shell-enabled.md
- [ 8d] decisions/rollback-steel-to-camofox.md
- [ 8d] experiments/index.md
- [ 4d] incidents/2026-07-29-mnemosyne-relay.md
- [ 8d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 4d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 6d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 5d] incidents/2026-08-15-skills-index-drift.md
- [ 16d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 3d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 8d] incidents/2026-08-29-gmail-read-state.md
- [ 7d] incidents/2026-09-02-sftpgo-outage.md
- [ 17d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 2d] incidents/2026-10-03-ct450-hermes-update.md
- [ 2d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 4d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 19d] incidents/2026-10-09-memory-recall-relevance.md
- [ 12d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 1d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 9d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 1d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 6d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 3d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 2d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 2d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 2d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 0d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 0d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 6d] infrastructure/backups.md
- [ 8d] infrastructure/credentials-map.md
- [ 17d] infrastructure/docker-services.md
- [ 8d] infrastructure/domains-and-tunnels.md
- [ 2d] infrastructure/hosts.md
- [ 14d] infrastructure/network-map.md
- [ 14d] infrastructure/proxmox.md
- [ 0d] infrastructure/tailscale.md
- [ 16d] queries/retained-research-answers.md
- [ 3d] reports/communities-wa-social-media-audit-2026.md
- [ 7d] reports/uber-eats/2026-07.md
- [ 7d] reports/uber-eats/2026-08.md
- [ 7d] reports/uber-eats/2026-09.md
- [ 7d] reports/uber-eats/index.md
- [ 8d] runbooks/backup-wiki.md
- [ 8d] runbooks/diagnose-docker.md
- [ 8d] runbooks/gateway-resource-resilience.md
- [ 0d] runbooks/job-radar.md
- [ 8d] runbooks/multiplexer-setup.md
- [ 2d] runbooks/provider-health.md
- [ 8d] runbooks/recover-docker-service.md
- [ 8d] runbooks/restart-browser.md
- [ 8d] runbooks/restart-hermes.md
- [ 8d] runbooks/restore-openrouter.md
- [ 8d] runbooks/test-openrouter.md
- [ 8d] runbooks/troubleshoot-tool-loop.md
- [ 2d] runbooks/update-hermes-safely.md
- [ 15d] systems/browser-backend.md
- [ 8d] systems/cloudflare-tunnel.md
- [ 0d] systems/conduit.md
- [ 2d] systems/current-profile.md
- [ 8d] systems/guanaco.md
- [ 8d] systems/headless-server.md
- [ 0d] systems/hermes-agent.md
- [ 8d] systems/hermes-ui.md
- [ 16d] systems/home-assistant.md
- [ 3d] systems/homepage-dashboard.md
- [ 17d] systems/ignis.md
- [ 2d] systems/memory-backend.md
- [ 2d] systems/messaging-integrations.md
- [ 4d] systems/mnemosyne.md
- [ 2d] systems/model-providers.md
- [ 16d] systems/nanobot.md
- [ 8d] systems/nextcloud.md
- [ 8d] systems/obsidian.md
- [ 6d] systems/omniroute.md
- [ 8d] systems/opencode-go.md
- [ 8d] systems/openrouter.md
- [ 17d] systems/paseo.md
- [ 9d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 8d] systems/terminal-backend.md
- [ 8d] systems/tool-search.md
## NO updated DATE (4)
- SCHEMA.md
- experiments/active/example.md
- index.md
- log.md
TOTAL: 2 stale, 103 fresh, 4 no-date of 109 pages
+123
View File
@@ -0,0 +1,123 @@
# Wiki freshness audit — 2026-10-29
Threshold: >21 days since `updated`
## STALE (0)
## FRESH (107)
- [ 9d] CONTRIBUTING.md
- [ 3d] agents/hermes-chaos.md
- [ 3d] agents/hermes-production.md
- [ 9d] agents/index.md
- [ 0d] agents/nanobot.md
- [ 9d] agents/open-claw.md
- [ 18d] agents/qwenpaw.md
- [ 9d] agents/zeroclaw.md
- [ 0d] comparisons/camoufox-vs-chrome.md
- [ 9d] comparisons/hermes-vs-nanobot.md
- [ 9d] comparisons/memory-backends.md
- [ 9d] comparisons/model-routers.md
- [ 0d] comparisons/steel-vs-camoufox.md
- [ 9d] concepts/agent-memory.md
- [ 0d] concepts/browser-automation.md
- [ 9d] concepts/model-routing.md
- [ 9d] concepts/persistent-shells.md
- [ 9d] concepts/tool-calling.md
- [ 0d] current-state.md
- [ 0d] decisions/index.md
- [ 9d] decisions/keep-openrouter-as-fallback.md
- [ 9d] decisions/persistent-shell-enabled.md
- [ 0d] decisions/rollback-steel-to-camofox.md
- [ 9d] experiments/index.md
- [ 5d] incidents/2026-07-29-mnemosyne-relay.md
- [ 9d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 5d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 7d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 6d] incidents/2026-08-15-skills-index-drift.md
- [ 17d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 4d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 9d] incidents/2026-08-29-gmail-read-state.md
- [ 8d] incidents/2026-09-02-sftpgo-outage.md
- [ 18d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 0d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 3d] incidents/2026-10-03-ct450-hermes-update.md
- [ 3d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 0d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 5d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 20d] incidents/2026-10-09-memory-recall-relevance.md
- [ 13d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 2d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 10d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 2d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 7d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 4d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 3d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 3d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 3d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 1d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 0d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 0d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 0d] index.md
- [ 7d] infrastructure/backups.md
- [ 9d] infrastructure/credentials-map.md
- [ 0d] infrastructure/docker-services.md
- [ 9d] infrastructure/domains-and-tunnels.md
- [ 0d] infrastructure/hosts.md
- [ 15d] infrastructure/network-map.md
- [ 15d] infrastructure/proxmox.md
- [ 0d] infrastructure/tailscale.md
- [ 17d] queries/retained-research-answers.md
- [ 4d] reports/communities-wa-social-media-audit-2026.md
- [ 8d] reports/uber-eats/2026-07.md
- [ 8d] reports/uber-eats/2026-08.md
- [ 8d] reports/uber-eats/2026-09.md
- [ 8d] reports/uber-eats/index.md
- [ 9d] runbooks/backup-wiki.md
- [ 9d] runbooks/diagnose-docker.md
- [ 9d] runbooks/gateway-resource-resilience.md
- [ 0d] runbooks/job-radar.md
- [ 9d] runbooks/multiplexer-setup.md
- [ 3d] runbooks/provider-health.md
- [ 0d] runbooks/recover-docker-service.md
- [ 9d] runbooks/restart-browser.md
- [ 9d] runbooks/restart-hermes.md
- [ 9d] runbooks/restore-openrouter.md
- [ 9d] runbooks/test-openrouter.md
- [ 9d] runbooks/troubleshoot-tool-loop.md
- [ 3d] runbooks/update-hermes-safely.md
- [ 0d] systems/browser-backend.md
- [ 9d] systems/cloudflare-tunnel.md
- [ 1d] systems/conduit.md
- [ 3d] systems/current-profile.md
- [ 9d] systems/guanaco.md
- [ 9d] systems/headless-server.md
- [ 1d] systems/hermes-agent.md
- [ 9d] systems/hermes-ui.md
- [ 17d] systems/home-assistant.md
- [ 0d] systems/homepage-dashboard.md
- [ 18d] systems/ignis.md
- [ 3d] systems/memory-backend.md
- [ 3d] systems/messaging-integrations.md
- [ 5d] systems/mnemosyne.md
- [ 3d] systems/model-providers.md
- [ 17d] systems/nanobot.md
- [ 9d] systems/nextcloud.md
- [ 9d] systems/obsidian.md
- [ 7d] systems/omniroute.md
- [ 9d] systems/opencode-go.md
- [ 9d] systems/openrouter.md
- [ 18d] systems/paseo.md
- [ 10d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 9d] systems/terminal-backend.md
- [ 9d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 0 stale, 107 fresh, 3 no-date of 110 pages
+125
View File
@@ -0,0 +1,125 @@
# Wiki freshness audit — 2026-10-30
Threshold: >21 days since `updated`
## STALE (0)
## FRESH (109)
- [ 10d] CONTRIBUTING.md
- [ 4d] agents/hermes-chaos.md
- [ 4d] agents/hermes-production.md
- [ 10d] agents/index.md
- [ 1d] agents/nanobot.md
- [ 10d] agents/open-claw.md
- [ 19d] agents/qwenpaw.md
- [ 10d] agents/zeroclaw.md
- [ 1d] comparisons/camoufox-vs-chrome.md
- [ 10d] comparisons/hermes-vs-nanobot.md
- [ 10d] comparisons/memory-backends.md
- [ 10d] comparisons/model-routers.md
- [ 1d] comparisons/steel-vs-camoufox.md
- [ 10d] concepts/agent-memory.md
- [ 1d] concepts/browser-automation.md
- [ 10d] concepts/model-routing.md
- [ 10d] concepts/persistent-shells.md
- [ 10d] concepts/tool-calling.md
- [ 0d] current-state.md
- [ 1d] decisions/index.md
- [ 10d] decisions/keep-openrouter-as-fallback.md
- [ 10d] decisions/persistent-shell-enabled.md
- [ 1d] decisions/rollback-steel-to-camofox.md
- [ 10d] experiments/index.md
- [ 6d] incidents/2026-07-29-mnemosyne-relay.md
- [ 10d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 6d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 8d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 7d] incidents/2026-08-15-skills-index-drift.md
- [ 18d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 5d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 10d] incidents/2026-08-29-gmail-read-state.md
- [ 9d] incidents/2026-09-02-sftpgo-outage.md
- [ 19d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 1d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 4d] incidents/2026-10-03-ct450-hermes-update.md
- [ 4d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 1d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 6d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 21d] incidents/2026-10-09-memory-recall-relevance.md
- [ 14d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 3d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 11d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 3d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 8d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 5d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 4d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 4d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 4d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 2d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 0d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 0d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 0d] incidents/2026-10-29-conduit-voice-silence.md
- [ 0d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 0d] index.md
- [ 8d] infrastructure/backups.md
- [ 10d] infrastructure/credentials-map.md
- [ 1d] infrastructure/docker-services.md
- [ 10d] infrastructure/domains-and-tunnels.md
- [ 0d] infrastructure/hosts.md
- [ 16d] infrastructure/network-map.md
- [ 16d] infrastructure/proxmox.md
- [ 0d] infrastructure/tailscale.md
- [ 18d] queries/retained-research-answers.md
- [ 5d] reports/communities-wa-social-media-audit-2026.md
- [ 9d] reports/uber-eats/2026-07.md
- [ 9d] reports/uber-eats/2026-08.md
- [ 9d] reports/uber-eats/2026-09.md
- [ 9d] reports/uber-eats/index.md
- [ 10d] runbooks/backup-wiki.md
- [ 10d] runbooks/diagnose-docker.md
- [ 10d] runbooks/gateway-resource-resilience.md
- [ 0d] runbooks/job-radar.md
- [ 10d] runbooks/multiplexer-setup.md
- [ 4d] runbooks/provider-health.md
- [ 1d] runbooks/recover-docker-service.md
- [ 10d] runbooks/restart-browser.md
- [ 10d] runbooks/restart-hermes.md
- [ 10d] runbooks/restore-openrouter.md
- [ 10d] runbooks/test-openrouter.md
- [ 10d] runbooks/troubleshoot-tool-loop.md
- [ 4d] runbooks/update-hermes-safely.md
- [ 1d] systems/browser-backend.md
- [ 10d] systems/cloudflare-tunnel.md
- [ 0d] systems/conduit.md
- [ 4d] systems/current-profile.md
- [ 10d] systems/guanaco.md
- [ 10d] systems/headless-server.md
- [ 0d] systems/hermes-agent.md
- [ 10d] systems/hermes-ui.md
- [ 18d] systems/home-assistant.md
- [ 0d] systems/homepage-dashboard.md
- [ 19d] systems/ignis.md
- [ 4d] systems/memory-backend.md
- [ 4d] systems/messaging-integrations.md
- [ 6d] systems/mnemosyne.md
- [ 4d] systems/model-providers.md
- [ 18d] systems/nanobot.md
- [ 10d] systems/nextcloud.md
- [ 10d] systems/obsidian.md
- [ 8d] systems/omniroute.md
- [ 10d] systems/opencode-go.md
- [ 10d] systems/openrouter.md
- [ 19d] systems/paseo.md
- [ 11d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 10d] systems/terminal-backend.md
- [ 10d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 0 stale, 109 fresh, 3 no-date of 112 pages
+127
View File
@@ -0,0 +1,127 @@
# Wiki freshness audit — 2026-10-31
Threshold: >21 days since `updated`
## STALE (1)
- [ 22d] incidents/2026-10-09-memory-recall-relevance.md (updated 2026-10-09)
## FRESH (110)
- [ 11d] CONTRIBUTING.md
- [ 5d] agents/hermes-chaos.md
- [ 5d] agents/hermes-production.md
- [ 11d] agents/index.md
- [ 2d] agents/nanobot.md
- [ 11d] agents/open-claw.md
- [ 20d] agents/qwenpaw.md
- [ 11d] agents/zeroclaw.md
- [ 2d] comparisons/camoufox-vs-chrome.md
- [ 11d] comparisons/hermes-vs-nanobot.md
- [ 11d] comparisons/memory-backends.md
- [ 11d] comparisons/model-routers.md
- [ 2d] comparisons/steel-vs-camoufox.md
- [ 11d] concepts/agent-memory.md
- [ 2d] concepts/browser-automation.md
- [ 11d] concepts/model-routing.md
- [ 11d] concepts/persistent-shells.md
- [ 11d] concepts/tool-calling.md
- [ 0d] current-state.md
- [ 2d] decisions/index.md
- [ 11d] decisions/keep-openrouter-as-fallback.md
- [ 11d] decisions/persistent-shell-enabled.md
- [ 2d] decisions/rollback-steel-to-camofox.md
- [ 11d] experiments/index.md
- [ 7d] incidents/2026-07-29-mnemosyne-relay.md
- [ 11d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 7d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 9d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 8d] incidents/2026-08-15-skills-index-drift.md
- [ 19d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 6d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 11d] incidents/2026-08-29-gmail-read-state.md
- [ 10d] incidents/2026-09-02-sftpgo-outage.md
- [ 20d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 2d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 5d] incidents/2026-10-03-ct450-hermes-update.md
- [ 5d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 2d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 7d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 15d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 4d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 12d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 4d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 9d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 6d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 5d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 5d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 5d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 3d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 1d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 1d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 1d] incidents/2026-10-29-conduit-voice-silence.md
- [ 0d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 1d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 0d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 0d] index.md
- [ 9d] infrastructure/backups.md
- [ 11d] infrastructure/credentials-map.md
- [ 2d] infrastructure/docker-services.md
- [ 11d] infrastructure/domains-and-tunnels.md
- [ 0d] infrastructure/hosts.md
- [ 17d] infrastructure/network-map.md
- [ 17d] infrastructure/proxmox.md
- [ 1d] infrastructure/tailscale.md
- [ 19d] queries/retained-research-answers.md
- [ 6d] reports/communities-wa-social-media-audit-2026.md
- [ 10d] reports/uber-eats/2026-07.md
- [ 10d] reports/uber-eats/2026-08.md
- [ 10d] reports/uber-eats/2026-09.md
- [ 10d] reports/uber-eats/index.md
- [ 11d] runbooks/backup-wiki.md
- [ 11d] runbooks/diagnose-docker.md
- [ 11d] runbooks/gateway-resource-resilience.md
- [ 1d] runbooks/job-radar.md
- [ 11d] runbooks/multiplexer-setup.md
- [ 5d] runbooks/provider-health.md
- [ 2d] runbooks/recover-docker-service.md
- [ 11d] runbooks/restart-browser.md
- [ 11d] runbooks/restart-hermes.md
- [ 11d] runbooks/restore-openrouter.md
- [ 11d] runbooks/test-openrouter.md
- [ 11d] runbooks/troubleshoot-tool-loop.md
- [ 5d] runbooks/update-hermes-safely.md
- [ 2d] systems/browser-backend.md
- [ 11d] systems/cloudflare-tunnel.md
- [ 1d] systems/conduit.md
- [ 5d] systems/current-profile.md
- [ 11d] systems/guanaco.md
- [ 11d] systems/headless-server.md
- [ 1d] systems/hermes-agent.md
- [ 11d] systems/hermes-ui.md
- [ 19d] systems/home-assistant.md
- [ 1d] systems/homepage-dashboard.md
- [ 20d] systems/ignis.md
- [ 5d] systems/memory-backend.md
- [ 5d] systems/messaging-integrations.md
- [ 0d] systems/mnemosyne.md
- [ 5d] systems/model-providers.md
- [ 19d] systems/nanobot.md
- [ 11d] systems/nextcloud.md
- [ 11d] systems/obsidian.md
- [ 9d] systems/omniroute.md
- [ 11d] systems/opencode-go.md
- [ 11d] systems/openrouter.md
- [ 20d] systems/paseo.md
- [ 12d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 11d] systems/terminal-backend.md
- [ 11d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 1 stale, 110 fresh, 3 no-date of 114 pages
+130
View File
@@ -0,0 +1,130 @@
# Wiki freshness audit — 2026-11-01
Threshold: >21 days since `updated`
## STALE (0)
## FRESH (114)
- [ 12d] CONTRIBUTING.md
- [ 6d] agents/hermes-chaos.md
- [ 6d] agents/hermes-production.md
- [ 12d] agents/index.md
- [ 3d] agents/nanobot.md
- [ 12d] agents/open-claw.md
- [ 21d] agents/qwenpaw.md
- [ 12d] agents/zeroclaw.md
- [ 3d] comparisons/camoufox-vs-chrome.md
- [ 12d] comparisons/hermes-vs-nanobot.md
- [ 12d] comparisons/memory-backends.md
- [ 12d] comparisons/model-routers.md
- [ 3d] comparisons/steel-vs-camoufox.md
- [ 12d] concepts/agent-memory.md
- [ 3d] concepts/browser-automation.md
- [ 12d] concepts/model-routing.md
- [ 12d] concepts/persistent-shells.md
- [ 12d] concepts/tool-calling.md
- [ 1d] current-state.md
- [ 3d] decisions/index.md
- [ 12d] decisions/keep-openrouter-as-fallback.md
- [ 12d] decisions/persistent-shell-enabled.md
- [ 3d] decisions/rollback-steel-to-camofox.md
- [ 12d] experiments/index.md
- [ 8d] incidents/2026-07-29-mnemosyne-relay.md
- [ 12d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 8d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 10d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 9d] incidents/2026-08-15-skills-index-drift.md
- [ 20d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 7d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 12d] incidents/2026-08-29-gmail-read-state.md
- [ 11d] incidents/2026-09-02-sftpgo-outage.md
- [ 21d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 3d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 6d] incidents/2026-10-03-ct450-hermes-update.md
- [ 6d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 3d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 8d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 0d] incidents/2026-10-09-memory-recall-relevance.md
- [ 16d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 5d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 13d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 5d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 10d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 7d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 6d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 6d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 6d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 4d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 2d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 2d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 2d] incidents/2026-10-29-conduit-voice-silence.md
- [ 1d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 2d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 1d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 0d] incidents/2026-10-31-asana-access-failures.md
- [ 0d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 0d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 0d] index.md
- [ 10d] infrastructure/backups.md
- [ 12d] infrastructure/credentials-map.md
- [ 0d] infrastructure/docker-services.md
- [ 12d] infrastructure/domains-and-tunnels.md
- [ 1d] infrastructure/hosts.md
- [ 18d] infrastructure/network-map.md
- [ 0d] infrastructure/proxmox.md
- [ 0d] infrastructure/tailscale.md
- [ 20d] queries/retained-research-answers.md
- [ 7d] reports/communities-wa-social-media-audit-2026.md
- [ 11d] reports/uber-eats/2026-07.md
- [ 11d] reports/uber-eats/2026-08.md
- [ 11d] reports/uber-eats/2026-09.md
- [ 11d] reports/uber-eats/index.md
- [ 12d] runbooks/backup-wiki.md
- [ 12d] runbooks/diagnose-docker.md
- [ 12d] runbooks/gateway-resource-resilience.md
- [ 2d] runbooks/job-radar.md
- [ 12d] runbooks/multiplexer-setup.md
- [ 6d] runbooks/provider-health.md
- [ 3d] runbooks/recover-docker-service.md
- [ 12d] runbooks/restart-browser.md
- [ 12d] runbooks/restart-hermes.md
- [ 12d] runbooks/restore-openrouter.md
- [ 12d] runbooks/test-openrouter.md
- [ 12d] runbooks/troubleshoot-tool-loop.md
- [ 6d] runbooks/update-hermes-safely.md
- [ 3d] systems/browser-backend.md
- [ 12d] systems/cloudflare-tunnel.md
- [ 0d] systems/conduit.md
- [ 6d] systems/current-profile.md
- [ 12d] systems/guanaco.md
- [ 12d] systems/headless-server.md
- [ 2d] systems/hermes-agent.md
- [ 12d] systems/hermes-ui.md
- [ 20d] systems/home-assistant.md
- [ 2d] systems/homepage-dashboard.md
- [ 21d] systems/ignis.md
- [ 6d] systems/memory-backend.md
- [ 0d] systems/messaging-integrations.md
- [ 0d] systems/mnemosyne.md
- [ 6d] systems/model-providers.md
- [ 20d] systems/nanobot.md
- [ 12d] systems/nextcloud.md
- [ 12d] systems/obsidian.md
- [ 0d] systems/omniroute.md
- [ 12d] systems/opencode-go.md
- [ 12d] systems/openrouter.md
- [ 21d] systems/paseo.md
- [ 13d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 12d] systems/terminal-backend.md
- [ 12d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 0 stale, 114 fresh, 3 no-date of 117 pages
+133
View File
@@ -0,0 +1,133 @@
# Wiki freshness audit — 2026-11-02
Threshold: >21 days since `updated`
## STALE (4)
- [ 22d] agents/qwenpaw.md (updated 2026-10-11)
- [ 22d] incidents/2026-09-28-hermes-managed-runtime-deps.md (updated 2026-10-11)
- [ 22d] systems/ignis.md (updated 2026-10-11)
- [ 22d] systems/paseo.md (updated 2026-10-11)
## FRESH (113)
- [ 13d] CONTRIBUTING.md
- [ 7d] agents/hermes-chaos.md
- [ 7d] agents/hermes-production.md
- [ 13d] agents/index.md
- [ 4d] agents/nanobot.md
- [ 13d] agents/open-claw.md
- [ 13d] agents/zeroclaw.md
- [ 4d] comparisons/camoufox-vs-chrome.md
- [ 13d] comparisons/hermes-vs-nanobot.md
- [ 13d] comparisons/memory-backends.md
- [ 13d] comparisons/model-routers.md
- [ 4d] comparisons/steel-vs-camoufox.md
- [ 13d] concepts/agent-memory.md
- [ 4d] concepts/browser-automation.md
- [ 13d] concepts/model-routing.md
- [ 13d] concepts/persistent-shells.md
- [ 13d] concepts/tool-calling.md
- [ 2d] current-state.md
- [ 4d] decisions/index.md
- [ 13d] decisions/keep-openrouter-as-fallback.md
- [ 13d] decisions/persistent-shell-enabled.md
- [ 4d] decisions/rollback-steel-to-camofox.md
- [ 13d] experiments/index.md
- [ 9d] incidents/2026-07-29-mnemosyne-relay.md
- [ 13d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 9d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 11d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 10d] incidents/2026-08-15-skills-index-drift.md
- [ 21d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 8d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 13d] incidents/2026-08-29-gmail-read-state.md
- [ 12d] incidents/2026-09-02-sftpgo-outage.md
- [ 4d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 7d] incidents/2026-10-03-ct450-hermes-update.md
- [ 7d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 4d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 9d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 1d] incidents/2026-10-09-memory-recall-relevance.md
- [ 17d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 6d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 14d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 6d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 11d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 8d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 7d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 7d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 7d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 5d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 3d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 3d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 3d] incidents/2026-10-29-conduit-voice-silence.md
- [ 2d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 3d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 2d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 1d] incidents/2026-10-31-asana-access-failures.md
- [ 1d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 1d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 0d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 0d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 0d] index.md
- [ 11d] infrastructure/backups.md
- [ 13d] infrastructure/credentials-map.md
- [ 1d] infrastructure/docker-services.md
- [ 13d] infrastructure/domains-and-tunnels.md
- [ 2d] infrastructure/hosts.md
- [ 19d] infrastructure/network-map.md
- [ 1d] infrastructure/proxmox.md
- [ 1d] infrastructure/tailscale.md
- [ 21d] queries/retained-research-answers.md
- [ 8d] reports/communities-wa-social-media-audit-2026.md
- [ 1d] reports/uber-eats/2026-07.md
- [ 1d] reports/uber-eats/2026-08.md
- [ 1d] reports/uber-eats/2026-09.md
- [ 1d] reports/uber-eats/2026-10.md
- [ 1d] reports/uber-eats/index.md
- [ 13d] runbooks/backup-wiki.md
- [ 13d] runbooks/diagnose-docker.md
- [ 13d] runbooks/gateway-resource-resilience.md
- [ 3d] runbooks/job-radar.md
- [ 13d] runbooks/multiplexer-setup.md
- [ 7d] runbooks/provider-health.md
- [ 4d] runbooks/recover-docker-service.md
- [ 13d] runbooks/restart-browser.md
- [ 13d] runbooks/restart-hermes.md
- [ 13d] runbooks/restore-openrouter.md
- [ 13d] runbooks/test-openrouter.md
- [ 13d] runbooks/troubleshoot-tool-loop.md
- [ 7d] runbooks/update-hermes-safely.md
- [ 4d] systems/browser-backend.md
- [ 13d] systems/cloudflare-tunnel.md
- [ 1d] systems/conduit.md
- [ 7d] systems/current-profile.md
- [ 13d] systems/guanaco.md
- [ 13d] systems/headless-server.md
- [ 3d] systems/hermes-agent.md
- [ 13d] systems/hermes-ui.md
- [ 21d] systems/home-assistant.md
- [ 3d] systems/homepage-dashboard.md
- [ 7d] systems/memory-backend.md
- [ 0d] systems/messaging-integrations.md
- [ 1d] systems/mnemosyne.md
- [ 7d] systems/model-providers.md
- [ 21d] systems/nanobot.md
- [ 13d] systems/nextcloud.md
- [ 13d] systems/obsidian.md
- [ 1d] systems/omniroute.md
- [ 13d] systems/opencode-go.md
- [ 13d] systems/openrouter.md
- [ 14d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 13d] systems/terminal-backend.md
- [ 13d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 4 stale, 113 fresh, 3 no-date of 120 pages
+134
View File
@@ -0,0 +1,134 @@
# Wiki freshness audit — 2026-11-03
Threshold: >21 days since `updated`
## STALE (8)
- [ 23d] agents/qwenpaw.md (updated 2026-10-11)
- [ 23d] incidents/2026-09-28-hermes-managed-runtime-deps.md (updated 2026-10-11)
- [ 23d] systems/ignis.md (updated 2026-10-11)
- [ 23d] systems/paseo.md (updated 2026-10-11)
- [ 22d] incidents/2026-08-18-mnemosyne-capture-pipeline.md (updated 2026-10-12)
- [ 22d] queries/retained-research-answers.md (updated 2026-10-12)
- [ 22d] systems/home-assistant.md (updated 2026-10-12)
- [ 22d] systems/nanobot.md (updated 2026-10-12)
## FRESH (110)
- [ 14d] CONTRIBUTING.md
- [ 8d] agents/hermes-chaos.md
- [ 8d] agents/hermes-production.md
- [ 14d] agents/index.md
- [ 5d] agents/nanobot.md
- [ 14d] agents/open-claw.md
- [ 14d] agents/zeroclaw.md
- [ 5d] comparisons/camoufox-vs-chrome.md
- [ 14d] comparisons/hermes-vs-nanobot.md
- [ 14d] comparisons/memory-backends.md
- [ 14d] comparisons/model-routers.md
- [ 5d] comparisons/steel-vs-camoufox.md
- [ 14d] concepts/agent-memory.md
- [ 5d] concepts/browser-automation.md
- [ 14d] concepts/model-routing.md
- [ 14d] concepts/persistent-shells.md
- [ 14d] concepts/tool-calling.md
- [ 3d] current-state.md
- [ 5d] decisions/index.md
- [ 14d] decisions/keep-openrouter-as-fallback.md
- [ 14d] decisions/persistent-shell-enabled.md
- [ 5d] decisions/rollback-steel-to-camofox.md
- [ 14d] experiments/index.md
- [ 10d] incidents/2026-07-29-mnemosyne-relay.md
- [ 14d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 10d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 12d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 11d] incidents/2026-08-15-skills-index-drift.md
- [ 9d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 14d] incidents/2026-08-29-gmail-read-state.md
- [ 13d] incidents/2026-09-02-sftpgo-outage.md
- [ 5d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 8d] incidents/2026-10-03-ct450-hermes-update.md
- [ 8d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 5d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 10d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 2d] incidents/2026-10-09-memory-recall-relevance.md
- [ 18d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 7d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 15d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 7d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 12d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 9d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 8d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 8d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 8d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 6d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 4d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 4d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 4d] incidents/2026-10-29-conduit-voice-silence.md
- [ 3d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 4d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 3d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 2d] incidents/2026-10-31-asana-access-failures.md
- [ 2d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 2d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 1d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 0d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 0d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 0d] index.md
- [ 12d] infrastructure/backups.md
- [ 14d] infrastructure/credentials-map.md
- [ 2d] infrastructure/docker-services.md
- [ 14d] infrastructure/domains-and-tunnels.md
- [ 3d] infrastructure/hosts.md
- [ 20d] infrastructure/network-map.md
- [ 2d] infrastructure/proxmox.md
- [ 2d] infrastructure/tailscale.md
- [ 9d] reports/communities-wa-social-media-audit-2026.md
- [ 2d] reports/uber-eats/2026-07.md
- [ 2d] reports/uber-eats/2026-08.md
- [ 2d] reports/uber-eats/2026-09.md
- [ 2d] reports/uber-eats/2026-10.md
- [ 2d] reports/uber-eats/index.md
- [ 14d] runbooks/backup-wiki.md
- [ 14d] runbooks/diagnose-docker.md
- [ 14d] runbooks/gateway-resource-resilience.md
- [ 4d] runbooks/job-radar.md
- [ 14d] runbooks/multiplexer-setup.md
- [ 8d] runbooks/provider-health.md
- [ 5d] runbooks/recover-docker-service.md
- [ 14d] runbooks/restart-browser.md
- [ 14d] runbooks/restart-hermes.md
- [ 14d] runbooks/restore-openrouter.md
- [ 14d] runbooks/test-openrouter.md
- [ 14d] runbooks/troubleshoot-tool-loop.md
- [ 8d] runbooks/update-hermes-safely.md
- [ 0d] systems/browser-backend.md
- [ 14d] systems/cloudflare-tunnel.md
- [ 2d] systems/conduit.md
- [ 8d] systems/current-profile.md
- [ 14d] systems/guanaco.md
- [ 14d] systems/headless-server.md
- [ 4d] systems/hermes-agent.md
- [ 14d] systems/hermes-ui.md
- [ 4d] systems/homepage-dashboard.md
- [ 8d] systems/memory-backend.md
- [ 1d] systems/messaging-integrations.md
- [ 2d] systems/mnemosyne.md
- [ 8d] systems/model-providers.md
- [ 14d] systems/nextcloud.md
- [ 14d] systems/obsidian.md
- [ 2d] systems/omniroute.md
- [ 14d] systems/opencode-go.md
- [ 14d] systems/openrouter.md
- [ 15d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 14d] systems/terminal-backend.md
- [ 14d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 8 stale, 110 fresh, 3 no-date of 121 pages
+136
View File
@@ -0,0 +1,136 @@
# Wiki freshness audit — 2026-11-04
Threshold: >21 days since `updated`
## STALE (8)
- [ 24d] agents/qwenpaw.md (updated 2026-10-11)
- [ 24d] incidents/2026-09-28-hermes-managed-runtime-deps.md (updated 2026-10-11)
- [ 24d] systems/ignis.md (updated 2026-10-11)
- [ 24d] systems/paseo.md (updated 2026-10-11)
- [ 23d] incidents/2026-08-18-mnemosyne-capture-pipeline.md (updated 2026-10-12)
- [ 23d] queries/retained-research-answers.md (updated 2026-10-12)
- [ 23d] systems/home-assistant.md (updated 2026-10-12)
- [ 23d] systems/nanobot.md (updated 2026-10-12)
## FRESH (112)
- [ 15d] CONTRIBUTING.md
- [ 9d] agents/hermes-chaos.md
- [ 9d] agents/hermes-production.md
- [ 15d] agents/index.md
- [ 6d] agents/nanobot.md
- [ 15d] agents/open-claw.md
- [ 15d] agents/zeroclaw.md
- [ 6d] comparisons/camoufox-vs-chrome.md
- [ 15d] comparisons/hermes-vs-nanobot.md
- [ 15d] comparisons/memory-backends.md
- [ 15d] comparisons/model-routers.md
- [ 6d] comparisons/steel-vs-camoufox.md
- [ 15d] concepts/agent-memory.md
- [ 6d] concepts/browser-automation.md
- [ 15d] concepts/model-routing.md
- [ 15d] concepts/persistent-shells.md
- [ 15d] concepts/tool-calling.md
- [ 4d] current-state.md
- [ 6d] decisions/index.md
- [ 15d] decisions/keep-openrouter-as-fallback.md
- [ 15d] decisions/persistent-shell-enabled.md
- [ 6d] decisions/rollback-steel-to-camofox.md
- [ 15d] experiments/index.md
- [ 11d] incidents/2026-07-29-mnemosyne-relay.md
- [ 15d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 11d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 13d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 12d] incidents/2026-08-15-skills-index-drift.md
- [ 10d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 15d] incidents/2026-08-29-gmail-read-state.md
- [ 14d] incidents/2026-09-02-sftpgo-outage.md
- [ 6d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 9d] incidents/2026-10-03-ct450-hermes-update.md
- [ 9d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 6d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 11d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 3d] incidents/2026-10-09-memory-recall-relevance.md
- [ 19d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 8d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 16d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 8d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 13d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 10d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 9d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 9d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 9d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 7d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 5d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 5d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 5d] incidents/2026-10-29-conduit-voice-silence.md
- [ 4d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 5d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 4d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 3d] incidents/2026-10-31-asana-access-failures.md
- [ 3d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 3d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 2d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 1d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 0d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 0d] incidents/2026-11-03-job-radar-stream-active-timeout.md
- [ 0d] index.md
- [ 13d] infrastructure/backups.md
- [ 15d] infrastructure/credentials-map.md
- [ 3d] infrastructure/docker-services.md
- [ 15d] infrastructure/domains-and-tunnels.md
- [ 4d] infrastructure/hosts.md
- [ 21d] infrastructure/network-map.md
- [ 3d] infrastructure/proxmox.md
- [ 3d] infrastructure/tailscale.md
- [ 10d] reports/communities-wa-social-media-audit-2026.md
- [ 3d] reports/uber-eats/2026-07.md
- [ 3d] reports/uber-eats/2026-08.md
- [ 3d] reports/uber-eats/2026-09.md
- [ 3d] reports/uber-eats/2026-10.md
- [ 3d] reports/uber-eats/index.md
- [ 15d] runbooks/backup-wiki.md
- [ 15d] runbooks/diagnose-docker.md
- [ 15d] runbooks/gateway-resource-resilience.md
- [ 5d] runbooks/job-radar.md
- [ 15d] runbooks/multiplexer-setup.md
- [ 9d] runbooks/provider-health.md
- [ 6d] runbooks/recover-docker-service.md
- [ 15d] runbooks/restart-browser.md
- [ 15d] runbooks/restart-hermes.md
- [ 15d] runbooks/restore-openrouter.md
- [ 15d] runbooks/test-openrouter.md
- [ 15d] runbooks/troubleshoot-tool-loop.md
- [ 9d] runbooks/update-hermes-safely.md
- [ 1d] systems/browser-backend.md
- [ 15d] systems/cloudflare-tunnel.md
- [ 3d] systems/conduit.md
- [ 9d] systems/current-profile.md
- [ 15d] systems/guanaco.md
- [ 15d] systems/headless-server.md
- [ 5d] systems/hermes-agent.md
- [ 0d] systems/hermes-hq-push.md
- [ 15d] systems/hermes-ui.md
- [ 5d] systems/homepage-dashboard.md
- [ 9d] systems/memory-backend.md
- [ 2d] systems/messaging-integrations.md
- [ 3d] systems/mnemosyne.md
- [ 9d] systems/model-providers.md
- [ 15d] systems/nextcloud.md
- [ 15d] systems/obsidian.md
- [ 3d] systems/omniroute.md
- [ 15d] systems/opencode-go.md
- [ 15d] systems/openrouter.md
- [ 16d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 15d] systems/terminal-backend.md
- [ 15d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 8 stale, 112 fresh, 3 no-date of 123 pages
+139
View File
@@ -0,0 +1,139 @@
# Wiki freshness audit — 2026-11-05
Threshold: >21 days since `updated`
## STALE (0)
## FRESH (123)
- [ 16d] CONTRIBUTING.md
- [ 10d] agents/hermes-chaos.md
- [ 10d] agents/hermes-production.md
- [ 16d] agents/index.md
- [ 7d] agents/nanobot.md
- [ 16d] agents/open-claw.md
- [ 1d] agents/qwenpaw.md
- [ 16d] agents/zeroclaw.md
- [ 7d] comparisons/camoufox-vs-chrome.md
- [ 16d] comparisons/hermes-vs-nanobot.md
- [ 16d] comparisons/memory-backends.md
- [ 16d] comparisons/model-routers.md
- [ 7d] comparisons/steel-vs-camoufox.md
- [ 16d] concepts/agent-memory.md
- [ 7d] concepts/browser-automation.md
- [ 16d] concepts/model-routing.md
- [ 16d] concepts/persistent-shells.md
- [ 16d] concepts/tool-calling.md
- [ 0d] current-state.md
- [ 7d] decisions/index.md
- [ 16d] decisions/keep-openrouter-as-fallback.md
- [ 16d] decisions/persistent-shell-enabled.md
- [ 7d] decisions/rollback-steel-to-camofox.md
- [ 16d] experiments/index.md
- [ 12d] incidents/2026-07-29-mnemosyne-relay.md
- [ 16d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 12d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 14d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 0d] incidents/2026-08-15-skills-index-drift.md
- [ 1d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 11d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 16d] incidents/2026-08-29-gmail-read-state.md
- [ 15d] incidents/2026-09-02-sftpgo-outage.md
- [ 1d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 7d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 10d] incidents/2026-10-03-ct450-hermes-update.md
- [ 10d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 7d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 12d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 4d] incidents/2026-10-09-memory-recall-relevance.md
- [ 20d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 9d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 17d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 9d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 14d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 11d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 10d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 10d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 10d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 8d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 6d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 6d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 6d] incidents/2026-10-29-conduit-voice-silence.md
- [ 5d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 6d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 5d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 4d] incidents/2026-10-31-asana-access-failures.md
- [ 4d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 4d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 3d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 2d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 1d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 1d] incidents/2026-11-03-job-radar-stream-active-timeout.md
- [ 0d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
- [ 0d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
- [ 0d] incidents/2026-11-05-skills-index-generator-divergence.md
- [ 0d] index.md
- [ 1d] infrastructure/backups.md
- [ 16d] infrastructure/credentials-map.md
- [ 0d] infrastructure/docker-services.md
- [ 16d] infrastructure/domains-and-tunnels.md
- [ 0d] infrastructure/hosts.md
- [ 0d] infrastructure/network-map.md
- [ 4d] infrastructure/proxmox.md
- [ 0d] infrastructure/tailscale.md
- [ 1d] queries/retained-research-answers.md
- [ 11d] reports/communities-wa-social-media-audit-2026.md
- [ 4d] reports/uber-eats/2026-07.md
- [ 4d] reports/uber-eats/2026-08.md
- [ 4d] reports/uber-eats/2026-09.md
- [ 4d] reports/uber-eats/2026-10.md
- [ 4d] reports/uber-eats/index.md
- [ 16d] runbooks/backup-wiki.md
- [ 16d] runbooks/diagnose-docker.md
- [ 16d] runbooks/gateway-resource-resilience.md
- [ 0d] runbooks/job-radar.md
- [ 16d] runbooks/multiplexer-setup.md
- [ 10d] runbooks/provider-health.md
- [ 7d] runbooks/recover-docker-service.md
- [ 16d] runbooks/restart-browser.md
- [ 16d] runbooks/restart-hermes.md
- [ 16d] runbooks/restore-openrouter.md
- [ 16d] runbooks/test-openrouter.md
- [ 16d] runbooks/troubleshoot-tool-loop.md
- [ 10d] runbooks/update-hermes-safely.md
- [ 2d] systems/browser-backend.md
- [ 16d] systems/cloudflare-tunnel.md
- [ 4d] systems/conduit.md
- [ 10d] systems/current-profile.md
- [ 16d] systems/guanaco.md
- [ 16d] systems/headless-server.md
- [ 6d] systems/hermes-agent.md
- [ 1d] systems/hermes-hq-push.md
- [ 16d] systems/hermes-ui.md
- [ 1d] systems/home-assistant.md
- [ 6d] systems/homepage-dashboard.md
- [ 1d] systems/ignis.md
- [ 10d] systems/memory-backend.md
- [ 3d] systems/messaging-integrations.md
- [ 4d] systems/mnemosyne.md
- [ 10d] systems/model-providers.md
- [ 1d] systems/nanobot.md
- [ 16d] systems/nextcloud.md
- [ 16d] systems/obsidian.md
- [ 4d] systems/omniroute.md
- [ 16d] systems/opencode-go.md
- [ 16d] systems/openrouter.md
- [ 1d] systems/paseo.md
- [ 17d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 16d] systems/terminal-backend.md
- [ 16d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 0 stale, 123 fresh, 3 no-date of 126 pages

Some files were not shown because too many files have changed in this diff Show More