Commit Graph
90 Commits
Author SHA1 Message Date
Tony0410 97660d3dcb audit(2026-11-01): record the whole-tree mirror catch-up in the log 2026-11-01 00:33:09 +08:00
Tony0410 0c826f2b5f audit(2026-11-01): Conduit updated to 0.122.2 (memory-timeout patch now upstream, call store re-shaped); document the local-voice-mode fault, the force-drained gateway reload that orphaned a cron run, and both dead Asana paths; 2026-10-31 settled 160/4 2026-11-01 00:30:42 +08:00
hermes 74a5b2d117 audit(2026-10-31): document CT460 5h30m host outage; correct the 'paused' health monitor (it was running, closed completed 10:02:48); reconcile fleet counts to 26/3 2026-10-31 00:39:11 +08:00
Tony0410 db6af66878 incident: m93p NFS mount loss took down all NFS-backed LXC containers (2026-10-30) 2026-10-30 17:41:34 +08:00
Tony0410 987eba72e6 Stocktake 2026-10-30: 0 stale by age, but 5 pages drifted from live state
Ran wiki-maintain.sh (skills-index regenerated: 302 skills / 69 categories,
unchanged; freshness report 0 stale / 109 fresh / 3 no-date of 112 pages;
Quartz rebuilt and deployed).

Freshness is not accuracy. A live-vs-page diff found real drift on five pages.
All corrections verified against live sources first.

- infrastructure/tailscale: node table rebuilt from tailscale status --json.
  50 non-funnel nodes (49 IPv4 peers + hermes), not 54. 2 IP changes, 4 peers
  expired out of the tailnet. Zero IP mismatches, zero missing/extra rows on
  re-verification. m93p records online (peer record chronically stale, third
  occurrence -- ping + SSH confirm). CT460 :443 route set (12 paths) and
  CT201 :443/:8443 tables re-probed and match.
- systems/scheduled-tasks: fleet is 29 jobs (25 enabled / 4 disabled), not
  28/25/3. Mnemosyne Health Monitor ea79a2107e36 was auto-paused 2026-10-30
  01:04:22 after its 00:01:08 slot closed "unknown" on a gateway-connection
  failure. Non-clean last_status is 4, not 3. "Failing (delivery only)" is 2.
  Seven job rows carried stale error claims for jobs since cleared to ok --
  each marked superseded with a dated correction, not rewritten (SCHEMA rule 9).
- NEW incidents/2026-10-30-mnemosyne-health-monitor-paused: the job's own
  failure notice was never delivered (deliveries.db has zero rows for
  2026-10-30), so the incident row is the only trace. Job not re-enabled and
  gateway not restarted by this run; root cause recorded as unverified.
- infrastructure/hosts: CT201 LAN is 192.168.178.77 (was .166); CT450 Hermes
  is v0.22.5 / bf0203a794a0, same build as CT460 (page said v0.22.3 /
  5b147ab036d6); cron count 29/25/4; orphaned "omni" peer has expired out of
  the tailnet entirely. Skills count of 302 left alone -- it is correct.
- systems/homepage-dashboard: live app count is 123 (was 114); grid is a
  responsive repeat(auto-fill, minmax(230px, 1fr)), not a fixed 5-column grid;
  tab bar is built from the service groups (renderTabs iterates state.groups),
  so the settings.yaml layout block and its tab:/columns: keys are inert --
  there is no "Overview" tab; saved theme is Boarding Pass / light, not dark
  amber; manager.py runs inside the homepage-manager container; and the "no
  Traefik container" evidence was wrong -- the container exists but is
  stopped (Exited (255) 2 months ago). The conclusion that Traefik does not
  occupy 443 stands. Same evidence error corrected in tailscale.md.

Also corrected two of my own first-pass errors before committing: the
homepage tab list (groups vs tabs) and the CT450 version (a stale root-level
/usr/local/bin/hermes reports v0.19.0; the live agent is the hermes-user
install at v0.22.5).

No secrets written -- provider names, key names and HTTP status codes only.
2026-10-30 05:41:11 +08:00
Tony0410 1d335ca8e6 wiki(audit): 2026-10-29 window — Conduit voice fault documented and repaired; radar two-stage split resolved both incidents (20 pages, 2 letters, Notion 455->475); cron 176/0 2026-10-30 00:21:21 +08:00
Tony0410 cc95dcc54d stocktake: re-verify Tailscale table, correct falsified cliprox/terra claim, regenerate skills-index
Wiki maintenance (wiki-maintain.sh, exit 0) + live-vs-page drift audit.

Script:
- systems/skills-index.md regenerated from the live skill tree: 302 skills in 69
  categories; 45 archived skills under ~/.hermes/skills/.archive/ now excluded and
  noted. (303 SKILL.md files exist outside .archive/, but
  productivity/obsidian-git-sync/obsidian-git-sync is a nested alias duplicate,
  so 302 unique is correct.)
- Freshness report raw/stocktake/freshness-20261029.txt: 2 stale / 105 fresh /
  3 no-date. After the corrections below and a second script run: 0 stale /
  107 fresh / 3 no-date.
- Quartz rebuilt (110 pages -> 644 files) and deployed to public/.

Drift corrections (live-verified):
- infrastructure/tailscale.md: State column rebuilt from tailscale status --json,
  11 rows changed. Two were state transitions, not age drift:
  * sftpgo (CT299, pve) is back online - tailscale ping pong in 0s,
    systemctl is-active sftpgo -> active, tailscale ip -4 inside the guest.
  * m93p is online despite a stale peer record (Online: false in the JSON) -
    tailscale ping pong in 1ms, SSH over the tailnet returned up 11 days.
    Second occurrence of this pattern after 2026-10-26.
  The other 9 rows were offline-age drift of exactly +1 day. Node count unchanged
  at 54 (53 live IPv4 peers + hermes); 76 peers total, 23 IPv6-only funnel
  ingress. No missing peers, no IP mismatches. Container identity re-confirmed
  from inside all 34 guests via pct exec tailscale ip -4 on both hypervisors;
  CT601 still absent from both.

- incidents/2026-10-01-cliproxy-connection-auth-expired.md: the residual claim
  "the connection credential itself was never re-authenticated" is falsified.
  Live POST /v1/chat/completions for cliprox/terra against custom:Omniroute:20128
  (CT600) returns HTTP 200 - the connection has since been re-authenticated. The
  same pin on custom:omni-101 (CT101) still returns HTTP 401, so the nodes carry
  separate credentials. Claim marked superseded, not deleted (SCHEMA rule 9); the
  remediation was never reversed and remains correct.

- incidents/2026-10-05-dead-model-pin-repin.md: every claim re-probed. All 7 jobs
  still on custom:omni-101 with routing aliases; both aliases HTTP 200;
  ollama-cloud/qwen3.5:397b still HTTP 410; nvidia/deepseek-v4-flash still
  HTTP 400; command-code/gpt-5.6-terra now HTTP 401 rather than the original 403
  - still dead, different failure signature.

- systems/scheduled-tasks.md: the Known-Issues entry repeating the
  "never re-authenticated" claim now carries the correction. Fleet figures
  re-checked against live jobs.json and remain accurate (29 jobs / 26 enabled /
  3 disabled, 5 non-clean last_status).

- log.md / index.md: stocktake entry appended and index last-updated refreshed.

Verified correct, no change: systems/homepage-dashboard (CT201 has 8084 and 8085
listening, no 8082, Homarr on 8102); CT201 Serve routes (/ -> 8085, /mgr -> 8084
on both :443 and :8443, Steel routes still gone); CT460 Serve routes (12 paths
on :443, plus :8443/webhook Funnel and :9443).

Noted for a human, not performed: vault mirror at /home/hermes/wiki/agent-estate-wiki
has drifted on both conventions (top-level 35 match / 3 stale / 72 missing;
wiki/ 41 match / 53 stale / 16 missing). A full mirror pass is larger than a
stocktake's mandate.
2026-10-29 03:42:54 +08:00
Tony0410andHermes 677bb0c959 docs(wiki): correct Steel Browser references across 15 pages — Steel fully removed, Camofox is sole backend
Root cause of the long-standing cloud_provider: steel residue: the 2026-07-30
rollback disabled the browser-steel plugin and set browser.engine: camofox but
left browser.cloud_provider: steel in config.yaml. The split config was
inert-but-wrong, so it survived ~3 months of audits unnoticed.

Corrects pages that still described Steel as present, running or configured:
- systems/browser-backend.md (incl. gateway-restart env-var note)
- comparisons/steel-vs-camoufox.md, comparisons/camoufox-vs-chrome.md
- decisions/rollback-steel-to-camofox.md, decisions/index.md
- infrastructure/docker-services.md, infrastructure/tailscale.md, infrastructure/hosts.md
- systems/homepage-dashboard.md, systems/scheduled-tasks.md
- concepts/browser-automation.md, runbooks/recover-docker-service.md
- current-state.md, index.md, agents/nanobot.md
- log.md (dated entry; supersedes the 00:3x audit's 'endpoint dark' wording)

Also restores the Ignis compose row in infrastructure/docker-services.md, which
the 2026-10-11 revision dropped (verified: /home/hermes/docker/ignis/docker-compose.yml
exists and the ignis container is Up on CT460).

Co-Authored-By: Hermes <hermes@local>
2026-10-29 02:02:49 +08:00
hermes ec0c30fb30 wiki(audit): 2026-10-28 window — radar stage-1 iteration cap (no handoff, stage-2 no-oped), stocktake 400 never alerted, Steel removed from the stack 2026-10-29 00:41:30 +08:00
Tony0410 2cbf0b65c1 audit: correct the 2026-10-28 log entry heading to this window and this story 2026-10-28 00:21:33 +08:00
Tony0410 3ae05871c4 audit: align verified_on for the 2026-10-26 orphan incident with the 2026-10-28 recurrence check 2026-10-28 00:21:24 +08:00
Tony0410 f880ba981d audit: 2026-10-28 nightly — radar budget overrun now a recorded failure (new incident page); radar split into two cron jobs; tailscale peer table re-verified; TTS provider corrected to the Mac mini speech server
Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76
(Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed
failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before
Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and
cover letters never ran (Notion DB held 451 entries, 0 added). Incident row
dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified).

Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5,
deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5,
topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3).

Also: the two incident rows the 2026-10-26 planned restart raised auto-closed
00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean
last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never
committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of
108) — committed here and index.md corrected; tailscale.md peer table rebuilt from
tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo
CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to
provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained
the voice pipeline section.

New: incidents/2026-10-27-job-radar-budget-exhausted.md
Nothing removed or archived. No secrets.
2026-10-28 00:20:59 +08:00
Tony0410 a2dd0bd650 audit: 2026-10-26 nightly — planned restart orphaned a live cron run (new incident page); 503 burst fully closed; radar cover-letter PDF pipeline moved to Chromium-over-HTML 2026-10-27 00:11:50 +08:00
Tony0410 3f796dc3b9 stocktake(2026-10-26): strike superseded follow-ups per SCHEMA rule 9
Mark-don't-delete: the CT450 '3 commits behind' follow-up and the two
mnemosyne-vector-recall follow-ups are struck through with their verified
2026-10-26 resolution rather than removed.

- CT450 tracks origin/main (there is no origin/master on that remote) and is
  0 commits behind.
- Coverage monitor b63ed5817212 is enabled, last_status ok, working coverage
  96.8% — correctly has nothing to fire on.
- fastembed 0.8.1 now imports in both managed runtimes plus the legacy venv.
2026-10-26 04:35:33 +08:00
Tony0410 7fe109b914 stocktake(2026-10-26): record measured vault-mirror drift counts in log.md
Vault at /home/hermes/wiki tracks 125 files under two conventions. This run
forward-synced all 22 touched-page copies (sha256-verified) and committed the
vault locally; it is local-only and was not pushed. Remaining drift measured by
hash sweep and recorded rather than silently mirrored:
  top-level: match=25 stale=2 missing=80
  wiki/     : match=34 stale=58 missing=15
2026-10-26 04:30:35 +08:00
Tony0410 4126093644 stocktake(2026-10-26): correct 8 stale pages against live state — 0 stale remaining
wiki-maintain.sh ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted,
service restarted). The freshness report listed 8 stale pages; each was diffed
against live state and corrected only where reality could be verified. A second
script run rewrote the same-day report with 0 stale / 103 fresh / 4 no-date,
so the committed report matches the tree it ships with.

- infrastructure/hosts.md: CT600 is LIVE (not retired); CT601 does not exist on
  either hypervisor; CT333 verified running; both hypervisors verified; config
  33->51; skills 66->302; cron 25->28 (25 enabled / 3 disabled); Xvfb :102->:99;
  providers split into Omniroute:20128 (CT600) and omni-101 (CT101, default).
- agents/hermes-production.md: Tailscale IP 100.96.100.82 -> 100.118.5.51 (the
  old value is the PVE hypervisor); v0.21.5 -> v0.22.3; dead 8082 claim removed;
  fallback chain rewritten from live config (first entry is the dead nous
  credential); all 8 auxiliary slots now no-think/freeapi/auto; cron 30->28.
- agents/hermes-chaos.md: v0.22.3 (same build as CT460); "3 commits behind" ->
  0 behind origin/master; default model smart-route on custom:omni-101; cron
  "11 paused" -> 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460.
- infrastructure/tailscale.md: node table rebuilt from status --json; octoagent
  and omarchymac-1 were missing entirely (52 -> 54 non-funnel nodes); m93p now
  online; every offline age refreshed; omni row corrected to "CT601 removed
  from both hypervisors"; Serve tables re-probed per host.
- systems/memory-backend.md: re-verified against live config; coverage 1899/2350
  (80.8%), working 96.8%, episodic 99.4% — above the 95% reindex threshold.
- runbooks/provider-health.md: the step-3 OmniRoute snippet raised IndexError
  (provider renamed omniroute -> Omniroute:20128; key moved to key_env) — fixed;
  new step 3b documents the omni-101 /v1/models false-failure quirk.
- runbooks/update-hermes-safely.md: added managed-runtime dependency re-assert
  step; clarified --branch vs stable channel (v0.22.5 available).
- incidents/2026-10-03-ct450-hermes-update.md + incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md:
  Current-state sections recording current versions and embedding coverage.

All 9 corrected pages verified 200 at http://127.0.0.1:9120/<path>.
Vault mirror at /home/hermes/wiki synced for all touched files (22 copies,
sha256-verified) and committed locally; the vault is local-only and is not pushed.
No secrets written — IDs, URLs, key env var names, counts and timestamps only.
2026-10-26 04:27:49 +08:00
Tony0410 123a74cb0b audit(2026-10-26): runaway Gmail digest + dead Nous credential; model default reverted to OmniRoute; conduit 0.69.2 live; freshness floor still 2026-10-08 2026-10-26 00:19:37 +08:00
Tony0410 2df1bce5ea wiki maintenance 2026-10-25: skills-index regen (302 skills/69 cats), freshness audit 0 stale, tailscale node table + CT201 route corrections
- skills-index.md regenerated from live skill tree: 302 skills / 69 categories
  (4 skills moved to .archive/ since 10-24), 45 archived
- freshness-20261025.txt: 0 stale, 101 fresh, 4 no-date of 105 pages
- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json
  (springroll-browser was missing; omarchybook + openmausbot now offline; all offline
  ages refreshed; homeassistant exit-node wording corrected) + CT201 serve routes
  corrected (443 AND 8443; /steel only on :443)
- systems/homepage-dashboard.md: same CT201 route + false Traefik claim corrected
- reports/communities-wa-social-media-audit-2026.md: re-verified against live WA source
- incidents/2026-08-26-work-outlook-basic-auth-disabled.md: re-probed, still open
- log.md: appended stocktake entry
2026-10-25 03:40:26 +08:00
Tony0410 3271ff12a5 audit 2026-10-25: third consecutive clean cron day; new incident (agent refused 3 Conduit call requests); correct 10-22 memory-fix evidence; conduit 0.69.0/f000bf0 2026-10-25 00:16:26 +08:00
Tony0410 ec69d107ae wiki maintenance 2026-10-24: skills-index regen (306 skills/69 cats), freshness audit, fix YAML frontmatter in mnemosyne-relay incident 2026-10-24 03:06:41 +08:00
Tony0410 573a4a2434 audit 2026-10-24: conduit page + memory-start-timeout incident; 10-09 storm resolved; window 169/0; frontmatter for 2 legacy incident pages 2026-10-24 00:24:44 +08:00
Tony0410 250c73530a 🦉 wiki maintenance 2026-10-23: model-providers drift fixed (default moved to nous 2026-10-20); freshness 2 stale/94 fresh; skills-index 306 2026-10-23 03:06:37 +08:00
Tony0410 38e96375fc audit 2026-10-23: current-state 2026-10-23 block; log entry 2026-10-23 00:18:26 +08:00
Tony0410 59d4b2e695 audit 2026-10-23: scheduled-tasks window (172/0, 5 incidents closed); reopen skills-index drift; skills-index generator now excludes .archive (306 skills, was 347) 2026-10-23 00:14:49 +08:00
Tony0410 1d92de31a0 🦉 wiki maintenance 2026-10-22: Quartz rebuild+deploy (new incident page live), skills-index 348, freshness 1 stale/95 fresh 2026-10-22 00:19:03 +08:00
Tony0410 3919ec8557 🦉 wiki audit 2026-10-22: omni-101 pre-dispatch filter skip (3rd failure mode, new page), Hermes 0.22.3 update with snapshot skipped, Gitea backup backlog 76
- new: incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- systems/hermes-agent.md: version -> 0.22.3 (5b147ab036), 22:47-22:57 update, snapshot stage skipped, advisories unactioned
- systems/scheduled-tasks.md: 2026-10-21 = 171 completed / 1 failed; non-clean stays 9 with 08eb86f02d37 out, dca8482e4f76 in
- systems/omniroute.md: one pin, three signatures in eight days
- incidents/2026-10-16-omniroute-503-resource-pressure.md: window update, stays monitoring
- incidents/2026-08-08-gitea-config-backup-push-dead.md: 2026-10-21 run, backlog 75 -> 76, message_id 8449
- infrastructure/backups.md: 76 commits; pre_update_backup:false now exercised
- current-state.md, index.md, log.md updated
2026-10-22 00:16:39 +08:00
Tony0410 c1d3b59260 🦉 wiki stocktake 2026-10-21: tailscale node table rebuilt, 2 missing Uber Eats reports recovered
Maintenance script ran clean (skills-index 346 -> 347, freshness audit, Quartz rebuilt
596 files and deployed, hermes-wiki-serve.service restarted).

infrastructure/tailscale.md — node table rebuilt from live `tailscale status --json`.
  The 2026-10-19 table had drifted: octop (100.92.163.122) and springroll
  (100.100.153.125) missing entirely, openmausbot / openmausbot-vm present in prose
  but absent from the table body, omarchymac now offline, all offline ages stale.
  Rebuilt keyed on DNS name rather than the JSON HostName — HostName is `ct260` for
  the freshrss node and BOTH OmniRoute nodes report the hostname `omniroute`, which
  collides silently. Now 51 non-funnel nodes. Container states cross-checked against
  live `pct list` on pve and m93p: CT255 stopped, CT601 stopped, CT299 running.

reports/uber-eats/ — 2026-08 and 2026-09 recovered. The generator wrote only to
  ~/wiki/reports/uber-eats/ and never to the canonical tree, so both months were
  missing from the published wiki. Root cause fixed in
  ~/.hermes/cron/scripts/generate-uber-eats-report.py: it now targets
  ~/workspace/agent-estate-wiki/reports/uber-eats/ and mirrors to the legacy tree,
  and now emits YAML frontmatter (without it the pages landed in the audit's
  "no updated DATE" bucket and Quartz could not read the page date). 2026-08
  previous-month column corrected from $0.00 to $264.51 / 11 orders.

incidents/2026-08-15-skills-index-drift.md — re-verified live; the named
  regenerate-skills-index.sh no longer exists on disk (regeneration now lives in
  wiki-maintain.sh step 1/3). Incident prevention confirmed working: 347 skills.
incidents/2026-09-02-sftpgo-outage.md — re-verified live: CT299 running, health
  endpoint HTTP 401 (auth required, service up), no recurrence.

reports/communities-wa-social-media-audit-2026.md — left stale (54d). Its source
  document no longer exists on disk, so the baseline cannot be re-verified against
  a live source. Flagged, not edited.

Freshness after this run: 1 stale, 94 fresh, 6 no-date of 101 pages.
2026-10-21 03:14:26 +08:00
Tony0410 b662254569 🦉 wiki audit 2026-10-21: gateway unclean-exit orphaned run (new), Gitea config-backup push dead since 2026-08-08 (new), model default → nous, 503 back to monitoring, canned-substitute trigger corrected 2026-10-21 00:11:44 +08:00
Tony0410 fbd4f297eb wiki: publish pending edits (2026-10-20) 2026-10-20 18:38:11 +08:00
Hermes Rhino 667ab7f41d log: authoring commit chain closed 2026-10-20 00:14:22 +08:00
Hermes Rhino 0248698030 log: final authoring commit chain for the 2026-10-20 audit 2026-10-20 00:14:08 +08:00
Hermes Rhino 8da6ddfa4d audit(2026-10-20): annotate the historical 10-17 162/5 tallies with the DB-corrected 163/5; mark the 10-19 count superseded 2026-10-20 00:14:02 +08:00
Hermes Rhino 766882ed29 log: record vault-drift figures + open mirror pass for the 2026-10-20 audit 2026-10-20 00:13:20 +08:00
Hermes Rhino 95fd7d33f1 audit(2026-10-20): omni-101 503 burst 2 — incident back to active; scheduled-tasks rebuilt from jobs.json
- burst 2 (2026-10-19): 9 model calls across 8 jobs, 159/13 day
- burst-2 table names/schedules/aliases corrected from live jobs.json
- non-clean last_status 7 -> 12; unclosed incident rows 13 (query stated)
- 2026-10-17 tail corrected 162/5 -> 163/5
2026-10-20 00:12:39 +08:00
Tony0410 93c55efc94 stocktake 2026-10-19: skills-index 346/70; freshness 47 stale/43 fresh/7 no-date; open-claw VM 403 verified absent from PVE 2026-10-19 03:03:15 +08:00
Tony0410 0ed522bfb8 audit(2026-10-19): degenerate-stream incident page + push-notifications page; refresh scheduled-tasks, current-state, index, log, messaging, mnemosyne, tailscale, proxmox 2026-10-19 00:21:26 +08:00
Tony0410 dd53e5fbe1 stocktake 2026-10-18: skills-index 346 skills/70 cats; freshness 47 stale/41 fresh/7 no-date; quartz build skipped (no quartz.config.ts) 2026-10-18 03:01:17 +08:00
hermes-audit 58fa15bb36 audit 2026-10-18 (follow-up): exact next-slot dates for the 8 non-clean jobs; bridge paragraph corrected with verified vault drift 2026-10-18 00:42:36 +08:00
hermes-audit 4137dd6945 audit 2026-10-18: new incident page (omniroute-1 503 resource pressure, 9 calls / 7 jobs), fleet re-verified 28 jobs / 25 enabled / 8 non-clean, Mosquito Tracker removed, radar bot token dead 2026-10-18 00:41:38 +08:00
Tony0410 76de919e6a stocktake 2026-10-16: skills-index 346 skills/70 cats; freshness 47 stale/40 fresh/7 no-date; tailscale re-verified live — omarchymac (100.84.180.69) added, mattermost state corrected, offline ages refreshed (46 non-funnel nodes) 2026-10-16 03:04:25 +08:00
hermes 2061b8607b audit(2026-10-16): fleet-wide 34h cron blackout 10-14 -> 10-15 — omniroute-1 refused connections
omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07
(httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error).
36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog
symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1).

STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable
pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls
completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence.

New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live).
Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute,
systems/mnemosyne (monitor blast-radius note), current-state, index, log.
2026-10-16 00:19:14 +08:00
Tony0410 deb6a8eec2 stocktake 2026-10-14: SECOND Proxmox host m93p (100.104.51.98, PVE 9.2.0, unclustered, IDs collide) discovered — gitea/ntfy/freshrss/openweb were never on pve; tailnet inventory rebuilt 30 -> 45 nodes (15 missing); VM 282 running; Camofox recovered to 29.9M
skills-index 346 skills / 70 cats; freshness 48 stale / 38 fresh / 7 no-date of 93 pages

- infra/proxmox.md: two-host rewrite, m93p LXC+QEMU inventory, 4-entry correction table, VM 282 stopped->running, storage via pvesm (ssd0 88.17%/63GiB free)
- infra/tailscale.md: 45-node inventory rebuilt from live status; 23 funnel-ingress-node IPv6 peers noted+excluded; CT460 serve routes re-verified exactly (13 routes + :8443 Funnel + :9443)
- infra/network-map.md, current-state.md, index.md, log.md updated
- Open: 3 incidents (unverified pages left stale, not rewritten); omniroute/hosts/model-providers/docker-services still single-host in framing — m93p runs its own omniroute CT101, needs a dedicated pass
2026-10-14 03:13:04 +08:00
Tony0410 828913e23b audit(2026-10-14): 10-13 broke the clean streak — wiki-stocktake failed on a dead opencode credential (HTTP 404, incident 1a8e90_d11698bd7275); stranded skills-index + freshness-20261013 recovered; retention floor moved to 2026-10-08 07:11 2026-10-14 00:17:31 +08:00
hermes-audit 1df564e152 audit(2026-10-13): close out the 2026-10-09 open-items snapshot (DHL retire DONE 2026-10-10; all eight storm incidents cleared) 2026-10-13 00:19:44 +08:00
hermes-audit ebf57146f1 audit(2026-10-13): 10-12 clean (174/0, 3rd day) but GitHub Trending Radar token DEAD (getMe 401) - supersedes the wrong 2026-10-06 'refuted' entry
Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline.

NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450).

Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07.
2026-10-13 00:19:15 +08:00
hermes 0eee5f101e stocktake 2026-10-12: skills-index 346 skills/70 cats; freshness 54 stale/32 fresh/7 no-date; nanobot + home-assistant re-verified live (nanobot web UI back up on :8766, OmniRoute :8900 still closed -> stays degraded; HA HTTP 200 on :8123 2026-10-12 03:17:10 +08:00
Tony0410 6fc109e546 audit(2026-10-12): cron fleet clean 2nd day (10-11 171/0, 10-10 finalised 175/0); UTC-vs-AWST false 'capture stalled' alarm corrected; proxmox storage table + host mounts; queries/retained-research-answers stub filled (VNC); capture-pipeline incident -> superseded; index stale labels fixed 2026-10-12 00:14:11 +08:00
hermes d981e30128 stocktake(2026-10-11): skills-index 346 skills/70 cats; freshness 62 stale/24 fresh/7 no-date; 5 live-drift corrections
Drift found and corrected against live sources (not guessed):
- OmniRoute: CT600 omniroute is RUNNING and serving :20128 (combo API live);
  CT601 omni is STOPPED with onboot=0 (peer offline 54d). Wiki had these inverted.
  Hermes custom:omni resolves to omniroute -> CT600, not CT601.
- Third OmniRoute node discovered: omniroute-1 at 100.78.63.20:20128, live (401),
  registered as custom provider omni-101 (5384 models, default smart-route,
  carries skills_hub/free-stack). Not in pct list; identity unconfirmed.
- Ignis: TWO live instances. CT460 ignis (:8080, vault wiki/agent-estate-wiki) and
  CT200 ignis-ignis-1 (image ignis-server:latest, host 8092, vault /opt/ignis/vaults).
  Both return HTTP 200. Page now distinguishes them.
- Tailscale: /agent-estate targets bare 127.0.0.1:8769 (was .../8769/agent-estate-role-dashboard.html);
  undocumented tailnet-only :9443 -> 127.0.0.1:5173 added; CT201 routes corrected
  (8085 targets + /steel 5173 + /steel-api 3000, replacing the wrong :8443 -> :8082 row).
- Proxmox: rebuilt from live pct list + qm list. Added CT475 agentzero and VM110 omarchy
  (both previously missing); 282 ubuntu-server-xfce corrected to stopped; dead rows removed.
  VM403 open-claw and VM401 hermes-old absent from qm list - flagged unconfirmed, not deleted.
- Nanobot: gateway nanobot-gateway (PID 2259048) alive on 8765/8766. It is NOT systemd
  nanobot.service/gateway.service (both return inactive - wrong unit names). Port 8900
  no longer listening; "Serve API" claim flagged unverified.
- Docker: old "runtime not available" was a CT460 socket artifact (root:docker srw-rw----,
  hermes not in group docker, no passwordless sudo in cron). Documented CT200 (~50 containers),
  CT333 (9 containers), CT460 via sudo.
- Mattermost CT255 confirmed stopped; matches Tailscale peer offline 21d.

Flagged, not investigated (no root cause):
- mycompose-db-1 on CT200 is in Restarting (1) - crash-looping.

62 pages remain >21d stale and were NOT rewritten this pass (concepts, decisions,
comparisons, most runbooks) - not probed against a live source, so reported not faked.
Quartz rebuild step still inert (no quartz.config.ts) - public/ not refreshed.

Artifacts: systems/skills-index.md regenerated, raw/stocktake/freshness-20261011.txt.
2026-10-11 03:11:07 +08:00
hermes 7bffeab157 audit(2026-10-10): DHL monitor retired (29 jobs / 26 enabled, verified by ID lookup); 10-10 fleet clean 174/0, 7 incidents closed - 5 open; gateway runtime env recurrence (mnemosyne + elevenlabs) verified fixed; new systems/paseo.md; mattermost unreachable; current-state + index + log updated 2026-10-11 00:22:20 +08:00
Hermes c11f11e652 stocktake 2026-10-10: skills-index + freshness report (62 stale/23 fresh/7 no-date of 92); current-state 10-10 section (timeout-storm window re-tallied 163/15, 10 open incidents, stocktake in-flight); tailscale.md verified vs live serve status; log entry
- Wiki maintenance run of wiki-maintain.sh, manual (scheduled 1a8e90ea2271 03:00 run in-flight)
- Verified live: tailscale serve status (all routes match), cron jobs.json (30 jobs), executions.db window, SKILL.md count
- Not verified: docker-services.md (docker.sock permission denied) — left stale; Quartz rebuild inert (no quartz.config.ts)
2026-10-10 03:05:36 +08:00