Commit Graph
69 Commits
Author SHA1 Message Date
Tony0410 1d335ca8e6 wiki(audit): 2026-10-29 window — Conduit voice fault documented and repaired; radar two-stage split resolved both incidents (20 pages, 2 letters, Notion 455->475); cron 176/0 2026-10-30 00:21:21 +08:00
Tony0410 cc95dcc54d stocktake: re-verify Tailscale table, correct falsified cliprox/terra claim, regenerate skills-index
Wiki maintenance (wiki-maintain.sh, exit 0) + live-vs-page drift audit.

Script:
- systems/skills-index.md regenerated from the live skill tree: 302 skills in 69
  categories; 45 archived skills under ~/.hermes/skills/.archive/ now excluded and
  noted. (303 SKILL.md files exist outside .archive/, but
  productivity/obsidian-git-sync/obsidian-git-sync is a nested alias duplicate,
  so 302 unique is correct.)
- Freshness report raw/stocktake/freshness-20261029.txt: 2 stale / 105 fresh /
  3 no-date. After the corrections below and a second script run: 0 stale /
  107 fresh / 3 no-date.
- Quartz rebuilt (110 pages -> 644 files) and deployed to public/.

Drift corrections (live-verified):
- infrastructure/tailscale.md: State column rebuilt from tailscale status --json,
  11 rows changed. Two were state transitions, not age drift:
  * sftpgo (CT299, pve) is back online - tailscale ping pong in 0s,
    systemctl is-active sftpgo -> active, tailscale ip -4 inside the guest.
  * m93p is online despite a stale peer record (Online: false in the JSON) -
    tailscale ping pong in 1ms, SSH over the tailnet returned up 11 days.
    Second occurrence of this pattern after 2026-10-26.
  The other 9 rows were offline-age drift of exactly +1 day. Node count unchanged
  at 54 (53 live IPv4 peers + hermes); 76 peers total, 23 IPv6-only funnel
  ingress. No missing peers, no IP mismatches. Container identity re-confirmed
  from inside all 34 guests via pct exec tailscale ip -4 on both hypervisors;
  CT601 still absent from both.

- incidents/2026-10-01-cliproxy-connection-auth-expired.md: the residual claim
  "the connection credential itself was never re-authenticated" is falsified.
  Live POST /v1/chat/completions for cliprox/terra against custom:Omniroute:20128
  (CT600) returns HTTP 200 - the connection has since been re-authenticated. The
  same pin on custom:omni-101 (CT101) still returns HTTP 401, so the nodes carry
  separate credentials. Claim marked superseded, not deleted (SCHEMA rule 9); the
  remediation was never reversed and remains correct.

- incidents/2026-10-05-dead-model-pin-repin.md: every claim re-probed. All 7 jobs
  still on custom:omni-101 with routing aliases; both aliases HTTP 200;
  ollama-cloud/qwen3.5:397b still HTTP 410; nvidia/deepseek-v4-flash still
  HTTP 400; command-code/gpt-5.6-terra now HTTP 401 rather than the original 403
  - still dead, different failure signature.

- systems/scheduled-tasks.md: the Known-Issues entry repeating the
  "never re-authenticated" claim now carries the correction. Fleet figures
  re-checked against live jobs.json and remain accurate (29 jobs / 26 enabled /
  3 disabled, 5 non-clean last_status).

- log.md / index.md: stocktake entry appended and index last-updated refreshed.

Verified correct, no change: systems/homepage-dashboard (CT201 has 8084 and 8085
listening, no 8082, Homarr on 8102); CT201 Serve routes (/ -> 8085, /mgr -> 8084
on both :443 and :8443, Steel routes still gone); CT460 Serve routes (12 paths
on :443, plus :8443/webhook Funnel and :9443).

Noted for a human, not performed: vault mirror at /home/hermes/wiki/agent-estate-wiki
has drifted on both conventions (top-level 35 match / 3 stale / 72 missing;
wiki/ 41 match / 53 stale / 16 missing). A full mirror pass is larger than a
stocktake's mandate.
2026-10-29 03:42:54 +08:00
Tony0410andHermes 677bb0c959 docs(wiki): correct Steel Browser references across 15 pages — Steel fully removed, Camofox is sole backend
Root cause of the long-standing cloud_provider: steel residue: the 2026-07-30
rollback disabled the browser-steel plugin and set browser.engine: camofox but
left browser.cloud_provider: steel in config.yaml. The split config was
inert-but-wrong, so it survived ~3 months of audits unnoticed.

Corrects pages that still described Steel as present, running or configured:
- systems/browser-backend.md (incl. gateway-restart env-var note)
- comparisons/steel-vs-camoufox.md, comparisons/camoufox-vs-chrome.md
- decisions/rollback-steel-to-camofox.md, decisions/index.md
- infrastructure/docker-services.md, infrastructure/tailscale.md, infrastructure/hosts.md
- systems/homepage-dashboard.md, systems/scheduled-tasks.md
- concepts/browser-automation.md, runbooks/recover-docker-service.md
- current-state.md, index.md, agents/nanobot.md
- log.md (dated entry; supersedes the 00:3x audit's 'endpoint dark' wording)

Also restores the Ignis compose row in infrastructure/docker-services.md, which
the 2026-10-11 revision dropped (verified: /home/hermes/docker/ignis/docker-compose.yml
exists and the ignis container is Up on CT460).

Co-Authored-By: Hermes <hermes@local>
2026-10-29 02:02:49 +08:00
hermes ec0c30fb30 wiki(audit): 2026-10-28 window — radar stage-1 iteration cap (no handoff, stage-2 no-oped), stocktake 400 never alerted, Steel removed from the stack 2026-10-29 00:41:30 +08:00
Tony0410 f880ba981d audit: 2026-10-28 nightly — radar budget overrun now a recorded failure (new incident page); radar split into two cron jobs; tailscale peer table re-verified; TTS provider corrected to the Mac mini speech server
Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76
(Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed
failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before
Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and
cover letters never ran (Notion DB held 451 entries, 0 added). Incident row
dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified).

Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5,
deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5,
topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3).

Also: the two incident rows the 2026-10-26 planned restart raised auto-closed
00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean
last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never
committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of
108) — committed here and index.md corrected; tailscale.md peer table rebuilt from
tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo
CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to
provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained
the voice pipeline section.

New: incidents/2026-10-27-job-radar-budget-exhausted.md
Nothing removed or archived. No secrets.
2026-10-28 00:20:59 +08:00
Tony0410 a2dd0bd650 audit: 2026-10-26 nightly — planned restart orphaned a live cron run (new incident page); 503 burst fully closed; radar cover-letter PDF pipeline moved to Chromium-over-HTML 2026-10-27 00:11:50 +08:00
Tony0410 4126093644 stocktake(2026-10-26): correct 8 stale pages against live state — 0 stale remaining
wiki-maintain.sh ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted,
service restarted). The freshness report listed 8 stale pages; each was diffed
against live state and corrected only where reality could be verified. A second
script run rewrote the same-day report with 0 stale / 103 fresh / 4 no-date,
so the committed report matches the tree it ships with.

- infrastructure/hosts.md: CT600 is LIVE (not retired); CT601 does not exist on
  either hypervisor; CT333 verified running; both hypervisors verified; config
  33->51; skills 66->302; cron 25->28 (25 enabled / 3 disabled); Xvfb :102->:99;
  providers split into Omniroute:20128 (CT600) and omni-101 (CT101, default).
- agents/hermes-production.md: Tailscale IP 100.96.100.82 -> 100.118.5.51 (the
  old value is the PVE hypervisor); v0.21.5 -> v0.22.3; dead 8082 claim removed;
  fallback chain rewritten from live config (first entry is the dead nous
  credential); all 8 auxiliary slots now no-think/freeapi/auto; cron 30->28.
- agents/hermes-chaos.md: v0.22.3 (same build as CT460); "3 commits behind" ->
  0 behind origin/master; default model smart-route on custom:omni-101; cron
  "11 paused" -> 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460.
- infrastructure/tailscale.md: node table rebuilt from status --json; octoagent
  and omarchymac-1 were missing entirely (52 -> 54 non-funnel nodes); m93p now
  online; every offline age refreshed; omni row corrected to "CT601 removed
  from both hypervisors"; Serve tables re-probed per host.
- systems/memory-backend.md: re-verified against live config; coverage 1899/2350
  (80.8%), working 96.8%, episodic 99.4% — above the 95% reindex threshold.
- runbooks/provider-health.md: the step-3 OmniRoute snippet raised IndexError
  (provider renamed omniroute -> Omniroute:20128; key moved to key_env) — fixed;
  new step 3b documents the omni-101 /v1/models false-failure quirk.
- runbooks/update-hermes-safely.md: added managed-runtime dependency re-assert
  step; clarified --branch vs stable channel (v0.22.5 available).
- incidents/2026-10-03-ct450-hermes-update.md + incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md:
  Current-state sections recording current versions and embedding coverage.

All 9 corrected pages verified 200 at http://127.0.0.1:9120/<path>.
Vault mirror at /home/hermes/wiki synced for all touched files (22 copies,
sha256-verified) and committed locally; the vault is local-only and is not pushed.
No secrets written — IDs, URLs, key env var names, counts and timestamps only.
2026-10-26 04:27:49 +08:00
Tony0410 123a74cb0b audit(2026-10-26): runaway Gmail digest + dead Nous credential; model default reverted to OmniRoute; conduit 0.69.2 live; freshness floor still 2026-10-08 2026-10-26 00:19:37 +08:00
Tony0410 2df1bce5ea wiki maintenance 2026-10-25: skills-index regen (302 skills/69 cats), freshness audit 0 stale, tailscale node table + CT201 route corrections
- skills-index.md regenerated from live skill tree: 302 skills / 69 categories
  (4 skills moved to .archive/ since 10-24), 45 archived
- freshness-20261025.txt: 0 stale, 101 fresh, 4 no-date of 105 pages
- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json
  (springroll-browser was missing; omarchybook + openmausbot now offline; all offline
  ages refreshed; homeassistant exit-node wording corrected) + CT201 serve routes
  corrected (443 AND 8443; /steel only on :443)
- systems/homepage-dashboard.md: same CT201 route + false Traefik claim corrected
- reports/communities-wa-social-media-audit-2026.md: re-verified against live WA source
- incidents/2026-08-26-work-outlook-basic-auth-disabled.md: re-probed, still open
- log.md: appended stocktake entry
2026-10-25 03:40:26 +08:00
Tony0410 3271ff12a5 audit 2026-10-25: third consecutive clean cron day; new incident (agent refused 3 Conduit call requests); correct 10-22 memory-fix evidence; conduit 0.69.0/f000bf0 2026-10-25 00:16:26 +08:00
Tony0410 ec69d107ae wiki maintenance 2026-10-24: skills-index regen (306 skills/69 cats), freshness audit, fix YAML frontmatter in mnemosyne-relay incident 2026-10-24 03:06:41 +08:00
Tony0410 573a4a2434 audit 2026-10-24: conduit page + memory-start-timeout incident; 10-09 storm resolved; window 169/0; frontmatter for 2 legacy incident pages 2026-10-24 00:24:44 +08:00
Tony0410 250c73530a 🦉 wiki maintenance 2026-10-23: model-providers drift fixed (default moved to nous 2026-10-20); freshness 2 stale/94 fresh; skills-index 306 2026-10-23 03:06:37 +08:00
Tony0410 59d4b2e695 audit 2026-10-23: scheduled-tasks window (172/0, 5 incidents closed); reopen skills-index drift; skills-index generator now excludes .archive (306 skills, was 347) 2026-10-23 00:14:49 +08:00
Tony0410 1d92de31a0 🦉 wiki maintenance 2026-10-22: Quartz rebuild+deploy (new incident page live), skills-index 348, freshness 1 stale/95 fresh 2026-10-22 00:19:03 +08:00
Tony0410 3919ec8557 🦉 wiki audit 2026-10-22: omni-101 pre-dispatch filter skip (3rd failure mode, new page), Hermes 0.22.3 update with snapshot skipped, Gitea backup backlog 76
- new: incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- systems/hermes-agent.md: version -> 0.22.3 (5b147ab036), 22:47-22:57 update, snapshot stage skipped, advisories unactioned
- systems/scheduled-tasks.md: 2026-10-21 = 171 completed / 1 failed; non-clean stays 9 with 08eb86f02d37 out, dca8482e4f76 in
- systems/omniroute.md: one pin, three signatures in eight days
- incidents/2026-10-16-omniroute-503-resource-pressure.md: window update, stays monitoring
- incidents/2026-08-08-gitea-config-backup-push-dead.md: 2026-10-21 run, backlog 75 -> 76, message_id 8449
- infrastructure/backups.md: 76 commits; pre_update_backup:false now exercised
- current-state.md, index.md, log.md updated
2026-10-22 00:16:39 +08:00
Tony0410 c1d3b59260 🦉 wiki stocktake 2026-10-21: tailscale node table rebuilt, 2 missing Uber Eats reports recovered
Maintenance script ran clean (skills-index 346 -> 347, freshness audit, Quartz rebuilt
596 files and deployed, hermes-wiki-serve.service restarted).

infrastructure/tailscale.md — node table rebuilt from live `tailscale status --json`.
  The 2026-10-19 table had drifted: octop (100.92.163.122) and springroll
  (100.100.153.125) missing entirely, openmausbot / openmausbot-vm present in prose
  but absent from the table body, omarchymac now offline, all offline ages stale.
  Rebuilt keyed on DNS name rather than the JSON HostName — HostName is `ct260` for
  the freshrss node and BOTH OmniRoute nodes report the hostname `omniroute`, which
  collides silently. Now 51 non-funnel nodes. Container states cross-checked against
  live `pct list` on pve and m93p: CT255 stopped, CT601 stopped, CT299 running.

reports/uber-eats/ — 2026-08 and 2026-09 recovered. The generator wrote only to
  ~/wiki/reports/uber-eats/ and never to the canonical tree, so both months were
  missing from the published wiki. Root cause fixed in
  ~/.hermes/cron/scripts/generate-uber-eats-report.py: it now targets
  ~/workspace/agent-estate-wiki/reports/uber-eats/ and mirrors to the legacy tree,
  and now emits YAML frontmatter (without it the pages landed in the audit's
  "no updated DATE" bucket and Quartz could not read the page date). 2026-08
  previous-month column corrected from $0.00 to $264.51 / 11 orders.

incidents/2026-08-15-skills-index-drift.md — re-verified live; the named
  regenerate-skills-index.sh no longer exists on disk (regeneration now lives in
  wiki-maintain.sh step 1/3). Incident prevention confirmed working: 347 skills.
incidents/2026-09-02-sftpgo-outage.md — re-verified live: CT299 running, health
  endpoint HTTP 401 (auth required, service up), no recurrence.

reports/communities-wa-social-media-audit-2026.md — left stale (54d). Its source
  document no longer exists on disk, so the baseline cannot be re-verified against
  a live source. Flagged, not edited.

Freshness after this run: 1 stale, 94 fresh, 6 no-date of 101 pages.
2026-10-21 03:14:26 +08:00
Tony0410 b662254569 🦉 wiki audit 2026-10-21: gateway unclean-exit orphaned run (new), Gitea config-backup push dead since 2026-08-08 (new), model default → nous, 503 back to monitoring, canned-substitute trigger corrected 2026-10-21 00:11:44 +08:00
Tony0410 fbd4f297eb wiki: publish pending edits (2026-10-20) 2026-10-20 18:38:11 +08:00
Hermes Rhino 8da6ddfa4d audit(2026-10-20): annotate the historical 10-17 162/5 tallies with the DB-corrected 163/5; mark the 10-19 count superseded 2026-10-20 00:14:02 +08:00
Hermes Rhino 95fd7d33f1 audit(2026-10-20): omni-101 503 burst 2 — incident back to active; scheduled-tasks rebuilt from jobs.json
- burst 2 (2026-10-19): 9 model calls across 8 jobs, 159/13 day
- burst-2 table names/schedules/aliases corrected from live jobs.json
- non-clean last_status 7 -> 12; unclosed incident rows 13 (query stated)
- 2026-10-17 tail corrected 162/5 -> 163/5
2026-10-20 00:12:39 +08:00
Tony0410 93c55efc94 stocktake 2026-10-19: skills-index 346/70; freshness 47 stale/43 fresh/7 no-date; open-claw VM 403 verified absent from PVE 2026-10-19 03:03:15 +08:00
Tony0410 0ed522bfb8 audit(2026-10-19): degenerate-stream incident page + push-notifications page; refresh scheduled-tasks, current-state, index, log, messaging, mnemosyne, tailscale, proxmox 2026-10-19 00:21:26 +08:00
Tony0410 dd53e5fbe1 stocktake 2026-10-18: skills-index 346 skills/70 cats; freshness 47 stale/41 fresh/7 no-date; quartz build skipped (no quartz.config.ts) 2026-10-18 03:01:17 +08:00
hermes-audit 4137dd6945 audit 2026-10-18: new incident page (omniroute-1 503 resource pressure, 9 calls / 7 jobs), fleet re-verified 28 jobs / 25 enabled / 8 non-clean, Mosquito Tracker removed, radar bot token dead 2026-10-18 00:41:38 +08:00
Tony0410 76de919e6a stocktake 2026-10-16: skills-index 346 skills/70 cats; freshness 47 stale/40 fresh/7 no-date; tailscale re-verified live — omarchymac (100.84.180.69) added, mattermost state corrected, offline ages refreshed (46 non-funnel nodes) 2026-10-16 03:04:25 +08:00
hermes 2061b8607b audit(2026-10-16): fleet-wide 34h cron blackout 10-14 -> 10-15 — omniroute-1 refused connections
omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07
(httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error).
36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog
symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1).

STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable
pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls
completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence.

New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live).
Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute,
systems/mnemosyne (monitor blast-radius note), current-state, index, log.
2026-10-16 00:19:14 +08:00
Tony0410 deb6a8eec2 stocktake 2026-10-14: SECOND Proxmox host m93p (100.104.51.98, PVE 9.2.0, unclustered, IDs collide) discovered — gitea/ntfy/freshrss/openweb were never on pve; tailnet inventory rebuilt 30 -> 45 nodes (15 missing); VM 282 running; Camofox recovered to 29.9M
skills-index 346 skills / 70 cats; freshness 48 stale / 38 fresh / 7 no-date of 93 pages

- infra/proxmox.md: two-host rewrite, m93p LXC+QEMU inventory, 4-entry correction table, VM 282 stopped->running, storage via pvesm (ssd0 88.17%/63GiB free)
- infra/tailscale.md: 45-node inventory rebuilt from live status; 23 funnel-ingress-node IPv6 peers noted+excluded; CT460 serve routes re-verified exactly (13 routes + :8443 Funnel + :9443)
- infra/network-map.md, current-state.md, index.md, log.md updated
- Open: 3 incidents (unverified pages left stale, not rewritten); omniroute/hosts/model-providers/docker-services still single-host in framing — m93p runs its own omniroute CT101, needs a dedicated pass
2026-10-14 03:13:04 +08:00
Tony0410 828913e23b audit(2026-10-14): 10-13 broke the clean streak — wiki-stocktake failed on a dead opencode credential (HTTP 404, incident 1a8e90_d11698bd7275); stranded skills-index + freshness-20261013 recovered; retention floor moved to 2026-10-08 07:11 2026-10-14 00:17:31 +08:00
hermes-audit 1df564e152 audit(2026-10-13): close out the 2026-10-09 open-items snapshot (DHL retire DONE 2026-10-10; all eight storm incidents cleared) 2026-10-13 00:19:44 +08:00
hermes-audit ebf57146f1 audit(2026-10-13): 10-12 clean (174/0, 3rd day) but GitHub Trending Radar token DEAD (getMe 401) - supersedes the wrong 2026-10-06 'refuted' entry
Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline.

NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450).

Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07.
2026-10-13 00:19:15 +08:00
hermes 0eee5f101e stocktake 2026-10-12: skills-index 346 skills/70 cats; freshness 54 stale/32 fresh/7 no-date; nanobot + home-assistant re-verified live (nanobot web UI back up on :8766, OmniRoute :8900 still closed -> stays degraded; HA HTTP 200 on :8123 2026-10-12 03:17:10 +08:00
Tony0410 6fc109e546 audit(2026-10-12): cron fleet clean 2nd day (10-11 171/0, 10-10 finalised 175/0); UTC-vs-AWST false 'capture stalled' alarm corrected; proxmox storage table + host mounts; queries/retained-research-answers stub filled (VNC); capture-pipeline incident -> superseded; index stale labels fixed 2026-10-12 00:14:11 +08:00
hermes d981e30128 stocktake(2026-10-11): skills-index 346 skills/70 cats; freshness 62 stale/24 fresh/7 no-date; 5 live-drift corrections
Drift found and corrected against live sources (not guessed):
- OmniRoute: CT600 omniroute is RUNNING and serving :20128 (combo API live);
  CT601 omni is STOPPED with onboot=0 (peer offline 54d). Wiki had these inverted.
  Hermes custom:omni resolves to omniroute -> CT600, not CT601.
- Third OmniRoute node discovered: omniroute-1 at 100.78.63.20:20128, live (401),
  registered as custom provider omni-101 (5384 models, default smart-route,
  carries skills_hub/free-stack). Not in pct list; identity unconfirmed.
- Ignis: TWO live instances. CT460 ignis (:8080, vault wiki/agent-estate-wiki) and
  CT200 ignis-ignis-1 (image ignis-server:latest, host 8092, vault /opt/ignis/vaults).
  Both return HTTP 200. Page now distinguishes them.
- Tailscale: /agent-estate targets bare 127.0.0.1:8769 (was .../8769/agent-estate-role-dashboard.html);
  undocumented tailnet-only :9443 -> 127.0.0.1:5173 added; CT201 routes corrected
  (8085 targets + /steel 5173 + /steel-api 3000, replacing the wrong :8443 -> :8082 row).
- Proxmox: rebuilt from live pct list + qm list. Added CT475 agentzero and VM110 omarchy
  (both previously missing); 282 ubuntu-server-xfce corrected to stopped; dead rows removed.
  VM403 open-claw and VM401 hermes-old absent from qm list - flagged unconfirmed, not deleted.
- Nanobot: gateway nanobot-gateway (PID 2259048) alive on 8765/8766. It is NOT systemd
  nanobot.service/gateway.service (both return inactive - wrong unit names). Port 8900
  no longer listening; "Serve API" claim flagged unverified.
- Docker: old "runtime not available" was a CT460 socket artifact (root:docker srw-rw----,
  hermes not in group docker, no passwordless sudo in cron). Documented CT200 (~50 containers),
  CT333 (9 containers), CT460 via sudo.
- Mattermost CT255 confirmed stopped; matches Tailscale peer offline 21d.

Flagged, not investigated (no root cause):
- mycompose-db-1 on CT200 is in Restarting (1) - crash-looping.

62 pages remain >21d stale and were NOT rewritten this pass (concepts, decisions,
comparisons, most runbooks) - not probed against a live source, so reported not faked.
Quartz rebuild step still inert (no quartz.config.ts) - public/ not refreshed.

Artifacts: systems/skills-index.md regenerated, raw/stocktake/freshness-20261011.txt.
2026-10-11 03:11:07 +08:00
hermes 7bffeab157 audit(2026-10-10): DHL monitor retired (29 jobs / 26 enabled, verified by ID lookup); 10-10 fleet clean 174/0, 7 incidents closed - 5 open; gateway runtime env recurrence (mnemosyne + elevenlabs) verified fixed; new systems/paseo.md; mattermost unreachable; current-state + index + log updated 2026-10-11 00:22:20 +08:00
Hermes c11f11e652 stocktake 2026-10-10: skills-index + freshness report (62 stale/23 fresh/7 no-date of 92); current-state 10-10 section (timeout-storm window re-tallied 163/15, 10 open incidents, stocktake in-flight); tailscale.md verified vs live serve status; log entry
- Wiki maintenance run of wiki-maintain.sh, manual (scheduled 1a8e90ea2271 03:00 run in-flight)
- Verified live: tailscale serve status (all routes match), cron jobs.json (30 jobs), executions.db window, SKILL.md count
- Not verified: docker-services.md (docker.sock permission denied) — left stale; Quartz rebuild inert (no quartz.config.ts)
2026-10-10 03:05:36 +08:00
Tony0410 8cab69bbd0 wiki audit 2026-10-09: fleet-wide cron timeout storm (15 failures/10 jobs) + memory recall-relevance incident; backfill 2026-10-08 window; commit stranded stocktake artifacts 2026-10-10 00:27:15 +08:00
Hermes 659356aa61 wiki audit 2026-10-08: DHL Camofox timeout auto-resolved, stocktake Quartz/skills-count findings, fleet green day 3 2026-10-08 00:12:32 +08:00
Tony0410 8c63ef24f0 chore: regenerate skills-index + freshness report 2026-10-07 2026-10-07 03:03:19 +08:00
Tony0410 8c1d12e29e wiki audit 2026-10-07 (correction): real window tally 176/0/1 from executions.db TEXT timestamps; stocktake incident ids/closure times corrected 2026-10-07 00:18:43 +08:00
Tony0410 01f1db3aae wiki audit 2026-10-07: stocktake repin verified by live run (344->346 skills), fleet 0 failing; vault skills-index re-mirrored 2026-10-07 00:15:06 +08:00
Tony0410 33f4e95d95 wiki-maintain: regenerate skills-index and freshness stocktake 2026-10-06 2026-10-06 03:03:02 +08:00
Tony0410 14b467e91b audit(2026-10-06): dead-model-pin repin resolved; failing set 7 -> 0
- new incident: incidents/2026-10-05-dead-model-pin-repin.md
- scheduled-tasks.md: Count block -> 0 failing, all streaks 0; 8 job rows updated;
  new 'Resolved 2026-10-05' subsection; 5 resolved bullets pruned from Open
- cliprox incident closed for cron (status: open -> resolved) + resolution section
- index.md: new incident linked, cron line + cliprox line updated
- current-state.md + log.md: 2026-10-06 entries
2026-10-06 00:20:53 +08:00
Tony0410 543e7782f8 wiki audit 2026-10-05: cliprox/terra pin settled 3/3 (Weekly Review failed 2026-10-04 18:00), failing set 6->7, streak-counter/executions.db-retention caveat 2026-10-05 00:09:58 +08:00
Tony0410 78048bdf0b wiki audit 2026-10-04: CT450 v0.21.5 update incident, mnemosyne vector-recall FTS-only repair + coverage refresh, managed runtimes 10->3, doctor vec0 caveat 2026-10-04 00:15:35 +08:00
Tony0410 b9f69d3f75 wiki audit 2026-10-03: cliprox second job confirmed (2/3), work-mailbox incident page, streak recount, skills-index 341->344 2026-10-03 00:18:51 +08:00
Tony0410 0c4ed3350b wiki audit 2026-10-02: 2 new cron failures, cliprox/terra connection auth incident, delivery-failure semantics corrected, profile rewrite, broken wikilinks repaired
- new: incidents/2026-10-01-cliproxy-connection-auth-expired.md (HTTP 401, credential fault, cannot self-recover)
- systems/scheduled-tasks.md: 30 jobs / 27 enabled; failing 3 -> 6; streaks advanced; wiki-tree drift recorded
- systems/messaging-integrations.md: two distinct thread-failure variants (lost vs thread-less fallback)
- systems/current-profile.md: full rewrite from live config (no profiles/default/, smart-route, OmniRoute)
- current-state.md / index.md / log.md: 2026-10-02 entry
- repair: copied 5 vault-only pages referenced by index.md (ignis, nextcloud, 3 incidents) so wikilinks resolve
- include pending stocktake output: skills-index.md 2026-09-27, raw/stocktake/freshness-2026-09-2{6,7}.txt
2026-10-02 00:17:15 +08:00
Tony0410 a32475bd54 wiki: nightly audit 2026-10-01 — cron fleet re-verified, Telegram delivery root cause, memory hygiene 2026-10-01 00:10:55 +08:00
Tony0410 4e7d99064e wiki: document vault repo git object-permission quirk + 2026-09-30 object recovery 2026-09-30 19:55:41 +08:00
Tony0410 58312cb434 Wiki audit 2026-09-30: managed-runtime dependency incident, cron fleet reconciliation, provider/model refresh 2026-09-30 19:38:49 +08:00