Tony0410
828913e23b
audit(2026-10-14): 10-13 broke the clean streak — wiki-stocktake failed on a dead opencode credential (HTTP 404, incident 1a8e90_d11698bd7275); stranded skills-index + freshness-20261013 recovered; retention floor moved to 2026-10-08 07:11
2026-10-14 00:17:31 +08:00
hermes-audit
ebf57146f1
audit(2026-10-13): 10-12 clean (174/0, 3rd day) but GitHub Trending Radar token DEAD (getMe 401) - supersedes the wrong 2026-10-06 'refuted' entry
...
Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline.
NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450).
Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07.
2026-10-13 00:19:15 +08:00
hermes
0eee5f101e
stocktake 2026-10-12: skills-index 346 skills/70 cats; freshness 54 stale/32 fresh/7 no-date; nanobot + home-assistant re-verified live (nanobot web UI back up on :8766, OmniRoute :8900 still closed -> stays degraded; HA HTTP 200 on :8123
2026-10-12 03:17:10 +08:00
Tony0410
6fc109e546
audit(2026-10-12): cron fleet clean 2nd day (10-11 171/0, 10-10 finalised 175/0); UTC-vs-AWST false 'capture stalled' alarm corrected; proxmox storage table + host mounts; queries/retained-research-answers stub filled (VNC); capture-pipeline incident -> superseded; index stale labels fixed
2026-10-12 00:14:11 +08:00
hermes
d981e30128
stocktake(2026-10-11): skills-index 346 skills/70 cats; freshness 62 stale/24 fresh/7 no-date; 5 live-drift corrections
...
Drift found and corrected against live sources (not guessed):
- OmniRoute: CT600 omniroute is RUNNING and serving :20128 (combo API live);
CT601 omni is STOPPED with onboot=0 (peer offline 54d). Wiki had these inverted.
Hermes custom:omni resolves to omniroute -> CT600, not CT601.
- Third OmniRoute node discovered: omniroute-1 at 100.78.63.20:20128, live (401),
registered as custom provider omni-101 (5384 models, default smart-route,
carries skills_hub/free-stack). Not in pct list; identity unconfirmed.
- Ignis: TWO live instances. CT460 ignis (:8080, vault wiki/agent-estate-wiki) and
CT200 ignis-ignis-1 (image ignis-server:latest, host 8092, vault /opt/ignis/vaults).
Both return HTTP 200. Page now distinguishes them.
- Tailscale: /agent-estate targets bare 127.0.0.1:8769 (was .../8769/agent-estate-role-dashboard.html);
undocumented tailnet-only :9443 -> 127.0.0.1:5173 added; CT201 routes corrected
(8085 targets + /steel 5173 + /steel-api 3000, replacing the wrong :8443 -> :8082 row).
- Proxmox: rebuilt from live pct list + qm list. Added CT475 agentzero and VM110 omarchy
(both previously missing); 282 ubuntu-server-xfce corrected to stopped; dead rows removed.
VM403 open-claw and VM401 hermes-old absent from qm list - flagged unconfirmed, not deleted.
- Nanobot: gateway nanobot-gateway (PID 2259048) alive on 8765/8766. It is NOT systemd
nanobot.service/gateway.service (both return inactive - wrong unit names). Port 8900
no longer listening; "Serve API" claim flagged unverified.
- Docker: old "runtime not available" was a CT460 socket artifact (root:docker srw-rw----,
hermes not in group docker, no passwordless sudo in cron). Documented CT200 (~50 containers),
CT333 (9 containers), CT460 via sudo.
- Mattermost CT255 confirmed stopped; matches Tailscale peer offline 21d.
Flagged, not investigated (no root cause):
- mycompose-db-1 on CT200 is in Restarting (1) - crash-looping.
62 pages remain >21d stale and were NOT rewritten this pass (concepts, decisions,
comparisons, most runbooks) - not probed against a live source, so reported not faked.
Quartz rebuild step still inert (no quartz.config.ts) - public/ not refreshed.
Artifacts: systems/skills-index.md regenerated, raw/stocktake/freshness-20261011.txt.
2026-10-11 03:11:07 +08:00
hermes
7bffeab157
audit(2026-10-10): DHL monitor retired (29 jobs / 26 enabled, verified by ID lookup); 10-10 fleet clean 174/0, 7 incidents closed - 5 open; gateway runtime env recurrence (mnemosyne + elevenlabs) verified fixed; new systems/paseo.md; mattermost unreachable; current-state + index + log updated
2026-10-11 00:22:20 +08:00
Hermes
c11f11e652
stocktake 2026-10-10: skills-index + freshness report (62 stale/23 fresh/7 no-date of 92); current-state 10-10 section (timeout-storm window re-tallied 163/15, 10 open incidents, stocktake in-flight); tailscale.md verified vs live serve status; log entry
...
- Wiki maintenance run of wiki-maintain.sh, manual (scheduled 1a8e90ea2271 03:00 run in-flight)
- Verified live: tailscale serve status (all routes match), cron jobs.json (30 jobs), executions.db window, SKILL.md count
- Not verified: docker-services.md (docker.sock permission denied) — left stale; Quartz rebuild inert (no quartz.config.ts)
2026-10-10 03:05:36 +08:00
Tony0410
8cab69bbd0
wiki audit 2026-10-09: fleet-wide cron timeout storm (15 failures/10 jobs) + memory recall-relevance incident; backfill 2026-10-08 window; commit stranded stocktake artifacts
2026-10-10 00:27:15 +08:00
Hermes
659356aa61
wiki audit 2026-10-08: DHL Camofox timeout auto-resolved, stocktake Quartz/skills-count findings, fleet green day 3
2026-10-08 00:12:32 +08:00
Tony0410
8c1d12e29e
wiki audit 2026-10-07 (correction): real window tally 176/0/1 from executions.db TEXT timestamps; stocktake incident ids/closure times corrected
2026-10-07 00:18:43 +08:00
Tony0410
01f1db3aae
wiki audit 2026-10-07: stocktake repin verified by live run (344->346 skills), fleet 0 failing; vault skills-index re-mirrored
2026-10-07 00:15:06 +08:00
Tony0410
14b467e91b
audit(2026-10-06): dead-model-pin repin resolved; failing set 7 -> 0
...
- new incident: incidents/2026-10-05-dead-model-pin-repin.md
- scheduled-tasks.md: Count block -> 0 failing, all streaks 0; 8 job rows updated;
new 'Resolved 2026-10-05' subsection; 5 resolved bullets pruned from Open
- cliprox incident closed for cron (status: open -> resolved) + resolution section
- index.md: new incident linked, cron line + cliprox line updated
- current-state.md + log.md: 2026-10-06 entries
2026-10-06 00:20:53 +08:00
Tony0410
543e7782f8
wiki audit 2026-10-05: cliprox/terra pin settled 3/3 (Weekly Review failed 2026-10-04 18:00), failing set 6->7, streak-counter/executions.db-retention caveat
2026-10-05 00:09:58 +08:00
Tony0410
78048bdf0b
wiki audit 2026-10-04: CT450 v0.21.5 update incident, mnemosyne vector-recall FTS-only repair + coverage refresh, managed runtimes 10->3, doctor vec0 caveat
2026-10-04 00:15:35 +08:00
Tony0410
b9f69d3f75
wiki audit 2026-10-03: cliprox second job confirmed (2/3), work-mailbox incident page, streak recount, skills-index 341->344
2026-10-03 00:18:51 +08:00
Tony0410
0c4ed3350b
wiki audit 2026-10-02: 2 new cron failures, cliprox/terra connection auth incident, delivery-failure semantics corrected, profile rewrite, broken wikilinks repaired
...
- new: incidents/2026-10-01-cliproxy-connection-auth-expired.md (HTTP 401, credential fault, cannot self-recover)
- systems/scheduled-tasks.md: 30 jobs / 27 enabled; failing 3 -> 6; streaks advanced; wiki-tree drift recorded
- systems/messaging-integrations.md: two distinct thread-failure variants (lost vs thread-less fallback)
- systems/current-profile.md: full rewrite from live config (no profiles/default/, smart-route, OmniRoute)
- current-state.md / index.md / log.md: 2026-10-02 entry
- repair: copied 5 vault-only pages referenced by index.md (ignis, nextcloud, 3 incidents) so wikilinks resolve
- include pending stocktake output: skills-index.md 2026-09-27, raw/stocktake/freshness-2026-09-2{6,7}.txt
2026-10-02 00:17:15 +08:00
Tony0410
a32475bd54
wiki: nightly audit 2026-10-01 — cron fleet re-verified, Telegram delivery root cause, memory hygiene
2026-10-01 00:10:55 +08:00
Tony0410
58312cb434
Wiki audit 2026-09-30: managed-runtime dependency incident, cron fleet reconciliation, provider/model refresh
2026-09-30 19:38:49 +08:00
Tony0410
f9ac274bac
Weekly maintenance: regenerate skills-index (340 skills), freshness audit 2026-09-20 (67 stale, 5 fresh, 7 no-date)
2026-09-20 03:02:05 +08:00
Tony0410
77250ea8c9
incident: SFTPGo outage 2026-09-02 — CT299 tailscaled death + stale lxc-attach recovery
...
- New incident page documenting CT299 SFTPGo outage (02:18 AWST)
- Root cause: swap exhaustion (43Gi, 100%) killed tailscaled; 5 stale lxc-attach PIDs blocked pct start
- Resolution: killed PIDs 3037951/3334928/3352203/3402511/4004148, pct stop/start 299
- Updated current-state.md with Recent Changes (2026-09-02), bumped updated to 2026-09-04
- Updated log.md with incident entry
- Updated index.md last-updated date
No secrets written. Verified end-to-end: tailscale direct, SFTPGo WebAdmin HTTP 401, SFTP banner SSH-2.0-SFTPGo_2.7.0.
2026-09-04 00:05:32 +08:00
Tony0410
f1ea8fd228
maintain: wiki stocktake refresh + scheduled-tasks cleanup
2026-08-16 03:08:09 +08:00
Tony0410
92e8028bb8
snapshot: preserve central wiki state 2026-08-15 (7 modified + 1 new file)
2026-08-15 22:11:30 +08:00