Hermes Wiki Audit
1b0cf1fa71
audit(2026-11-11) follow-up: settle the incident-ledger row count and note acked_at is NULL
2026-11-12 00:18:26 +08:00
Hermes Wiki Audit
729bb597cf
audit(2026-11-11): the audit's own 429 killed the 11-10 window; stocktake + radar recover with output
...
Window 2026-11-10 23:59 -> 2026-11-11 23:59 (two windows: the 11-10 audit failed).
NEW incident: incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
- cbfe434a3b5e ran 23:59:54.486572 -> 2026-11-11 00:12:29.646987 = 12m35s, failed, delivery suppressed
- RuntimeError: HTTP 429 ... all targets skipped by pre-dispatch filters (credits 100% conn:3021bab6)
reset after 6h24m31s; 6 candidates, attempted 0; tenant tn-5ud77abtl1 quota exceeded
- incident row cbfe43_ed01845310d8 (rate_limit) alerted_at NULL, closed_at NULL = only open row
- consequence: the 2026-11-10 window was never audited; non-clean last_status 1 -> 2
RESOLVED with output (dated Recovery sections added):
- stocktake 1a8e90ea2271 03:00:56.081504 -> 04:11:38.724852 = 1h10m42s, 36 stale re-verified -> 0
- job radar stage1 07:01:08 -> 07:37:34 = 36m26s, first handoff since 2026-11-06 (22,682 B)
- radar stage2 recovery fire 07:35:56 -> 08:03:47 closed row 5fc292_773822455856 resolved
UPDATED pages: current-state.md, systems/scheduled-tasks.md, systems/opencode-go.md
(opencode-zen 401 api_key_expired now observed 7x, paired with nous 404 credits 14x),
incidents/2026-10-22-nous-portal-refresh-token-invalid.md, index.md, log.md
2026-11-12 00:16:39 +08:00
hermes
b820993ecc
wiki audit 2026-11-07: Job Radar concurrent-fire claim discard (new incident page), scheduled-tasks re-verified section, index+log+runbook updates, freshness 2026-11-07
2026-11-07 00:48:09 +08:00
hermes
3b5daccb4d
wiki: nightly audit 2026-11-05 window — network-map re-verified by live probe (9377=Camofox not Stratos, 8790/5174 gone, +9120/9222/7777/6767/8085/9900, full Tailscale Serve table, two dead routes 8769/9123); commit pending 11-04 stocktake re-verifications; log + skills-index
2026-11-05 01:22:45 +08:00
hermes
e4086ec18e
wiki: nightly audit 2026-11-04 window — new incident (Job Radar + Newsletter Digest both killed by upstream stream failure; Stage 2 rebuilt handoff and delivered 5 roles + 1 letter); CT299 second freeze documented; CL_MODEL 402 caveat in job-radar runbook; scheduled-tasks/current-state/index/log updated
2026-11-05 01:04:31 +08:00
Hermes Wiki Audit
785fbc40b2
wiki: nightly audit 2026-11-03 window — new Job Radar stream-timeout incident, hermes-hq-push system page; update 11-02 recurrence, scheduled-tasks, index, log
2026-11-04 00:16:23 +08:00
hermes
74a5b2d117
audit(2026-10-31): document CT460 5h30m host outage; correct the 'paused' health monitor (it was running, closed completed 10:02:48); reconcile fleet counts to 26/3
2026-10-31 00:39:11 +08:00
hermes
ec0c30fb30
wiki(audit): 2026-10-28 window — radar stage-1 iteration cap (no handoff, stage-2 no-oped), stocktake 400 never alerted, Steel removed from the stack
2026-10-29 00:41:30 +08:00
hermes
2061b8607b
audit(2026-10-16): fleet-wide 34h cron blackout 10-14 -> 10-15 — omniroute-1 refused connections
...
omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07
(httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error).
36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog
symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1).
STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable
pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls
completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence.
New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live).
Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute,
systems/mnemosyne (monitor blast-radius note), current-state, index, log.
2026-10-16 00:19:14 +08:00
hermes
0eee5f101e
stocktake 2026-10-12: skills-index 346 skills/70 cats; freshness 54 stale/32 fresh/7 no-date; nanobot + home-assistant re-verified live (nanobot web UI back up on :8766, OmniRoute :8900 still closed -> stays degraded; HA HTTP 200 on :8123
2026-10-12 03:17:10 +08:00
Hermes
c11f11e652
stocktake 2026-10-10: skills-index + freshness report (62 stale/23 fresh/7 no-date of 92); current-state 10-10 section (timeout-storm window re-tallied 163/15, 10 open incidents, stocktake in-flight); tailscale.md verified vs live serve status; log entry
...
- Wiki maintenance run of wiki-maintain.sh, manual (scheduled 1a8e90ea2271 03:00 run in-flight)
- Verified live: tailscale serve status (all routes match), cron jobs.json (30 jobs), executions.db window, SKILL.md count
- Not verified: docker-services.md (docker.sock permission denied) — left stale; Quartz rebuild inert (no quartz.config.ts)
2026-10-10 03:05:36 +08:00
Hermes
659356aa61
wiki audit 2026-10-08: DHL Camofox timeout auto-resolved, stocktake Quartz/skills-count findings, fleet green day 3
2026-10-08 00:12:32 +08:00