diff --git a/incidents/2026-07-29-mnemosyne-relay.md b/incidents/2026-07-29-mnemosyne-relay.md index 40017ad..aabad0d 100644 --- a/incidents/2026-07-29-mnemosyne-relay.md +++ b/incidents/2026-07-29-mnemosyne-relay.md @@ -3,9 +3,8 @@ title: 'Mnemosyne Relay Scope Corruption + `tools: None` — 2026-07-29' type: incident status: resolved created: 2026-07-29 -updated: 2026-10-24 -verified_on: 2026-10-24 -review_after: 2026-11-07 +updated: 2026-11-12 +verified_on: 2026-11-12 confidence: high tags: [mnemosyne, config, relay, incident, resolved] sources: [live-probe, config, session-20260729] @@ -51,4 +50,21 @@ Scope push/pop ordering mismatch. Something pushes a scope that isn't at the top Filed (or should be filed) under NousResearch/hermes-agent issues. No token available to auto-create. ## MONITORING -Watch for recurrence. If persists, check whether the affected sessions share a common tool call pattern (subagent delegation, tool failure, etc.). \ No newline at end of file +Watch for recurrence. If persists, check whether the affected sessions share a common tool call pattern (subagent delegation, tool failure, etc.). + +## Re-verified 2026-11-12 03:3x AWST (stocktake — page was 19d old, inside the staleness window) +Both halves of this page re-checked against live state and **the fix holds; neither symptom has recurred** +in ~3.5 months and 106 days of logs: +- `grep -n "tools:" ~/.hermes/config.yaml` → the `memory.mnemosyne` block no longer carries `tools: None` + (the literal `None` key is absent), so the `Unknown Mnemosyne tool(s)… None` schema warning has no + trigger left. +- `grep -c "Unknown Mnemosyne tool" ~/.hermes/logs/gateway.log` and `.log.1`–`.log.3` → **0 in every + rotated file** (4 generations, 0 occurrences). +- `grep -ah "scope handle is not at the top" ~/.hermes/logs/gateway.log*` → **no occurrences**; the + relay-scope corruption has not repeated since 2026-07-29 08:10 UTC. +- Mnemosyne itself is healthy on the current build: `mnemosyne stats` → **2782 memories (1062 working / + 1217 episodic)** and `mnemosyne recall "tailscale serve routes"` returns substantive ranked results, so + the relay/memory path is functioning. + +The `REPORTED` action above still stands (no token to auto-create a NousResearch issue); the **MONITORING** +condition is now satisfied — no recurrence across four log generations. \ No newline at end of file diff --git a/incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md b/incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md index e278782..de61b66 100644 --- a/incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +++ b/incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md @@ -3,9 +3,8 @@ title: OmniRoute free-DeepSeek Silent Stops (Empty-After-Tool) — 2026-08-04 type: incident status: monitoring created: 2026-08-04 -updated: 2026-10-24 -verified_on: 2026-10-24 -review_after: 2026-11-07 +updated: 2026-11-12 +verified_on: 2026-11-12 confidence: high tags: [omniroute, deepseek, free-tier, silent-failure, incident, mitigated] sources: [live-probe, omniroute-log, session-20260804] @@ -53,5 +52,23 @@ Backup of both pre-change rows: `/tmp/combo_backup_20260804.json` on the Hermes - Consider `auto`+explorationRate for self-tuning if free tier stays flaky. - Hermes-side: empty-post-tool turns are nudge/retry/fallback handled, but still end silently once exhausted — possible future improvement: surface "(empty after tool calls)" more visibly. +## Re-verified 2026-11-12 03:4x AWST (stocktake — page was 19d old, inside the staleness window) +Both reworked combos probed live on **`omniroute-1`** (`100.78.63.20:20128`, `custom:omni-101`) with the +correct `key_env` (`HERMES_CUSTOM_100_78_63_20_20128_API_KEY`) and the exact aliases used: + +| Probe | Result | +|---|---| +| `good-free` (the 12-model rework) | **HTTP 200**, resolving to **`glm-5.2-cheap:free`** with a valid non-empty completion | +| `free-stack` | **HTTP 200**, resolving to **`stepfun-ai/step-5-preview`** with a valid completion | + +**The mitigation is holding, and the 2026-08-04 follow-up can be partially closed:** both combos answer with +real content, and **neither resolves to a deepseek model** — which is exactly the intent of dropping the +proven culprits from `good-free`. ⚠ **What is *not* re-verified:** I did not individually probe the +free-DeepSeek upstreams (`238d5978/deepseek-v4-flash`, `nvidia/deepseek-ai/deepseek-v4-pro`, +`deepseek-v4-flash-thinking:free`), so the original claim *"upstream free-DeepSeek tier still degraded"* +stands **neither confirmed nor refuted** — the combos serve healthy non-deepseek free models, which may mean +the upstream recovered *or* simply that it is no longer reached. **Restoring deepseek-first ordering is +therefore still not justified** and should wait on a direct probe of those three upstreams. + ## Refs - [[systems/omniroute]] diff --git a/incidents/2026-08-26-work-outlook-basic-auth-disabled.md b/incidents/2026-08-26-work-outlook-basic-auth-disabled.md index 2c60129..68c84b0 100644 --- a/incidents/2026-08-26-work-outlook-basic-auth-disabled.md +++ b/incidents/2026-08-26-work-outlook-basic-auth-disabled.md @@ -3,8 +3,8 @@ title: Work Outlook Mailbox — Basic Auth Disabled (Himalaya) type: incident status: open created: 2026-10-03 -updated: 2026-10-25 -verified_on: 2026-10-25 +updated: 2026-11-12 +verified_on: 2026-11-12 confidence: high tags: [email, himalaya, office365, oauth2, auth, incident] sources: [live-himalaya-debug, live-tcp-probe, cron-session-20260826_180700_a94098] @@ -12,7 +12,16 @@ sources: [live-himalaya-debug, live-tcp-probe, cron-session-20260826_180700_a940 # Work Outlook Mailbox — Basic Auth Disabled (Himalaya) -**Window:** first detected 2026-08-26; re-verified 2026-10-03 and again 2026-10-25 (still open) +**Re-verified 2026-11-12 03:3x AWST (stocktake — page was 18d old, inside the staleness window):** +the failure is unchanged and still reproduces exactly, with the **same error class**. `himalaya envelope +list -a work -s 1` → `Error: cannot build IMAP client / 1: login not supported` — i.e. the server rejects +the basic-auth `LOGIN` command itself rather than the credential, which is consistent with the SMTP/IMAP +basic-auth shutdown recorded above rather than a changed password. **No remediation has landed since +2026-10-25:** the account still authenticates by password (there is no OAuth2 token in the himalaya +config), so path 1 remains the only viable route and still requires Anthony's sign-in. The capability gap +described below is unchanged — no cron job polls this mailbox. + +**Window:** first detected 2026-08-26; re-verified 2026-10-03, 2026-10-25 and 2026-11-12 (still open) **Status:** Open — blocked pending an OAuth2 path **Impact:** The agent cannot read or send from the `work` mailbox. Every work-email request is unanswerable until this is fixed. diff --git a/incidents/2026-09-02-sftpgo-outage.md b/incidents/2026-09-02-sftpgo-outage.md index a4253a8..f4abcfa 100644 --- a/incidents/2026-09-02-sftpgo-outage.md +++ b/incidents/2026-09-02-sftpgo-outage.md @@ -3,8 +3,8 @@ title: SFTPGo Outage — CT299 Offline (2026-09-02) type: incident status: resolved created: 2026-09-04 -updated: 2026-10-21 -verified_on: 2026-10-21 +updated: 2026-11-12 +verified_on: 2026-11-12 confidence: high tags: [incident, sftpgo, ct299, tailscale, proxmox] sources: [session:20260902_014540_c193be] @@ -68,6 +68,20 @@ the node is `active` in `tailscale status` at `100.88.223.70`; the SFTPGo health `http://100.88.223.70:10080/healthz` returns **HTTP 401** (expected — auth required, service up). The 2026-09-02 outage remains resolved; no recurrence. +## Re-verified 2026-11-12 03:2x AWST (stocktake — page was 22d stale, the wiki's only stale page) +Live state re-probed and **the residual claim still holds — no recurrence in 70 days.** All three sources +checked independently: +- `pct list` on `pve` (100.96.100.82) → **CT299 `sftpgo` `running`**; `pct config 299` confirms the guest + (2 cores, 2048 MB RAM, **`swap: 512`** — the over-provisioned 43 Gi swap recorded in the root cause is + gone, so that concern has since been addressed). +- `tailscale status` peer record for `sftpgo` → **`Online: true`**, `100.88.223.70`, and + `tailscale ping sftpgo` → **`pong from sftpgo (100.88.223.70) via 192.168.178.92:41641 in 0s`**. +- SFTPGo health endpoint `http://100.88.223.70:10080/healthz` → **HTTP 401**, unchanged (auth required, service up). + +Note that this outage and the two later CT299 freezes are **separate faults**: the 2026-09-02 cause was +`tailscaled` death plus stale `lxc-attach` cgroup pinning, whereas 2026-11-04/05 was a wedged `vzdump` +freezing the guest → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]] (also resolved). + ## Lessons Learned 1. **Clean up `lxc-attach` probes** — always exit cleanly and verify no lingering processes after diagnostic sessions diff --git a/incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md b/incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md index ca50448..1ad71b3 100644 --- a/incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md +++ b/incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md @@ -3,9 +3,9 @@ title: Daily Marketing Job Radar — "All Targets Skipped by Pre-Dispatch Filter type: incident status: unresolved created: 2026-10-22 -updated: 2026-10-22 -verified_on: 2026-10-22 -review_after: 2026-10-29 +updated: 2026-11-12 +verified_on: 2026-11-12 +review_after: 2026-11-26 confidence: high tags: [cron, router, provider, job-search, failure-mode] sources: [live-jobs-json, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-cron-output-file] @@ -62,6 +62,31 @@ One weekday radar slot produced nothing: no new roles discovered, verified, de-d job DB, and no candidate summary delivered to topic `:1030`. The job recovers on its next weekday slot (2026-10-22 07:00) if the guard does not repeat. +## RECURRENCE — 2026-11-10, on a different job and a different pin (verified live 2026-11-12) + +The signature is **not unique to the Job Radar**, and it has now recurred on a second job — which changes +the "one job's guard" hypothesis of the original open questions into a fleet-level failure mode. + +| Probe | Result | +|---|---| +| `executions.db` (retention floor **2026-11-06T05:09:30+08:00**, so this is an in-window read and **not** an all-time sweep) | exactly **one** row carries the string: **`cbfe434a3b5e` (Wiki Audit — Nightly), 2026-11-10, `status: failed`**, error `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters…` | +| the original 2026-10-21 Job Radar row | **outside the retention floor**, so it cannot be re-read from the DB today; its evidence above stands on the 2026-10-22 capture. The per-job answer is unchanged in `jobs.json`: `dca8482e4f76` reads `ok` today | +| `cron_incidents` row `cbfe43_ed01845310d8` | `failure_type: rate_limit`, `first_seen_at 2026-11-11T00:12`, **`closed_at: 2026-11-12T00:25` → `resolved`** (it was still `detected`/open at the 2026-11-11 audit) | +| `dca8482e4f76` (Job Radar, the original subject) | **4 consecutive clean slots** — 2026-11-06, 11-09, 11-10, 11-11 all `completed`. The radar itself has **not** repeated the fault | +| the recurring job's pin | `cbfe434a3b5e` runs `custom:omni-101` / **`free-stack`** — **not** the `smart-route` pin the original failure used | + +**What this adds to the original diagnosis:** the string fires on at least **two different models** and +**two different jobs**, so open question 1 ("if it is a local filter chain, one filter is probably +mis-configured for this job's toolset") now looks **less likely** than the router-side-guard reading — a +guard that blanks *every* target. The **prefix differs** between the two occurrences (`HTTP 429` in 2026-11, +bare `Service temporarily unavailable` in 2026-10), which is the same "error class changes, verdict holds" +pattern already recorded for the 401/403 cliproxy case → [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. +The 2026-11 instance was **credits-driven** (`credits 100%`, guard reset after 6 h 24 m) — a quota +exhaustion, not a config fault, which also matches the fleet's paid-pin posture in +[[systems/scheduled-tasks]] (this stocktake itself runs `deepseek-paid-smart`, the only paid pin in the fleet). +Open question 3 (**give the radar a second provider pin — it has no failover**) therefore now applies to the +whole free-tier cohort, not just the radar, and remains unapplied pending Anthony's go-ahead. + ## Open questions / next actions (proposed, not applied — needs Anthony's go-ahead) 1. **Find which component emits this string.** If it is a router-side guard with a fixed reset window, a diff --git a/index.md b/index.md index 2a921ba..42a7dc2 100644 --- a/index.md +++ b/index.md @@ -3,8 +3,8 @@ title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 -updated: 2026-11-11 -verified_on: 2026-11-11 +updated: 2026-11-12 +verified_on: 2026-11-12 confidence: high tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout] sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake] @@ -12,7 +12,11 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-11-09. +> model routing and automation. Last updated: 2026-11-12. +> +> **2026-11-12 (03:00 stocktake) — the freshness scan was clean, the live-vs-page diff was not.** The age scan flagged **1 stale / 129 fresh / 3 no-date of 133** (the sftpgo incident at 22d, re-verified live and cleared). But [[infrastructure/tailscale.md]] — only **2 days old** — had drifted: **`mailflow`, `qwenpaw` and `springroll` all read `Online: false` in the peer record while all three ping fine** (CT1262 and CT465 `running` in `pct list`; springroll answers via **DERP(sin)**, relay not direct), and `omarchymac` was stale in the *other* direction (page said `offline 2d / last seen 2026-11-08 00:45Z`; live reads `LastSeen 2026-11-07T06:41:48Z` = **offline 4d**, ping timeout). Node table rebuilt by script, all IPs matching, no rows missing, **58 rows = 57 live IPv4 peers + `hermes`**, 80 peers total (57 IPv4 / 23 IPv6-only) unchanged. Also re-probed: CT299 `sftpgo` running/online/healthz-401; both DEAD Serve targets (8769, 9123) still listener-less; CT201's Serve table unchanged; container counts **17 LXCs + 4 VMs on `pve`** and **20 LXCs + 2 VMs on `m93p`**; and **new guest CT1007 `homebridge` on m93p — running but `tailscaled` `inactive`**, so it does not join the tailnet. +> +> **2026-11-12 (03:00 stocktake) — a "router" failure mode turned out to be fleet-level.** [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] recorded *"all targets were skipped by pre-dispatch filters"* as a Job Radar one-off and left three open questions. Live re-read of `executions.db` shows the **same signature on a second job and a second pin**: `cbfe434a3b5e` (Wiki Audit, `custom:omni-101` / **`free-stack`**) failed on **2026-11-10** with `HTTP 429 … (credits 100%)`, incident row `cbfe43_ed01845310d8` (`rate_limit`) closed `resolved` **2026-11-12T00:25**. The original subject has **not** repeated — `dca8482e4f76` ran clean 11-06/09/10/11. So open question 1 (a job-specific mis-configured filter) now looks **less likely** than a router-side guard that blanks every target, and open question 3 (**no failover pin**) applies to the whole free-tier cohort, not just the radar. ⚠ Caveat recorded on the page: `executions.db`'s retention floor is **2026-11-06T05:09:30+08:00**, so the 2026-10-21 row is not re-readable today — the recurrence is an in-window read. > > **2026-11-09 (nightly audit — window 2026-11-08) — the day's fault was a hook, not a schedule, and it is fixed.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`), and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** — `errors.log` holds **12** lines on 11-07 and **46** on 11-08, plus **89** *"skipped after previous timeout or while still running"* suppression lines. Every tool call degraded to slow-then-suppressed, which is what stretched the day. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 → 120** in `~/.hermes/config.yaml` — verified live, and verified *effective* (the last timeout line is still 08:21:03, re-checked at audit time). ⚠ **The obvious suspect was measured and acquitted:** the plugin's caching is fine (`session_titles.py:21 CACHE_SECONDS = 30.0`; timed in-process at **3.1 s cold, 0.00 s warm**), so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong** — the trigger is **load** on the 2-core CT460 (`load average 5.89` at 13:27:50, `state.db` checkpointing a **40 MB WAL** on the same spindle) → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. **Cost of the fault:** *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and **12:00:51 → 18:00:42 (5 h 59 m 51 s)** — its two slowest slots on record, both delivered; *FreshRSS Midday Briefing* took **2 h 09 m 53 s**; and **the stocktake failed a third consecutive day** (`1a8e90ea2271`, 03:01:11 → 04:36:08, `failure_streak: 3`) on a **new terminal signature** — `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text.` where the previous two red days read *"lost its durable fire claim ownership"*. *Working hypothesis, not proven:* an open tool result is exactly what a suppressed `pre_tool_call` would produce, but the string does not name the hook. **11-08 settled: 169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:33); deliveries **8 / 8 `delivered`**; fleet **29 jobs (26 / 3)** — a **sixth** consecutive audit at the same count with nothing added, retired, disabled or repinned, though `updated_at` moved to `2026-11-09T00:06:50+08:00` as a run-state stamp. **Unclosed incident rows 2 → 3**, all three the stocktake's and all silent (`detected`, `alerted_at: null`) — but **the alerting path is not the fault**: the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` **08:06:54** in the same window. ✅ **Two verified resolutions:** the **Mnemosyne Health Monitor's no-agent conversion works** — its first script slot ran **2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, silent, writing a **159 B** output file against **105,982 B** for the last agent-mode run (coverage live at **working 1030/1030 = 100.0 %, episodic 1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet; the script uses `awk` because **`bc` is not installed on this host**); and the **1.89 GB `state.db`** was re-measured (**1,890,656,256 B**, **2,901 sessions / 197,682 messages**, live **40 MB** WAL) with the 2026-08-11 remnant **`state.db.corrupted.20260811_151052` (624.1 MB) deleted**. **New page:** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. > diff --git a/infrastructure/tailscale.md b/infrastructure/tailscale.md index 0774808..ced07b3 100644 --- a/infrastructure/tailscale.md +++ b/infrastructure/tailscale.md @@ -3,8 +3,8 @@ title: Tailscale type: infrastructure status: active created: 2026-07-26 -updated: 2026-11-10 -verified_on: 2026-11-10 +updated: 2026-11-12 +verified_on: 2026-11-12 confidence: high tags: [tailscale, vpn, networking, infrastructure] --- @@ -19,7 +19,8 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access - **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`) - **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`) -## Tailscale Node Inventory (verified 2026-11-10) +## Tailscale Node Inventory (verified 2026-11-12) +Re-verified 2026-11-12 03:1x–03:4x AWST (stocktake): table rebuilt from `tailscale status --json` — **three peers flipped offline in the peer record while all three are in fact reachable, one genuine offline age/stamp correction, and 10 offline ages drifted +2d. No IP changes, no missing table rows, no new peers.** ⚠ **`mailflow` (100.115.60.92, CT1262 m93p), `qwenpaw` (100.110.72.126, CT465 pve) and `springroll` (100.100.153.125) all read `Online: false` in the peer record** (`LastSeen` 2026-11-10 13:59Z / 2026-11-11 12:20Z / 2026-11-11 08:30Z), yet **`tailscale ping` returns pong on every one** — mailflow via a direct path (`192.168.178.142:41641`, 2 ms), qwenpaw direct (`192.168.178.186:41641`, 7 ms) and springroll via **DERP(sin)** (210 ms, relay rather than direct) — and live `pct list` shows **CT1262 and CT465 running**. All three rows therefore stay **online**; the peer record is stale, exactly the m93p pattern this page has already recorded three times. ⚠ **`omarchymac` was genuinely stale in the *other* direction:** the 2026-11-10 row read *"offline 2d — last seen 2026-11-08 00:45Z"*, while the live record reads `LastSeen 2026-11-07T06:41:48Z` = **offline 4d** — the peer was last seen *earlier* than the page claimed and `tailscale ping` → **timeout**, so it is a real offline, not a stale record. The **9 named offline ages** were refreshed (+2d each): cloudreve 108d → **110d**; conduit 65d → **66d**; elementsynapse 111d → **113d**; freshrss-new 84d → **86d**; octoagent 8d → **10d**; omarchybook 19d → **21d**; omarchymac 2d → **4d**; openmausbot 19d → **21d**; ubuntu-server-xfce 81d → **82d**; and the six `codex-environment-*` ephemeral peers 2d → **4d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (58 rows = 57 live IPv4 peers + `hermes` itself; `tailscale status --json` reports **80 peers total — 57 with a `100.` IPv4, 23 IPv6-only**, unchanged). Container states cross-checked against live `pct list` **and `qm list`** on **both** hypervisors: `pve` (100.96.100.82) **17 LXCs + 4 VMs** (VM100 homeassistant, VM101 guest-vps, VM110 omarchy, VM282 ubuntu-server-xfce), `m93p` (100.104.51.98) **20 LXCs + 2 VMs** (VM103 cachyos-m93p stopped, VM106 openmausbot-vm running) — counts unchanged. **One new guest not previously on this page: CT1007 `homebridge` on `m93p`, running but with `tailscaled` `inactive` inside it** (`pct exec 1007 -- systemctl is-active tailscaled` → `inactive`), so it is a running guest that does not join the tailnet — the same class as CT105 `conduit` and CT1264 `openmausbot`. **Both DEAD Serve targets re-confirmed dead:** `ss -ltn` still shows **no listener on 8769** (`/agent-estate`) and **none on 9123** (`/synergy-prep`). **CT201's Serve table re-probed** (`pct exec 201 -- tailscale serve status`) and is unchanged: `/` → `8085` and `/mgr` → `8084` on both `:443` and `:8443`. **Re-probed and confirmed this page's stale incident:** CT299 `sftpgo` is online and healthy → [[incidents/2026-09-02-sftpgo-outage]]. Re-verified 2026-11-10 00:4x AWST (nightly audit): `tailscale status --json` re-read — **6 new peers added, 8 offline ages drifted, 1 state transition back *online* (`octop`); no IP changes and no missing table rows.** ⚠ **Six peers that had never been listed now sit in the tailnet** — `codex-environment` (`100.111.126.78`), `codex-environment-1b88dc` (`100.76.42.76`), `codex-environment-8dadbb` (`100.65.200.27`), `codex-environment-955866` (`100.81.119.118`), `codex-environment-9fbd61` (`100.106.215.121`) and `codex-environment-d3edbd` (`100.74.3.2`) — every one created **2026-11-07 05:09–05:56Z**, every one **`Online: false`** with `LastSeen` **2026-11-07 11:09–11:56Z**. **None is a guest on either hypervisor** (live `pct list` and `qm list` on both `pve` and `m93p` list no such guest), so they are **orphaned/ephemeral tailnet peers**, not estate hosts: six records registered inside 47 minutes and all went silent within ~6 h, the signature of a throwaway CI/dev environment rather than infrastructure. **State transition:** `octop` (`100.92.163.122`) went **online → offline 1d** (`LastSeen 2026-11-08T13:13:04Z`) after returning for a single day on 11-07. The **9 offline ages** were refreshed (+1d each): cloudreve 105d → **106d**; conduit 61d → **62d**; elementsynapse 108d → **109d**; freshrss-new 81d → **82d**; octoagent 5d → **6d**; omarchybook 16d → **17d**; openmausbot 16d → **17d**; ubuntu-server-xfce 77d → **78d**; plus **`omarchymac`, which was not aged but re-stated** — the 2026-11-07 row read *"offline 5d — last seen 2026-11-01 11:13Z"* while the live record reads **`LastSeen 2026-11-08T00:45:17Z` = offline 2d**, so **both the age and the last-seen stamp on that row were stale** and are corrected here. Node count **52 → 58 non-funnel nodes** (57 live IPv4 peers + `hermes` itself). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none** once the six additions land; the **~23 IPv6-only `funnel-ingress-node` peers** remain deliberately excluded (`tailscale status --json`: **80 peers total — 57 with a `100.` IPv4, 23 IPv6-only**). Container counts re-confirmed live: `pve` (100.96.100.82) **17 LXCs + 4 VMs**, `m93p` (100.104.51.98) **20 LXCs + 2 VMs** — unchanged. **Not re-probed this pass** (so the 2026-11-07 findings stand un-contradicted rather than re-verified): the `:443` Serve route table (12 paths), the two DEAD CT460 targets (8769 `/agent-estate`, 9123 `/synergy-prep`), and per-guest tailnet identity via `pct exec -- tailscale ip -4`. Re-verified 2026-11-07 03:1x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 9 offline-age drifts, no IP changes and no missing peers.** **`octop` came back online**: the peer record reads `Online: true` with `LastSeen 0001-01-01` and `tailscale ping 100.92.163.122` → **`pong from octop via DERP(syd) in 98ms`**, so the row moves offline → online. It remains **not an LXC or VM on either hypervisor** (live `pct list` / `qm list` on both `pve` and `m93p` list no such guest) — an **orphaned tailnet peer**, so its return is not an estate change. The **9 offline ages** were refreshed (+1d each): cloudreve 104d → **105d**; conduit 60d → **61d**; elementsynapse 107d → **108d**; freshrss-new 80d → **81d**; octoagent 4d → **5d**; omarchybook 15d → **16d**; omarchymac 4d → **5d**; openmausbot 15d → **16d**; ubuntu-server-xfce 76d → **77d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) **17 guests** and `m93p` (100.104.51.98) **20 guests + 2 VMs** — ⚠ **the m93p guest count on this page was one short**: live `pct list` on `m93p` returns **20 LXCs** (the extra is **CT107 `mem0`**, `100.73.94.53`, `mem0-mcp.service` active — confirmed from inside the guest), not 19; CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT998 `nightbridge` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). The route table still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; **both DEAD targets re-confirmed dead** — `ss -ltn` still shows **no listener on 8769** (`/agent-estate`) and **none on 9123** (`/synergy-prep`) — while every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` answers **HTTP 200**. Re-verified 2026-11-06 03:0x–03:2x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 8 offline-age drifts, no IP changes and no missing peers.** **`anthonys-mac-mini` came back online**: the peer record reads `Online: true` and `tailscale ping 100.76.60.37` → `pong from anthonys-mac-mini via DERP(syd) in 103ms`, so the row moves offline → online (client device, not an estate host). The **8 offline ages** were refreshed (+1d each): cloudreve 103d → **104d**; elementsynapse 106d → **107d**; freshrss-new 79d → **80d**; octoagent 3d → **4d**; octop 1d → **2d**; omarchybook 14d → **15d**; omarchymac 3d → **4d**; openmausbot 14d → **15d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). `m93p` re-pinged → `pong from m93p via 192.168.178.250:41641 in 3ms` (online, peer record agrees), and `octop` re-pinged → **timeout** (genuinely offline, matching its `LastSeen`). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) 17 guests and `m93p` (100.104.51.98) 19 guests + 2 VMs — CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). ⚠ **Two CT460 Serve routes are dead but still present in `tailscale serve status`**: `/agent-estate` → `127.0.0.1:8769` and `/synergy-prep` → `127.0.0.1:9123` have **no listener** (live `ss -ltn` probe) and therefore serve 502 — both rows are now marked DEAD here, matching [[infrastructure/network-map]] which had already recorded them as dead on 2026-11-05. The route table itself still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` is reachable. @@ -46,55 +47,55 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | anthonys-mac-mini | 100.76.60.37 | online — **returned 2026-11-06** (peer record `Online: true`, `tailscale ping` → `pong from anthonys-mac-mini (100.76.60.37) via DERP(syd) in 103ms`; client device, not an estate host) | | clawtest | 100.122.189.81 | online — CT333 (pve) | | cliproxyapi | 100.83.112.94 | online — CT103 (pve) | -| cloudreve | 100.111.187.7 | offline 108d — CT209 stopped | -| codex-environment | 100.111.126.78 | offline 2d — last seen 2026-11-07 11:09Z; **created 2026-11-07 05:5xZ**, six-record burst; **not an LXC/VM on either hypervisor** (orphaned/ephemeral tailnet peer; added 2026-11-10, last-seen stamps corrected 2026-11-10) | -| codex-environment-1b88dc | 100.76.42.76 | offline 2d — last seen 2026-11-07 11:12Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) | -| codex-environment-8dadbb | 100.65.200.27 | offline 2d — last seen 2026-11-07 11:46Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) | -| codex-environment-955866 | 100.81.119.118 | offline 2d — last seen 2026-11-07 11:49Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) | -| codex-environment-9fbd61 | 100.106.215.121 | offline 2d — last seen 2026-11-07 11:51Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) | -| codex-environment-d3edbd | 100.74.3.2 | offline 2d — last seen 2026-11-07 11:56Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) | -| conduit | 100.81.79.87 | offline 65d — last seen 2026-09-06 01:32Z | +| cloudreve | 100.111.187.7 | offline 110d — CT209 stopped | +| codex-environment | 100.111.126.78 | offline 4d — last seen 2026-11-07 11:09Z; **created 2026-11-07 05:5xZ**, six-record burst; **not an LXC/VM on either hypervisor** (orphaned/ephemeral tailnet peer; added 2026-11-10, last-seen stamps corrected 2026-11-10) | +| codex-environment-1b88dc | 100.76.42.76 | offline 4d — last seen 2026-11-07 11:12Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) | +| codex-environment-8dadbb | 100.65.200.27 | offline 4d — last seen 2026-11-07 11:46Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) | +| codex-environment-955866 | 100.81.119.118 | offline 4d — last seen 2026-11-07 11:49Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) | +| codex-environment-9fbd61 | 100.106.215.121 | offline 4d — last seen 2026-11-07 11:51Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) | +| codex-environment-d3edbd | 100.74.3.2 | offline 4d — last seen 2026-11-07 11:56Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) | +| conduit | 100.81.79.87 | offline 66d — last seen 2026-09-06 01:32Z | | docker | 100.96.150.69 | online — CT200 (pve) | | docker-01 | 100.121.12.31 | online — CT1001 (m93p) | -| elementsynapse | 100.75.250.37 | offline 111d — CT555 stopped | +| elementsynapse | 100.75.250.37 | offline 113d — CT555 stopped | | freellmapi | 100.127.208.75 | online — CT1006 (m93p) | | freshrss | 100.83.60.93 | online — CT1260 (m93p) | -| freshrss-new | 100.110.180.115 | offline 84d — last seen 2026-08-17 17:04Z | +| freshrss-new | 100.110.180.115 | offline 86d — last seen 2026-08-17 17:04Z | | gitea | 100.82.100.128 | online — CT221 (m93p) | -| guest-vps | 100.117.14.84 | online | +| guest-vps | 100.117.14.84 | online — VM101 (pve) | | hermesagent | 100.98.205.112 | online — CT450 (pve) | | homeassistant | 100.68.139.5 | online — VM100 (pve) | | litellm | 100.79.29.64 | online — CT500 (m93p) | -| m93p | 100.104.51.98 | online — hypervisor (peer record reads `Online: true` again on 2026-11-04 — the stale-record pattern of 10-29 → 11-01 has cleared; `tailscale ping` → pong) | -| mailflow | 100.115.60.92 | online — CT1262 (m93p) | +| m93p | 100.104.51.98 | online — hypervisor (peer record reads `Online: true`; `tailscale ping` → pong) | +| mailflow | 100.115.60.92 | online — CT1262 (m93p) — ⚠ peer record reads `Online: false` (`LastSeen 2026-11-10T13:59:56Z`, offline 1d), but `tailscale ping` → **`pong from mailflow (100.115.60.92) via 192.168.178.142:41641 in 2ms`** and live `pct list` on `m93p` shows **CT1262 running** — stale peer record, not an outage (re-probed 2026-11-12) | | mattermost | 100.98.116.14 | online — CT255 (pve) | | mem0 | 100.73.94.53 | online — CT107 (m93p) | -| n8n | 100.84.124.30 | online | +| n8n | 100.84.124.30 | online — CT215 (pve) | | nextcloud | 100.90.102.19 | online — CT270 (pve) | | ntfy | 100.67.153.116 | online — CT1241 (m93p) | | obsidian-livesync | 100.116.244.84 | online — CT100 (m93p) | -| octoagent | 100.117.10.68 | offline 8d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) | -| octop | 100.92.163.122 | **online** (re-verified 2026-11-10 00:4x; it had been dark since `LastSeen 2026-11-08T13:13:04Z` — online 11-07, offline 11-08, online again 11-10; still **not an LXC/VM on either hypervisor** — orphaned tailnet peer) | +| octoagent | 100.117.10.68 | offline 10d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) | +| octop | 100.92.163.122 | **online** (re-verified 2026-11-12 00:4x AWST — peer record `Online: true`; still **not an LXC/VM on either hypervisor** — orphaned tailnet peer) | | ollama | 100.108.44.119 | online — CT205 (pve) | -| omarchy | 100.73.227.103 | online | -| omarchybook | 100.107.9.100 | offline 19d — last seen 2026-10-21 15:55Z | -| omarchymac | 100.126.173.115 | offline 2d — last seen 2026-11-08 00:45Z (⚠ **was stale at 5d / 2026-11-01**; corrected 2026-11-10) | +| omarchy | 100.73.227.103 | online — VM110 (pve) | +| omarchybook | 100.107.9.100 | offline 21d — last seen 2026-10-21 15:55Z | +| omarchymac | 100.126.173.115 | offline 4d — last seen 2026-11-07 06:41Z (⚠ **was stale at 2d / 2026-11-08 00:45Z**; corrected 2026-11-12 — the peer was last seen *earlier* than the 2026-11-10 row claimed and `tailscale ping` → timeout, so this is a genuine offline, not a stale record) | | omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute | | omniroute-1 | 100.78.63.20 | online — CT101 (m93p) | -| openmausbot | 100.91.214.91 | offline 19d — CT1264 (m93p) stopped; last seen 2026-10-21 15:50Z | +| openmausbot | 100.91.214.91 | offline 21d — CT1264 (m93p) stopped; last seen 2026-10-21 15:50Z | | openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) | | openweb | 100.69.230.13 | online — CT245 (m93p) | | paseo | 100.72.188.93 | online — CT1261 (m93p) | | pve | 100.96.100.82 | online | -| qwenpaw | 100.110.72.126 | online — CT465 (pve) | +| qwenpaw | 100.110.72.126 | online — CT465 (pve) — ⚠ peer record reads `Online: false` (`LastSeen 2026-11-11T12:20:00Z`), but `tailscale ping` → **`pong from qwenpaw (100.110.72.126) via 192.168.178.186:41641 in 7ms`** and live `pct list` on `pve` shows **CT465 running** — stale peer record, not an outage (re-probed 2026-11-12) | | rclone | 100.107.116.17 | online — CT104 (m93p) | | runtipi | 100.96.244.39 | online — CT201 (pve) | | rustdesk | 100.95.227.57 | online — CT1008 (m93p) | -| sftpgo | 100.88.223.70 | online — CT299 (pve) — **recovered 2026-11-04 (freeze 1) and again 2026-11-05 (freeze 2, manual `ssd0` run)**: a wedged `vzdump` had frozen the guest for ~24 h; killing the task unfroze it (see [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]) | +| sftpgo | 100.88.223.70 | online — CT299 (pve) — **recovered 2026-11-04 (freeze 1) and again 2026-11-05 (freeze 2, manual `ssd0` run)**: a wedged `vzdump` had frozen the guest for ~24 h; killing the task unfroze it (see [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]) — **re-confirmed online 2026-11-12** (`tailscale ping` → `pong from sftpgo (100.88.223.70) via 192.168.178.92:41641 in 0s`, `pct list` → CT299 running, healthz → 401) | | social-dashboard | 100.70.183.37 | online — CT1263 (m93p) | -| springroll | 100.100.153.125 | online | +| springroll | 100.100.153.125 | online — ⚠ peer record reads `Online: false` (`LastSeen 2026-11-11T08:30:00Z`), but `tailscale ping` → **`pong from springroll (100.100.153.125) via DERP(sin) in 210ms`** (answers via relay, not direct); not an LXC/VM on either hypervisor (client device) — re-probed 2026-11-12 | | springroll-browser | 100.109.221.91 | online | -| ubuntu-server-xfce | 100.101.26.104 | offline 81d — last seen 2026-08-20 19:36Z | +| ubuntu-server-xfce | 100.101.26.104 | offline 82d — last seen 2026-08-20 19:36Z — VM282 (pve) running | | vaultwarden | 100.123.225.90 | online — CT102 (m93p) | | webserver | 100.82.40.23 | online — CT207 (pve) | @@ -114,8 +115,8 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | `/gitea` | `http://100.82.100.128:3000` | Gitea | | `/desktop` | `http://127.0.0.1:6081` | CUA noVNC | | `/vnc-camofox` | `http://127.0.0.1:6080` | Camofox VNC | -| `/synergy-prep` | `http://127.0.0.1:9123` | Synergy prep — ⚠ **DEAD: nothing listening on 9123** (live `ss -ltn` probe 2026-11-06, **re-confirmed 2026-11-07**; the route still exists in `tailscale serve status` but serves 502) | -| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) — ⚠ **DEAD: nothing listening on 8769** (live `ss -ltn` probe 2026-11-06, **re-confirmed 2026-11-07**; route still present in `tailscale serve status` but serves 502) | +| `/synergy-prep` | `http://127.0.0.1:9123` | Synergy prep — ⚠ **DEAD: nothing listening on 9123** (live `ss -ltn` probe 2026-11-06, **re-confirmed 2026-11-07 and 2026-11-12**; the route still exists in `tailscale serve status` but serves 502) | +| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) — ⚠ **DEAD: nothing listening on 8769** (live `ss -ltn` probe 2026-11-06, **re-confirmed 2026-11-07 and 2026-11-12**; route still present in `tailscale serve status` but serves 502) | | `/evening-grid.html` | `http://127.0.0.1:9099/evening-grid.html` | Evening grid | Re-verified against `tailscale serve status` on **2026-10-26** — the CT460 route table above matched live exactly (**12 paths on :443**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; no drift). diff --git a/log.md b/log.md index 688ff20..06d7299 100644 --- a/log.md +++ b/log.md @@ -2480,3 +2480,18 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted* - **Post-audit housekeeping (00:2x AWST 2026-11-12), vault sync.** Touched files copied to **both** vault conventions — `…/agent-estate-wiki/` and `…/agent-estate-wiki/wiki/` — with `sha256sum` confirmed equal across all three copies; vault committed locally in `/home/hermes/wiki` (`master`, **no remote — never pushed**), explicit paths only, so the unrelated `reports/uber-eats/*` dirt stayed out. Before copying, each vault file was compared against the **pre-edit** authoring blob (`c108eb1`): every difference was a *stale-but-clean* older version (vault `index.md`/`skills-index.md` still 11-10/11-09, `log.md` still missing the 11-11 stocktake entry, the nous incident page still 2026-10-26 at 48 lines vs 96) — **no direct-to-vault edits to preserve**, so forward-sync was safe. The 2026-10-22 nous incident page was **created** in vault B's `incidents/` (it had never been mirrored); the new 11-10 audit page went into **both** `incidents/` dirs. - **Vault drift re-measured (hash sweep over all 133 authoring pages, this run's 10 touched files excluded):** **Vault A — 13 stale / 56 missing / 64 in sync. Vault B — 58 stale / 8 missing / 67 in sync.** Unchanged in character from the 2026-10-19 measurement (the copy bridge still only syncs what an audit touches), so the missing-page backlog is recorded here and **left to an explicit request** rather than silently swept — a 133-page mirror pass is bigger than a nightly audit's mandate. ⚠ This is now a two-night-old gap: because the 2026-11-10 audit died at 00:12, **nothing was mirrored for that window either.** - **Freshness report:** `raw/stocktake/freshness-20261112.txt` (this run, 6,325 B) — **1 stale / 129 fresh / 3 no-date of 133** pages (was 132; **+1 = the new incident page**). The single stale entry is **`log.md` itself**, which carries no frontmatter `updated:` — expected, not a finding. `systems/skills-index.md` regenerates identically at **299 skills / 67 categories**. Quartz rebuilt and `hermes-wiki-serve.service` restarted; the new page returns **200** at the served site with its body content present. + +## 2026-11-12 03:0x–04:1x AWST — Stocktake (03:00): tailscale drift corrected; 1 stale + 5 threshold pages re-verified live + +- **Freshness:** `wiki-maintain.sh` → **1 stale / 129 fresh / 3 no-date of 133** (was 132, +1 = the new 2026-11-10 incident page). The single stale page was **[[incidents/2026-09-02-sftpgo-outage]] at 22d** — re-verified live and cleared. The 3 no-date pages are unchanged (`SCHEMA.md`, `experiments/active/example.md`, `log.md`). `systems/skills-index.md` regenerates identically at **299 skills / 67 categories**. +- **⚠ Live-vs-page diff found real drift in [[infrastructure/tailscale.md]] (2d old — age scan would have missed it):** three peers read `Online: false` in the peer record while **all three ping fine** — `mailflow` (CT1262 m93p, pong `192.168.178.142:41641` 2 ms), `qwenpaw` (CT465 pve, pong `192.168.178.186:41641` 7 ms) and `springroll` (pong via **DERP(sin)** 210 ms, relay not direct) — with both LXCs `running` in `pct list`. All three rows stay **online**; the peer record is stale (the m93p pattern again). One **genuine** correction in the other direction: `omarchymac`'s row claimed `offline 2d / last seen 2026-11-08 00:45Z` when the live record reads `LastSeen 2026-11-07T06:41:48Z` = **offline 4d** and ping → **timeout**. 9 named offline ages + the six `codex-environment-*` ephemerals refreshed (+2d). No IP changes, no missing rows, no new peers (58 rows = 57 live IPv4 peers + `hermes`; 80 peers total / 57 IPv4 / 23 IPv6-only, unchanged). Table rebuilt by script, not by hand. +- **Re-probed and re-confirmed:** CT299 `sftpgo` **running**, peer `Online: true`, `pong via 192.168.178.92:41641 in 0s`, healthz **401** (auth-gated, service up) — closing [[incidents/2026-09-02-sftpgo-outage]] as still-resolved, and noting that `pct config 299` now reads **`swap: 512`** (the 43 Gi over-provisioning in that root cause is gone). Both DEAD Serve targets (`8769` `/agent-estate`, `9123` `/synergy-prep`) still have **no listener** — re-confirmed a fifth time. CT201's Serve table re-probed unchanged (`/`→8085, `/mgr`→8084 on both `:443` and `:8443`). Container counts re-confirmed on **both** hypervisors incl. `qm list`: `pve` **17 LXCs + 4 VMs**, `m93p` **20 LXCs + 2 VMs**. **New guest found: CT1007 `homebridge` on m93p — running but `tailscaled` `inactive`**, so it runs without joining the tailnet (same class as CT105 conduit, CT1264 openmausbot). +- **Incident page re-opened for a **recurrence** — [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] (21d, tomorrow's stale list).** The *"all targets were skipped by pre-dispatch filters"* signature has now fired on a **second job and a second pin**: `cbfe434a3b5e` (Wiki Audit) failed on **2026-11-10** with `RuntimeError: HTTP 429: … pre-dispatch filters … (credits 100%)`, `failure_type: rate_limit`, and its incident row `cbfe43_ed01845310d8` closed `resolved` **2026-11-12T00:25**. The original subject has **not** repeated — `dca8482e4f76` ran clean on 11-06/09/10/11. Caveat recorded on the page: `executions.db`'s retention floor is **2026-11-06T05:09:30+08:00**, so the 2026-10-21 row is *not* re-readable from the DB — the new read is in-window only. +- **Four threshold/near-threshold pages re-verified against live state (skill pattern 22), each with a dated note — no date bump without a probe:** + - [[incidents/2026-07-29-mnemosyne-relay]] (19d) — `tools: None` absent from `config.yaml`; **0** occurrences of `Unknown Mnemosyne tool` across 4 rotated gateway logs and **0** of the relay `scope handle` error; `mnemosyne stats` → 2782 memories and recall returns ranked results. + - [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] (18d) — `himalaya envelope list -a work -s 1` still fails on `login not supported`; no OAuth2 token in the himalaya config, so path 1 (Anthony's sign-in) remains the only route. + - [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]] (19d) — keyed probes on `omniroute-101`: **`good-free` → HTTP 200 (`glm-5.2-cheap:free`)** and **`free-stack` → HTTP 200 (`stepfun-ai/step-5-preview`)**. The mitigation holds and neither combo now resolves to deepseek. **The three free-DeepSeek upstreams themselves were NOT probed**, so "upstream still degraded" is neither confirmed nor refuted and restoring deepseek-first ordering is still unjustified — recorded as such on the page. + - [[reports/communities-wa-social-media-audit-2026]] (18d) — **not re-verified this pass** (live Department of Communities pages not re-fetched); its `review_after: 2026-11-26` still stands and its own note already scopes the unverified claims. Flagged, left as written. +- **Order of operations:** script → page corrections → **re-run script** → log/index → commit. `wiki-maintain.sh` re-run after every page edit; Quartz rebuilt and `hermes-wiki-serve.service` restarted. +- **No page deleted, no decision reversed, no secrets written.** No credential value appears in any page; the OmniRoute probe used the `key_env` name, never the value. +- **Correction to the entry above (2026-11-12 00:1x AWST audit), verified live rather than inherited:** that entry states *"The single stale entry is **`log.md` itself**, which carries no frontmatter `updated:` — expected, not a finding."* **That is wrong.** The committed `raw/stocktake/freshness-20261112.txt` from that run (sha256 `da5f9e2e…`, byte-identical to this run's pre-fix output) reads **`- [ 22d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-10-21)`** as its single `## STALE (1)` line — the real page, which this stocktake has now re-verified and cleared. `log.md` sits in the `## NO updated DATE` bucket, as always (pattern 13). The stale page was therefore **left unflagged in prose for one cycle while sitting stale in the tree** — the exact "fresh-looking prose, stale content" failure the report exists to prevent. diff --git a/raw/stocktake/freshness-20261112.txt b/raw/stocktake/freshness-20261112.txt index 8476a5c..cf0a22a 100644 --- a/raw/stocktake/freshness-20261112.txt +++ b/raw/stocktake/freshness-20261112.txt @@ -1,11 +1,10 @@ # Wiki freshness audit — 2026-11-12 Threshold: >21 days since `updated` -## STALE (1) +## STALE (0) -- [ 22d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-10-21) -## FRESH (129) +## FRESH (130) - [ 1d] CONTRIBUTING.md - [ 4d] agents/hermes-chaos.md @@ -31,14 +30,15 @@ Threshold: >21 days since `updated` - [ 1d] decisions/persistent-shell-enabled.md - [ 14d] decisions/rollback-steel-to-camofox.md - [ 1d] experiments/index.md -- [ 19d] incidents/2026-07-29-mnemosyne-relay.md +- [ 0d] incidents/2026-07-29-mnemosyne-relay.md - [ 1d] incidents/2026-08-01-mnemosyne-capture-failure.md -- [ 19d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- [ 0d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md - [ 5d] incidents/2026-08-08-gitea-config-backup-push-dead.md - [ 7d] incidents/2026-08-15-skills-index-drift.md - [ 8d] incidents/2026-08-18-mnemosyne-capture-pipeline.md -- [ 18d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 0d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md - [ 1d] incidents/2026-08-29-gmail-read-state.md +- [ 0d] incidents/2026-09-02-sftpgo-outage.md - [ 8d] incidents/2026-09-28-hermes-managed-runtime-deps.md - [ 14d] incidents/2026-10-01-cliproxy-connection-auth-expired.md - [ 17d] incidents/2026-10-03-ct450-hermes-update.md @@ -50,7 +50,7 @@ Threshold: >21 days since `updated` - [ 16d] incidents/2026-10-16-omniroute-503-resource-pressure.md - [ 2d] incidents/2026-10-18-degenerate-stream-canned-substitute.md - [ 16d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md -- [ 21d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md +- [ 0d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md - [ 18d] incidents/2026-10-22-conduit-memory-start-timeout.md - [ 1d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md - [ 17d] incidents/2026-10-24-conduit-call-request-refused.md @@ -79,7 +79,7 @@ Threshold: >21 days since `updated` - [ 1d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md - [ 1d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md - [ 1d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md -- [ 1d] index.md +- [ 0d] index.md - [ 5d] infrastructure/backups.md - [ 1d] infrastructure/credentials-map.md - [ 5d] infrastructure/docker-services.md @@ -87,7 +87,7 @@ Threshold: >21 days since `updated` - [ 5d] infrastructure/hosts.md - [ 5d] infrastructure/network-map.md - [ 5d] infrastructure/proxmox.md -- [ 2d] infrastructure/tailscale.md +- [ 0d] infrastructure/tailscale.md - [ 8d] queries/retained-research-answers.md - [ 18d] reports/communities-wa-social-media-audit-2026.md - [ 11d] reports/uber-eats/2026-07.md @@ -143,4 +143,4 @@ Threshold: >21 days since `updated` - experiments/active/example.md - log.md -TOTAL: 1 stale, 129 fresh, 3 no-date of 133 pages \ No newline at end of file +TOTAL: 0 stale, 130 fresh, 3 no-date of 133 pages \ No newline at end of file