Stocktake 2026-11-12: tailscale live-vs-page drift corrected; sftpgo incident re-verified; pre-dispatch-filter signature found recurring on a second job/pin

- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json.
  mailflow/qwenpaw/springroll read Online:false in the peer record but all three ping
  (CT1262 + CT465 running; springroll via DERP(sin)) - stale peer records, rows stay online.
  omarchymac genuinely stale in the other direction (offline 2d/11-08 -> offline 4d, ping timeout).
  9 named offline ages + 6 codex ephemeral peers refreshed. New guest CT1007 homebridge
  (running, tailscaled inactive). DEAD Serve targets 8769/9123 re-confirmed dead; CT201 table
  unchanged; container counts 17 LXC+4 VM (pve) / 20 LXC+2 VM (m93p) confirmed.
- incidents/2026-09-02-sftpgo-outage.md: 22d stale, re-probed live (pct list running, peer
  Online:true, pong, healthz 401; swap now 512 not 43Gi). Cleared.
- incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md: signature recurred 2026-11-10 on
  cbfe434a3b5e (custom:omni-101/free-stack, HTTP 429 credits 100%); incident row resolved
  2026-11-12T00:25. Original subject clean 4 slots. Retention-floor caveat recorded.
- Threshold re-verifications (19d/18d): mnemosyne-relay (tools:None gone, 0 log hits),
  work-outlook (himalaya still login not supported), omniroute-free-deepseek
  (good-free/free-stack both HTTP 200, non-deepseek resolution).
- Correction recorded: previous entry mis-described the 2026-11-12 stale page as log.md.
- wiki-maintain.sh re-run after all edits: freshness 0 stale / 130 fresh / 3 no-date of 133.
This commit is contained in:
Hermes
2026-11-12 03:19:12 +08:00
parent b94de554f2
commit ab3652aa20
9 changed files with 158 additions and 57 deletions
+20 -4
View File
@@ -3,9 +3,8 @@ title: 'Mnemosyne Relay Scope Corruption + `tools: None` — 2026-07-29'
type: incident
status: resolved
created: 2026-07-29
updated: 2026-10-24
verified_on: 2026-10-24
review_after: 2026-11-07
updated: 2026-11-12
verified_on: 2026-11-12
confidence: high
tags: [mnemosyne, config, relay, incident, resolved]
sources: [live-probe, config, session-20260729]
@@ -51,4 +50,21 @@ Scope push/pop ordering mismatch. Something pushes a scope that isn't at the top
Filed (or should be filed) under NousResearch/hermes-agent issues. No token available to auto-create.
## MONITORING
Watch for recurrence. If persists, check whether the affected sessions share a common tool call pattern (subagent delegation, tool failure, etc.).
Watch for recurrence. If persists, check whether the affected sessions share a common tool call pattern (subagent delegation, tool failure, etc.).
## Re-verified 2026-11-12 03:3x AWST (stocktake — page was 19d old, inside the staleness window)
Both halves of this page re-checked against live state and **the fix holds; neither symptom has recurred**
in ~3.5 months and 106 days of logs:
- `grep -n "tools:" ~/.hermes/config.yaml` → the `memory.mnemosyne` block no longer carries `tools: None`
(the literal `None` key is absent), so the `Unknown Mnemosyne tool(s)… None` schema warning has no
trigger left.
- `grep -c "Unknown Mnemosyne tool" ~/.hermes/logs/gateway.log` and `.log.1`–`.log.3` → **0 in every
rotated file** (4 generations, 0 occurrences).
- `grep -ah "scope handle is not at the top" ~/.hermes/logs/gateway.log*` → **no occurrences**; the
relay-scope corruption has not repeated since 2026-07-29 08:10 UTC.
- Mnemosyne itself is healthy on the current build: `mnemosyne stats` → **2782 memories (1062 working /
1217 episodic)** and `mnemosyne recall "tailscale serve routes"` returns substantive ranked results, so
the relay/memory path is functioning.
The `REPORTED` action above still stands (no token to auto-create a NousResearch issue); the **MONITORING**
condition is now satisfied — no recurrence across four log generations.
@@ -3,9 +3,8 @@ title: OmniRoute free-DeepSeek Silent Stops (Empty-After-Tool) — 2026-08-04
type: incident
status: monitoring
created: 2026-08-04
updated: 2026-10-24
verified_on: 2026-10-24
review_after: 2026-11-07
updated: 2026-11-12
verified_on: 2026-11-12
confidence: high
tags: [omniroute, deepseek, free-tier, silent-failure, incident, mitigated]
sources: [live-probe, omniroute-log, session-20260804]
@@ -53,5 +52,23 @@ Backup of both pre-change rows: `/tmp/combo_backup_20260804.json` on the Hermes
- Consider `auto`+explorationRate for self-tuning if free tier stays flaky.
- Hermes-side: empty-post-tool turns are nudge/retry/fallback handled, but still end silently once exhausted — possible future improvement: surface "(empty after tool calls)" more visibly.
## Re-verified 2026-11-12 03:4x AWST (stocktake — page was 19d old, inside the staleness window)
Both reworked combos probed live on **`omniroute-1`** (`100.78.63.20:20128`, `custom:omni-101`) with the
correct `key_env` (`HERMES_CUSTOM_100_78_63_20_20128_API_KEY`) and the exact aliases used:
| Probe | Result |
|---|---|
| `good-free` (the 12-model rework) | **HTTP 200**, resolving to **`glm-5.2-cheap:free`** with a valid non-empty completion |
| `free-stack` | **HTTP 200**, resolving to **`stepfun-ai/step-5-preview`** with a valid completion |
**The mitigation is holding, and the 2026-08-04 follow-up can be partially closed:** both combos answer with
real content, and **neither resolves to a deepseek model** — which is exactly the intent of dropping the
proven culprits from `good-free`. ⚠ **What is *not* re-verified:** I did not individually probe the
free-DeepSeek upstreams (`238d5978/deepseek-v4-flash`, `nvidia/deepseek-ai/deepseek-v4-pro`,
`deepseek-v4-flash-thinking:free`), so the original claim *"upstream free-DeepSeek tier still degraded"*
stands **neither confirmed nor refuted** — the combos serve healthy non-deepseek free models, which may mean
the upstream recovered *or* simply that it is no longer reached. **Restoring deepseek-first ordering is
therefore still not justified** and should wait on a direct probe of those three upstreams.
## Refs
- [[systems/omniroute]]
@@ -3,8 +3,8 @@ title: Work Outlook Mailbox — Basic Auth Disabled (Himalaya)
type: incident
status: open
created: 2026-10-03
updated: 2026-10-25
verified_on: 2026-10-25
updated: 2026-11-12
verified_on: 2026-11-12
confidence: high
tags: [email, himalaya, office365, oauth2, auth, incident]
sources: [live-himalaya-debug, live-tcp-probe, cron-session-20260826_180700_a94098]
@@ -12,7 +12,16 @@ sources: [live-himalaya-debug, live-tcp-probe, cron-session-20260826_180700_a940
# Work Outlook Mailbox — Basic Auth Disabled (Himalaya)
**Window:** first detected 2026-08-26; re-verified 2026-10-03 and again 2026-10-25 (still open)
**Re-verified 2026-11-12 03:3x AWST (stocktake — page was 18d old, inside the staleness window):**
the failure is unchanged and still reproduces exactly, with the **same error class**. `himalaya envelope
list -a work -s 1` → `Error: cannot build IMAP client / 1: login not supported` — i.e. the server rejects
the basic-auth `LOGIN` command itself rather than the credential, which is consistent with the SMTP/IMAP
basic-auth shutdown recorded above rather than a changed password. **No remediation has landed since
2026-10-25:** the account still authenticates by password (there is no OAuth2 token in the himalaya
config), so path 1 remains the only viable route and still requires Anthony's sign-in. The capability gap
described below is unchanged — no cron job polls this mailbox.
**Window:** first detected 2026-08-26; re-verified 2026-10-03, 2026-10-25 and 2026-11-12 (still open)
**Status:** Open — blocked pending an OAuth2 path
**Impact:** The agent cannot read or send from the `work` mailbox. Every work-email
request is unanswerable until this is fixed.
+16 -2
View File
@@ -3,8 +3,8 @@ title: SFTPGo Outage — CT299 Offline (2026-09-02)
type: incident
status: resolved
created: 2026-09-04
updated: 2026-10-21
verified_on: 2026-10-21
updated: 2026-11-12
verified_on: 2026-11-12
confidence: high
tags: [incident, sftpgo, ct299, tailscale, proxmox]
sources: [session:20260902_014540_c193be]
@@ -68,6 +68,20 @@ the node is `active` in `tailscale status` at `100.88.223.70`; the SFTPGo health
`http://100.88.223.70:10080/healthz` returns **HTTP 401** (expected — auth required, service up).
The 2026-09-02 outage remains resolved; no recurrence.
## Re-verified 2026-11-12 03:2x AWST (stocktake — page was 22d stale, the wiki's only stale page)
Live state re-probed and **the residual claim still holds — no recurrence in 70 days.** All three sources
checked independently:
- `pct list` on `pve` (100.96.100.82) → **CT299 `sftpgo` `running`**; `pct config 299` confirms the guest
(2 cores, 2048 MB RAM, **`swap: 512`** — the over-provisioned 43 Gi swap recorded in the root cause is
gone, so that concern has since been addressed).
- `tailscale status` peer record for `sftpgo` → **`Online: true`**, `100.88.223.70`, and
`tailscale ping sftpgo` → **`pong from sftpgo (100.88.223.70) via 192.168.178.92:41641 in 0s`**.
- SFTPGo health endpoint `http://100.88.223.70:10080/healthz` → **HTTP 401**, unchanged (auth required, service up).
Note that this outage and the two later CT299 freezes are **separate faults**: the 2026-09-02 cause was
`tailscaled` death plus stale `lxc-attach` cgroup pinning, whereas 2026-11-04/05 was a wedged `vzdump`
freezing the guest → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]] (also resolved).
## Lessons Learned
1. **Clean up `lxc-attach` probes** — always exit cleanly and verify no lingering processes after diagnostic sessions
@@ -3,9 +3,9 @@ title: Daily Marketing Job Radar — "All Targets Skipped by Pre-Dispatch Filter
type: incident
status: unresolved
created: 2026-10-22
updated: 2026-10-22
verified_on: 2026-10-22
review_after: 2026-10-29
updated: 2026-11-12
verified_on: 2026-11-12
review_after: 2026-11-26
confidence: high
tags: [cron, router, provider, job-search, failure-mode]
sources: [live-jobs-json, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-cron-output-file]
@@ -62,6 +62,31 @@ One weekday radar slot produced nothing: no new roles discovered, verified, de-d
job DB, and no candidate summary delivered to topic `:1030`. The job recovers on its next weekday slot
(2026-10-22 07:00) if the guard does not repeat.
## RECURRENCE — 2026-11-10, on a different job and a different pin (verified live 2026-11-12)
The signature is **not unique to the Job Radar**, and it has now recurred on a second job — which changes
the "one job's guard" hypothesis of the original open questions into a fleet-level failure mode.
| Probe | Result |
|---|---|
| `executions.db` (retention floor **2026-11-06T05:09:30+08:00**, so this is an in-window read and **not** an all-time sweep) | exactly **one** row carries the string: **`cbfe434a3b5e` (Wiki Audit — Nightly), 2026-11-10, `status: failed`**, error `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters…` |
| the original 2026-10-21 Job Radar row | **outside the retention floor**, so it cannot be re-read from the DB today; its evidence above stands on the 2026-10-22 capture. The per-job answer is unchanged in `jobs.json`: `dca8482e4f76` reads `ok` today |
| `cron_incidents` row `cbfe43_ed01845310d8` | `failure_type: rate_limit`, `first_seen_at 2026-11-11T00:12`, **`closed_at: 2026-11-12T00:25` → `resolved`** (it was still `detected`/open at the 2026-11-11 audit) |
| `dca8482e4f76` (Job Radar, the original subject) | **4 consecutive clean slots** — 2026-11-06, 11-09, 11-10, 11-11 all `completed`. The radar itself has **not** repeated the fault |
| the recurring job's pin | `cbfe434a3b5e` runs `custom:omni-101` / **`free-stack`** — **not** the `smart-route` pin the original failure used |
**What this adds to the original diagnosis:** the string fires on at least **two different models** and
**two different jobs**, so open question 1 ("if it is a local filter chain, one filter is probably
mis-configured for this job's toolset") now looks **less likely** than the router-side-guard reading — a
guard that blanks *every* target. The **prefix differs** between the two occurrences (`HTTP 429` in 2026-11,
bare `Service temporarily unavailable` in 2026-10), which is the same "error class changes, verdict holds"
pattern already recorded for the 401/403 cliproxy case → [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
The 2026-11 instance was **credits-driven** (`credits 100%`, guard reset after 6 h 24 m) — a quota
exhaustion, not a config fault, which also matches the fleet's paid-pin posture in
[[systems/scheduled-tasks]] (this stocktake itself runs `deepseek-paid-smart`, the only paid pin in the fleet).
Open question 3 (**give the radar a second provider pin — it has no failover**) therefore now applies to the
whole free-tier cohort, not just the radar, and remains unapplied pending Anthony's go-ahead.
## Open questions / next actions (proposed, not applied — needs Anthony's go-ahead)
1. **Find which component emits this string.** If it is a router-side guard with a fixed reset window, a