Stocktake 2026-11-12: tailscale live-vs-page drift corrected; sftpgo incident re-verified; pre-dispatch-filter signature found recurring on a second job/pin
- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json. mailflow/qwenpaw/springroll read Online:false in the peer record but all three ping (CT1262 + CT465 running; springroll via DERP(sin)) - stale peer records, rows stay online. omarchymac genuinely stale in the other direction (offline 2d/11-08 -> offline 4d, ping timeout). 9 named offline ages + 6 codex ephemeral peers refreshed. New guest CT1007 homebridge (running, tailscaled inactive). DEAD Serve targets 8769/9123 re-confirmed dead; CT201 table unchanged; container counts 17 LXC+4 VM (pve) / 20 LXC+2 VM (m93p) confirmed. - incidents/2026-09-02-sftpgo-outage.md: 22d stale, re-probed live (pct list running, peer Online:true, pong, healthz 401; swap now 512 not 43Gi). Cleared. - incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md: signature recurred 2026-11-10 on cbfe434a3b5e (custom:omni-101/free-stack, HTTP 429 credits 100%); incident row resolved 2026-11-12T00:25. Original subject clean 4 slots. Retention-floor caveat recorded. - Threshold re-verifications (19d/18d): mnemosyne-relay (tools:None gone, 0 log hits), work-outlook (himalaya still login not supported), omniroute-free-deepseek (good-free/free-stack both HTTP 200, non-deepseek resolution). - Correction recorded: previous entry mis-described the 2026-11-12 stale page as log.md. - wiki-maintain.sh re-run after all edits: freshness 0 stale / 130 fresh / 3 no-date of 133.
This commit is contained in:
@@ -3,9 +3,8 @@ title: 'Mnemosyne Relay Scope Corruption + `tools: None` — 2026-07-29'
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-07-29
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-11-07
|
||||
updated: 2026-11-12
|
||||
verified_on: 2026-11-12
|
||||
confidence: high
|
||||
tags: [mnemosyne, config, relay, incident, resolved]
|
||||
sources: [live-probe, config, session-20260729]
|
||||
@@ -51,4 +50,21 @@ Scope push/pop ordering mismatch. Something pushes a scope that isn't at the top
|
||||
Filed (or should be filed) under NousResearch/hermes-agent issues. No token available to auto-create.
|
||||
|
||||
## MONITORING
|
||||
Watch for recurrence. If persists, check whether the affected sessions share a common tool call pattern (subagent delegation, tool failure, etc.).
|
||||
Watch for recurrence. If persists, check whether the affected sessions share a common tool call pattern (subagent delegation, tool failure, etc.).
|
||||
|
||||
## Re-verified 2026-11-12 03:3x AWST (stocktake — page was 19d old, inside the staleness window)
|
||||
Both halves of this page re-checked against live state and **the fix holds; neither symptom has recurred**
|
||||
in ~3.5 months and 106 days of logs:
|
||||
- `grep -n "tools:" ~/.hermes/config.yaml` → the `memory.mnemosyne` block no longer carries `tools: None`
|
||||
(the literal `None` key is absent), so the `Unknown Mnemosyne tool(s)… None` schema warning has no
|
||||
trigger left.
|
||||
- `grep -c "Unknown Mnemosyne tool" ~/.hermes/logs/gateway.log` and `.log.1`–`.log.3` → **0 in every
|
||||
rotated file** (4 generations, 0 occurrences).
|
||||
- `grep -ah "scope handle is not at the top" ~/.hermes/logs/gateway.log*` → **no occurrences**; the
|
||||
relay-scope corruption has not repeated since 2026-07-29 08:10 UTC.
|
||||
- Mnemosyne itself is healthy on the current build: `mnemosyne stats` → **2782 memories (1062 working /
|
||||
1217 episodic)** and `mnemosyne recall "tailscale serve routes"` returns substantive ranked results, so
|
||||
the relay/memory path is functioning.
|
||||
|
||||
The `REPORTED` action above still stands (no token to auto-create a NousResearch issue); the **MONITORING**
|
||||
condition is now satisfied — no recurrence across four log generations.
|
||||
@@ -3,9 +3,8 @@ title: OmniRoute free-DeepSeek Silent Stops (Empty-After-Tool) — 2026-08-04
|
||||
type: incident
|
||||
status: monitoring
|
||||
created: 2026-08-04
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-11-07
|
||||
updated: 2026-11-12
|
||||
verified_on: 2026-11-12
|
||||
confidence: high
|
||||
tags: [omniroute, deepseek, free-tier, silent-failure, incident, mitigated]
|
||||
sources: [live-probe, omniroute-log, session-20260804]
|
||||
@@ -53,5 +52,23 @@ Backup of both pre-change rows: `/tmp/combo_backup_20260804.json` on the Hermes
|
||||
- Consider `auto`+explorationRate for self-tuning if free tier stays flaky.
|
||||
- Hermes-side: empty-post-tool turns are nudge/retry/fallback handled, but still end silently once exhausted — possible future improvement: surface "(empty after tool calls)" more visibly.
|
||||
|
||||
## Re-verified 2026-11-12 03:4x AWST (stocktake — page was 19d old, inside the staleness window)
|
||||
Both reworked combos probed live on **`omniroute-1`** (`100.78.63.20:20128`, `custom:omni-101`) with the
|
||||
correct `key_env` (`HERMES_CUSTOM_100_78_63_20_20128_API_KEY`) and the exact aliases used:
|
||||
|
||||
| Probe | Result |
|
||||
|---|---|
|
||||
| `good-free` (the 12-model rework) | **HTTP 200**, resolving to **`glm-5.2-cheap:free`** with a valid non-empty completion |
|
||||
| `free-stack` | **HTTP 200**, resolving to **`stepfun-ai/step-5-preview`** with a valid completion |
|
||||
|
||||
**The mitigation is holding, and the 2026-08-04 follow-up can be partially closed:** both combos answer with
|
||||
real content, and **neither resolves to a deepseek model** — which is exactly the intent of dropping the
|
||||
proven culprits from `good-free`. ⚠ **What is *not* re-verified:** I did not individually probe the
|
||||
free-DeepSeek upstreams (`238d5978/deepseek-v4-flash`, `nvidia/deepseek-ai/deepseek-v4-pro`,
|
||||
`deepseek-v4-flash-thinking:free`), so the original claim *"upstream free-DeepSeek tier still degraded"*
|
||||
stands **neither confirmed nor refuted** — the combos serve healthy non-deepseek free models, which may mean
|
||||
the upstream recovered *or* simply that it is no longer reached. **Restoring deepseek-first ordering is
|
||||
therefore still not justified** and should wait on a direct probe of those three upstreams.
|
||||
|
||||
## Refs
|
||||
- [[systems/omniroute]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Work Outlook Mailbox — Basic Auth Disabled (Himalaya)
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-10-03
|
||||
updated: 2026-10-25
|
||||
verified_on: 2026-10-25
|
||||
updated: 2026-11-12
|
||||
verified_on: 2026-11-12
|
||||
confidence: high
|
||||
tags: [email, himalaya, office365, oauth2, auth, incident]
|
||||
sources: [live-himalaya-debug, live-tcp-probe, cron-session-20260826_180700_a94098]
|
||||
@@ -12,7 +12,16 @@ sources: [live-himalaya-debug, live-tcp-probe, cron-session-20260826_180700_a940
|
||||
|
||||
# Work Outlook Mailbox — Basic Auth Disabled (Himalaya)
|
||||
|
||||
**Window:** first detected 2026-08-26; re-verified 2026-10-03 and again 2026-10-25 (still open)
|
||||
**Re-verified 2026-11-12 03:3x AWST (stocktake — page was 18d old, inside the staleness window):**
|
||||
the failure is unchanged and still reproduces exactly, with the **same error class**. `himalaya envelope
|
||||
list -a work -s 1` → `Error: cannot build IMAP client / 1: login not supported` — i.e. the server rejects
|
||||
the basic-auth `LOGIN` command itself rather than the credential, which is consistent with the SMTP/IMAP
|
||||
basic-auth shutdown recorded above rather than a changed password. **No remediation has landed since
|
||||
2026-10-25:** the account still authenticates by password (there is no OAuth2 token in the himalaya
|
||||
config), so path 1 remains the only viable route and still requires Anthony's sign-in. The capability gap
|
||||
described below is unchanged — no cron job polls this mailbox.
|
||||
|
||||
**Window:** first detected 2026-08-26; re-verified 2026-10-03, 2026-10-25 and 2026-11-12 (still open)
|
||||
**Status:** Open — blocked pending an OAuth2 path
|
||||
**Impact:** The agent cannot read or send from the `work` mailbox. Every work-email
|
||||
request is unanswerable until this is fixed.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: SFTPGo Outage — CT299 Offline (2026-09-02)
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-09-04
|
||||
updated: 2026-10-21
|
||||
verified_on: 2026-10-21
|
||||
updated: 2026-11-12
|
||||
verified_on: 2026-11-12
|
||||
confidence: high
|
||||
tags: [incident, sftpgo, ct299, tailscale, proxmox]
|
||||
sources: [session:20260902_014540_c193be]
|
||||
@@ -68,6 +68,20 @@ the node is `active` in `tailscale status` at `100.88.223.70`; the SFTPGo health
|
||||
`http://100.88.223.70:10080/healthz` returns **HTTP 401** (expected — auth required, service up).
|
||||
The 2026-09-02 outage remains resolved; no recurrence.
|
||||
|
||||
## Re-verified 2026-11-12 03:2x AWST (stocktake — page was 22d stale, the wiki's only stale page)
|
||||
Live state re-probed and **the residual claim still holds — no recurrence in 70 days.** All three sources
|
||||
checked independently:
|
||||
- `pct list` on `pve` (100.96.100.82) → **CT299 `sftpgo` `running`**; `pct config 299` confirms the guest
|
||||
(2 cores, 2048 MB RAM, **`swap: 512`** — the over-provisioned 43 Gi swap recorded in the root cause is
|
||||
gone, so that concern has since been addressed).
|
||||
- `tailscale status` peer record for `sftpgo` → **`Online: true`**, `100.88.223.70`, and
|
||||
`tailscale ping sftpgo` → **`pong from sftpgo (100.88.223.70) via 192.168.178.92:41641 in 0s`**.
|
||||
- SFTPGo health endpoint `http://100.88.223.70:10080/healthz` → **HTTP 401**, unchanged (auth required, service up).
|
||||
|
||||
Note that this outage and the two later CT299 freezes are **separate faults**: the 2026-09-02 cause was
|
||||
`tailscaled` death plus stale `lxc-attach` cgroup pinning, whereas 2026-11-04/05 was a wedged `vzdump`
|
||||
freezing the guest → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]] (also resolved).
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
1. **Clean up `lxc-attach` probes** — always exit cleanly and verify no lingering processes after diagnostic sessions
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Daily Marketing Job Radar — "All Targets Skipped by Pre-Dispatch Filter
|
||||
type: incident
|
||||
status: unresolved
|
||||
created: 2026-10-22
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
review_after: 2026-10-29
|
||||
updated: 2026-11-12
|
||||
verified_on: 2026-11-12
|
||||
review_after: 2026-11-26
|
||||
confidence: high
|
||||
tags: [cron, router, provider, job-search, failure-mode]
|
||||
sources: [live-jobs-json, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-cron-output-file]
|
||||
@@ -62,6 +62,31 @@ One weekday radar slot produced nothing: no new roles discovered, verified, de-d
|
||||
job DB, and no candidate summary delivered to topic `:1030`. The job recovers on its next weekday slot
|
||||
(2026-10-22 07:00) if the guard does not repeat.
|
||||
|
||||
## RECURRENCE — 2026-11-10, on a different job and a different pin (verified live 2026-11-12)
|
||||
|
||||
The signature is **not unique to the Job Radar**, and it has now recurred on a second job — which changes
|
||||
the "one job's guard" hypothesis of the original open questions into a fleet-level failure mode.
|
||||
|
||||
| Probe | Result |
|
||||
|---|---|
|
||||
| `executions.db` (retention floor **2026-11-06T05:09:30+08:00**, so this is an in-window read and **not** an all-time sweep) | exactly **one** row carries the string: **`cbfe434a3b5e` (Wiki Audit — Nightly), 2026-11-10, `status: failed`**, error `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters…` |
|
||||
| the original 2026-10-21 Job Radar row | **outside the retention floor**, so it cannot be re-read from the DB today; its evidence above stands on the 2026-10-22 capture. The per-job answer is unchanged in `jobs.json`: `dca8482e4f76` reads `ok` today |
|
||||
| `cron_incidents` row `cbfe43_ed01845310d8` | `failure_type: rate_limit`, `first_seen_at 2026-11-11T00:12`, **`closed_at: 2026-11-12T00:25` → `resolved`** (it was still `detected`/open at the 2026-11-11 audit) |
|
||||
| `dca8482e4f76` (Job Radar, the original subject) | **4 consecutive clean slots** — 2026-11-06, 11-09, 11-10, 11-11 all `completed`. The radar itself has **not** repeated the fault |
|
||||
| the recurring job's pin | `cbfe434a3b5e` runs `custom:omni-101` / **`free-stack`** — **not** the `smart-route` pin the original failure used |
|
||||
|
||||
**What this adds to the original diagnosis:** the string fires on at least **two different models** and
|
||||
**two different jobs**, so open question 1 ("if it is a local filter chain, one filter is probably
|
||||
mis-configured for this job's toolset") now looks **less likely** than the router-side-guard reading — a
|
||||
guard that blanks *every* target. The **prefix differs** between the two occurrences (`HTTP 429` in 2026-11,
|
||||
bare `Service temporarily unavailable` in 2026-10), which is the same "error class changes, verdict holds"
|
||||
pattern already recorded for the 401/403 cliproxy case → [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||||
The 2026-11 instance was **credits-driven** (`credits 100%`, guard reset after 6 h 24 m) — a quota
|
||||
exhaustion, not a config fault, which also matches the fleet's paid-pin posture in
|
||||
[[systems/scheduled-tasks]] (this stocktake itself runs `deepseek-paid-smart`, the only paid pin in the fleet).
|
||||
Open question 3 (**give the radar a second provider pin — it has no failover**) therefore now applies to the
|
||||
whole free-tier cohort, not just the radar, and remains unapplied pending Anthony's go-ahead.
|
||||
|
||||
## Open questions / next actions (proposed, not applied — needs Anthony's go-ahead)
|
||||
|
||||
1. **Find which component emits this string.** If it is a router-side guard with a fixed reset window, a
|
||||
|
||||
Reference in New Issue
Block a user