audit(2026-11-01): Conduit updated to 0.122.2 (memory-timeout patch now upstream, call store re-shaped); document the local-voice-mode fault, the force-drained gateway reload that orphaned a cron run, and both dead Asana paths; 2026-10-31 settled 160/4

This commit is contained in:
Tony0410
2026-11-01 00:30:42 +08:00
parent 74a5b2d117
commit 0c826f2b5f
8 changed files with 488 additions and 22 deletions
+26
View File
@@ -38,6 +38,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
- **History:** as at 2026-07-22 this was `custom:litellm` → `litellm:4000` with model `freellmapi-auto`
- **Provider model count (2026-10-01):** 5 custom providers — `omni-101` 5,384, `Omniroute:20128` 5,398, `litellm` 1,728, `command-code` 71, `guacano` 46
- **MCP servers:** 5 configured (`Not human search`, `asana`, `cua-driver`, `hound`, `mcp-portal`) — verified 2026-10-01
⚠ **Auth state re-verified 2026-11-01: `asana` is enabled but has never been authenticated.** `hermes mcp list` shows it ✓ enabled (`https://mcp.asana.com/v2/mcp`, `auth: oauth`), but `~/.hermes/mcp-tokens/` holds **only** `asana.client.json` (319 B, mtime 2026-10-26 01:07) — no token file — so every attempt logs `MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Run \`hermes mcp login asana\` interactively first`. The other four servers are unaffected → [[incidents/2026-10-31-asana-access-failures]]
- **Plugins enabled:** 11 — verified 2026-10-01
- **free-stack combo (OmniRoute):** Nemotron 3 Ultra 550B configured with `max_tokens: 8192`, `reasoning_effort: medium`, `chat_template_kwargs.force_nonempty_content: true` (see [[omniroute]])
- See [[model-providers]] for full breakdown.
@@ -227,3 +228,28 @@ and all three were fixed on 2026-10-29 between 08:02 and 08:4x AWST:
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
- **Restart hygiene:** a gateway restart resets in-process counters (Mattermost retries, cron admission
bookkeeping). Never read a counter as an outage duration.
## 2026-11-01 00:2x AWST — gateway reloads on 10-31 (one force-drained, one clean) and the Asana MCP auth gap
- **Two gateway starts on 2026-10-31, and the first one killed a live cron run.**
- **18:06:41** — `SIGUSR1 received (systemctl reload / hermes gateway restart)`: *"Restart requested with
2 active work unit(s); deferring stop() until they finish (cap=1800s)"*. The countdown ran every 30 s
from **1770 s** to **988 s** (18:20:13) against a single `managed` unit (`pid 397`); a **second
`SIGUSR1`** landed at **18:11:23**. The cap expired at **18:37:38** → *"Skipping `.clean_shutdown`
marker — drain timed out with interrupted agents"*. New gateway **18:38:43**. The 12:01:06 slot of
`ea79a2107e36` (Mnemosyne Health Monitor) closed **`failed`** at **18:39:15** on *"lost its durable
fire claim ownership"* after **6 h 38 m** → [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]].
- **19:23:39 → 19:24:40** — a second restart, **clean**: `drain took 0.00s, timed_out=False`,
*"Previous gateway exited cleanly — skipping session suspension"*. Current gateway **PID 175269**, active
since **19:23:56** (4 h 43 m at the audit probe). Nothing failed after 19:15:21.
- **Two Telegram adapter rebuilds, both self-recovered:** `05:03:42` and `09:12:11` —
`Fatal telegram adapter error (telegram_network_error): Telegram updater.stop() did not finish before the
network-recovery deadline; rebuilding the adapter instead of reusing an Updater whose lifecycle lock may
still be held. (likely CLOSE-WAIT socket)`.
- **Plugin update machinery was active minutes before the reload** (`.update-<id>/plugin` capability checks
18:05:41 and 18:19:53, `plugin scan passed` 18:20:22) and the 04:56:54 log line advertised updates for
`claude-subscription-directsdk-experimental`, `conduit_push` and `homeassistant`. **Agent version
unchanged** — `hermes-agent` HEAD is still `bf0203a7` (2026-10-22).
- **MCP inventory unchanged at 5 servers, but `asana` has never authenticated** (see the MCP line above and
[[incidents/2026-10-31-asana-access-failures]]): **7 728** `MCP OAuth setup failed for 'asana'` log lines in
the retained window (≥2026-10-21), **1 156** of them on 10-31.