audit(2026-11-01): Conduit updated to 0.122.2 (memory-timeout patch now upstream, call store re-shaped); document the local-voice-mode fault, the force-drained gateway reload that orphaned a cron run, and both dead Asana paths; 2026-10-31 settled 160/4
This commit is contained in:
@@ -38,6 +38,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
||||
- **History:** as at 2026-07-22 this was `custom:litellm` → `litellm:4000` with model `freellmapi-auto`
|
||||
- **Provider model count (2026-10-01):** 5 custom providers — `omni-101` 5,384, `Omniroute:20128` 5,398, `litellm` 1,728, `command-code` 71, `guacano` 46
|
||||
- **MCP servers:** 5 configured (`Not human search`, `asana`, `cua-driver`, `hound`, `mcp-portal`) — verified 2026-10-01
|
||||
⚠ **Auth state re-verified 2026-11-01: `asana` is enabled but has never been authenticated.** `hermes mcp list` shows it ✓ enabled (`https://mcp.asana.com/v2/mcp`, `auth: oauth`), but `~/.hermes/mcp-tokens/` holds **only** `asana.client.json` (319 B, mtime 2026-10-26 01:07) — no token file — so every attempt logs `MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Run \`hermes mcp login asana\` interactively first`. The other four servers are unaffected → [[incidents/2026-10-31-asana-access-failures]]
|
||||
- **Plugins enabled:** 11 — verified 2026-10-01
|
||||
- **free-stack combo (OmniRoute):** Nemotron 3 Ultra 550B configured with `max_tokens: 8192`, `reasoning_effort: medium`, `chat_template_kwargs.force_nonempty_content: true` (see [[omniroute]])
|
||||
- See [[model-providers]] for full breakdown.
|
||||
@@ -227,3 +228,28 @@ and all three were fixed on 2026-10-29 between 08:02 and 08:4x AWST:
|
||||
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
- **Restart hygiene:** a gateway restart resets in-process counters (Mattermost retries, cron admission
|
||||
bookkeeping). Never read a counter as an outage duration.
|
||||
|
||||
## 2026-11-01 00:2x AWST — gateway reloads on 10-31 (one force-drained, one clean) and the Asana MCP auth gap
|
||||
|
||||
- **Two gateway starts on 2026-10-31, and the first one killed a live cron run.**
|
||||
- **18:06:41** — `SIGUSR1 received (systemctl reload / hermes gateway restart)`: *"Restart requested with
|
||||
2 active work unit(s); deferring stop() until they finish (cap=1800s)"*. The countdown ran every 30 s
|
||||
from **1770 s** to **988 s** (18:20:13) against a single `managed` unit (`pid 397`); a **second
|
||||
`SIGUSR1`** landed at **18:11:23**. The cap expired at **18:37:38** → *"Skipping `.clean_shutdown`
|
||||
marker — drain timed out with interrupted agents"*. New gateway **18:38:43**. The 12:01:06 slot of
|
||||
`ea79a2107e36` (Mnemosyne Health Monitor) closed **`failed`** at **18:39:15** on *"lost its durable
|
||||
fire claim ownership"* after **6 h 38 m** → [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]].
|
||||
- **19:23:39 → 19:24:40** — a second restart, **clean**: `drain took 0.00s, timed_out=False`,
|
||||
*"Previous gateway exited cleanly — skipping session suspension"*. Current gateway **PID 175269**, active
|
||||
since **19:23:56** (4 h 43 m at the audit probe). Nothing failed after 19:15:21.
|
||||
- **Two Telegram adapter rebuilds, both self-recovered:** `05:03:42` and `09:12:11` —
|
||||
`Fatal telegram adapter error (telegram_network_error): Telegram updater.stop() did not finish before the
|
||||
network-recovery deadline; rebuilding the adapter instead of reusing an Updater whose lifecycle lock may
|
||||
still be held. (likely CLOSE-WAIT socket)`.
|
||||
- **Plugin update machinery was active minutes before the reload** (`.update-<id>/plugin` capability checks
|
||||
18:05:41 and 18:19:53, `plugin scan passed` 18:20:22) and the 04:56:54 log line advertised updates for
|
||||
`claude-subscription-directsdk-experimental`, `conduit_push` and `homeassistant`. **Agent version
|
||||
unchanged** — `hermes-agent` HEAD is still `bf0203a7` (2026-10-22).
|
||||
- **MCP inventory unchanged at 5 servers, but `asana` has never authenticated** (see the MCP line above and
|
||||
[[incidents/2026-10-31-asana-access-failures]]): **7 728** `MCP OAuth setup failed for 'asana'` log lines in
|
||||
the retained window (≥2026-10-21), **1 156** of them on 10-31.
|
||||
|
||||
Reference in New Issue
Block a user