audit(2026-11-01): Conduit updated to 0.122.2 (memory-timeout patch now upstream, call store re-shaped); document the local-voice-mode fault, the force-drained gateway reload that orphaned a cron run, and both dead Asana paths; 2026-10-31 settled 160/4
This commit is contained in:
@@ -0,0 +1,102 @@
|
||||
---
|
||||
title: Asana Is Unreachable From CT460 on Both Paths — MCP Server Never Authenticated, Hosted Connector Rejects the Call Shape
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-01
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
review_after: 2026-11-08
|
||||
confidence: high
|
||||
tags: [hermes, automation, mcp, connector, credential, silent-failure]
|
||||
sources: [live-mcp-list, live-config-yaml, live-mcp-tokens, live-logs, live-probe]
|
||||
---
|
||||
|
||||
# Asana Is Unreachable From CT460 on Both Paths — MCP Server Never Authenticated, Hosted Connector Rejects the Call Shape
|
||||
|
||||
## Headline
|
||||
|
||||
The estate has **two independent Asana integrations on CT460, and neither can complete a read** — which
|
||||
went unnoticed until an interactive session spent 2026-10-31 evening retrying both to answer a single
|
||||
question ("where does Remembrance Day stand?").
|
||||
|
||||
| Path | Surface | Live state 2026-10-31/11-01 |
|
||||
|---|---|---|
|
||||
| **MCP server `asana`** | `https://mcp.asana.com/v2/mcp`, `auth: oauth`, `enabled: true` (`hermes mcp list` → ✓ enabled) | **Never authenticated.** Every attempt logs `MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Run \`hermes mcp login asana\` interactively first to complete initial authorization.` |
|
||||
| **Hosted connector** | `connectors__asana__GET_A_TASK`, `…__SEARCH_TASKS_IN_WORKSPACE` | Reaches Asana but returns **`422 INVALID_ARGUMENTS` — "Input should be a valid list on parameter \`opt_fields\`"**. 12+ identical retries in one session. |
|
||||
|
||||
## Evidence
|
||||
|
||||
**MCP server — no token, ever.** `~/.hermes/mcp-tokens/` holds exactly one file:
|
||||
|
||||
```
|
||||
-rw------- 1 hermes hermes 319 Oct 26 01:07 asana.client.json
|
||||
```
|
||||
|
||||
A registered **client** with **no token file** → the OAuth flow was started (client registered
|
||||
**2026-10-26 01:07**) and never completed. `config.yaml → mcp_servers.asana` reads
|
||||
`{auth: oauth, enabled: true, url: https://mcp.asana.com/v2/mcp}`. The earliest Asana trace in retained
|
||||
logs is an **HTTP 401** from `mcp.asana.com/v2/mcp` at **2026-10-27 16:35:05**, followed immediately by
|
||||
OAuth discovery calls.
|
||||
|
||||
**Failure volume** (log lines across `logs/agent.log*` + `logs/errors.log*`; the failure is written to
|
||||
both files, so line counts roughly double the number of events):
|
||||
|
||||
| Day | `MCP OAuth setup failed for 'asana'` lines |
|
||||
|---|---|
|
||||
| 2026-10-21 | 455 |
|
||||
| 2026-10-22 | 469 |
|
||||
| 2026-10-23 | 576 |
|
||||
| 2026-10-24 | 576 |
|
||||
| 2026-10-25 | 576 |
|
||||
| 2026-10-26 | 301 |
|
||||
| 2026-10-27 | 439 |
|
||||
| 2026-10-28 | 1 150 |
|
||||
| 2026-10-29 | 1 154 |
|
||||
| 2026-10-30 | 862 |
|
||||
| **2026-10-31** | **1 156** |
|
||||
| 2026-11-01 (to 00:16) | 14 |
|
||||
| **Total retained** | **7 728** |
|
||||
|
||||
The earliest retained line is **2026-10-21 05:09:43** (`errors.log.2`) — the gap **predates the retained
|
||||
log window**; this is a chronic condition, not a 10-31 regression. The 10-31 peak coincides with the
|
||||
evening Pulse session hammering the path.
|
||||
|
||||
**Hosted connector — the call, not the credential.** From `20261031_210329_b35440` (the Conduit Pulse
|
||||
"Remembrance Day" session), repeated at 22:17:47, 22:20:42, 22:26:31, 22:31:25, 22:36:45, 23:14:45,
|
||||
23:26:46, 23:28:55, 23:30:34, 23:39:31, 23:41:48, 23:51:19 — each call burning 11–20 s:
|
||||
|
||||
```
|
||||
tools.managed_gateway_identity: connector connectors.execute 422 INVALID_ARGUMENTS
|
||||
Tool tool_call returned error: {"results":[{"name":"connectors__asana__GET_A_TASK","error":
|
||||
{"code":"INVALID_ARGUMENTS","message":"Invalid request data provided\n- Input should be a valid list
|
||||
on parameter `opt_fields`…"}}]}
|
||||
```
|
||||
|
||||
The connector is **authorised** (it is the MCP path that cannot authenticate), so the block is purely
|
||||
argument shape: `opt_fields` must be a **JSON list**, and a string/comma-form is rejected.
|
||||
|
||||
## Impact
|
||||
|
||||
- **Non-interactive sessions (cron, background jobs, Conduit background investigations) cannot read
|
||||
Asana at all** — the MCP path needs an interactive browser login that cron cannot provide.
|
||||
- **Interactive sessions** get the hosted connector, but the agent re-issued the same mis-shaped call
|
||||
a dozen times instead of adapting — ~3 minutes of model time spent failing identically.
|
||||
- Asana-backed user requests (task status behind the Conduit Pulse flow) degrade to guesses.
|
||||
|
||||
## Remediation (not performed — this audit records, it does not re-authorise)
|
||||
|
||||
1. Run **`hermes mcp login asana`** from an interactive session to complete the OAuth flow; confirm
|
||||
`~/.hermes/mcp-tokens/asana.json` (or equivalent token file) appears afterwards, then re-probe the
|
||||
MCP server.
|
||||
2. **Or** disable the `asana` MCP server if the hosted connector is the intended path — two Asana paths
|
||||
on one host is itself the source of the confusion, and one of them has never worked.
|
||||
3. For the connector: pass **`opt_fields` as a JSON list**, not a comma-separated string.
|
||||
4. Watch for the silent-cost pattern: a path that fails **every** time still burns a tool call each
|
||||
time. 1 100+ failures/day means the failure is being retried, not fixed.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/hermes-agent]] — MCP server inventory and auth state
|
||||
- [[systems/conduit]] — the Pulse flow that needed the Asana task status
|
||||
- [[systems/scheduled-tasks]] — cron fleet (the surface that cannot authenticate interactively)
|
||||
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the same "green row, dead destination" class
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
title: Conduit "voice mode: local" never applies — Mac mini speech server's LLM proxy is disabled
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-01
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
confidence: high
|
||||
tags: [conduit, voice, local-live, mac-mini, speech-server, investigation]
|
||||
sources:
|
||||
- session 20261031_193709_40b9c2 (Conduit voice session, started 2026-10-31 19:37:09 AWST)
|
||||
- ~/.hermes/conduit-settings.json, ~/.hermes/conduit-calls.json, ~/.hermes/config.yaml
|
||||
- live probes of the Mac mini speech server (LLM endpoint, /v1/usage, realtime socket)
|
||||
---
|
||||
|
||||
# Conduit "voice mode: local" never applies
|
||||
|
||||
**Status: OPEN — investigation in progress. Nothing was changed on the Mac mini speech server.**
|
||||
|
||||
On 2026-10-31 Anthony reported (from the phone): switching the Conduit app's voice
|
||||
setting to **"local"** shows a **successful** connection test, but the change never
|
||||
takes effect. A background job launched from that voice conversation
|
||||
(`20261031_193709_40b9c2`, started **19:37:09 AWST**, 30 messages / 17 tool calls) was
|
||||
still open at the 2026-11-01 00:2x audit.
|
||||
|
||||
## Why the test succeeds and the change still does nothing
|
||||
|
||||
The "test connection" button checks **socket reachability only** — it never exercises
|
||||
the model path. So it passes while every local turn is still impossible:
|
||||
|
||||
| # | Finding | Evidence |
|
||||
|---|---------|----------|
|
||||
| 1 | The Mac mini speech server's **LLM proxy is disabled** | its LLM endpoint answers **HTTP 501** — *"The LLM proxy is disabled. Start the server with `--enable_llm_proxy` to enable it."* |
|
||||
| 2 | The Mac has never served a local turn | `/v1/usage` → **`llm_proxy.requests: 0`** across **25** observed turns |
|
||||
| 3 | The realtime handshake is not understood | the Mac's realtime socket **accepts the connection** but rejects **`session.update`** as an **unknown event** |
|
||||
| 4 | The app's test button proves nothing about the model path | it returns success on socket reachability alone |
|
||||
|
||||
Findings 1–3 are the strongest single explanation: **even with `voice.mode = local`
|
||||
set on the phone, the Mac cannot call a model back.** Finding 4 is why the UI reports
|
||||
success.
|
||||
|
||||
## `voice.mode` is phone-side, not host-side
|
||||
|
||||
The host never reads a `voice.mode` key — there is none to read:
|
||||
|
||||
- `~/.hermes/config.yaml → voice` holds **`auto_tts`**, **`local_live`**,
|
||||
**`speech_server`** — **no `mode`**.
|
||||
- `~/.hermes/conduit-settings.json` contains **no `voice.*` keys at all**; its only
|
||||
device state is `devices/<UUID>/mode: small` for three devices.
|
||||
- The device itself reports **`voice.mode: "classic"`**.
|
||||
|
||||
So "set mode to local and the change does not apply" cannot be diagnosed on the host
|
||||
config — the selector lives in the app, and what the host does with it is a
|
||||
**`local_live`** path decision.
|
||||
|
||||
## A misread that was caught mid-session (worth keeping)
|
||||
|
||||
`local_live.py:882` is the **only** writer of `_local_live_last_request`, and it sits
|
||||
inside the **chat-completions route**. A "recent `last_request_at`" reading during the
|
||||
investigation therefore indicated **this investigation's own probe**, not phone
|
||||
traffic — the agent initially read it as phone activity and corrected the call in the
|
||||
same session. Live check: `conduit-calls.json → last_request_at` is **`null`**.
|
||||
|
||||
Lesson: on this integration, `last_request_at` is proof of *some* HTTP request hitting
|
||||
the local-live route, **not** proof of a phone call.
|
||||
|
||||
## Configuration at the time of writing
|
||||
|
||||
Changed during the 2026-10-31 session (`config.yaml` mtime **21:02:30**):
|
||||
|
||||
- `voice.local_live.front_model` = `{base_url: http://100.78.63.20:20128/v1, model: mistral/voxtral-small-latest}`
|
||||
— was `mistral/ministral-14b-latest`.
|
||||
|
||||
Unchanged: `voice.speech_server.url = wss://…:8443/v1/realtime`, `voice.auto_tts: false`,
|
||||
`tts.provider: ollama_local`, `stt.provider: local`.
|
||||
|
||||
The Conduit dashboard gateway was restarted at **19:35:02** (pid 176771), minutes
|
||||
before the session opened.
|
||||
|
||||
## Prompting context (candidate, not proven)
|
||||
|
||||
The plugin **self-updated to `0.122.2` on 2026-10-31 18:05**, which drove a gateway
|
||||
reload at **18:06:41** → [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]].
|
||||
A newly shipped local-live path is a reasonable regression candidate, but the
|
||||
disabled LLM proxy would break local mode on **any** plugin version, so the version
|
||||
change is not yet implicated.
|
||||
|
||||
## Open items
|
||||
|
||||
- **Enable the Mac mini's LLM proxy** (`--enable_llm_proxy`) or confirm the intended
|
||||
Local Live architecture — with the proxy off, finding 1 alone explains the symptom.
|
||||
- Confirm the **realtime event vocabulary** the plugin expects vs. what the Mac's
|
||||
socket implements (`session.update` is currently rejected as unknown).
|
||||
- Decide whether `front_model` should be a **text** or **audio** model — a
|
||||
chat-completions `base_url` pointed at an audio model may itself be the mismatch.
|
||||
- Re-test from the phone after any change; the app's own test button is not evidence.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/conduit]]
|
||||
- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
|
||||
- [[incidents/2026-10-29-conduit-voice-silence]]
|
||||
- [[incidents/2026-10-22-conduit-memory-start-timeout]]
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
title: Gateway Reload on 2026-10-31 Force-Drained After Its Full 1 800 s Cap and Orphaned a Live Cron Run — Mnemosyne Health Monitor (6 h 38 m in)
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-01
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
review_after: 2026-11-08
|
||||
confidence: high
|
||||
tags: [gateway, cron, scheduler, lifecycle, restart, memory, silent-failure]
|
||||
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-probe]
|
||||
---
|
||||
|
||||
# Gateway Reload on 2026-10-31 Force-Drained After Its Full 1 800 s Cap and Orphaned a Live Cron Run — Mnemosyne Health Monitor (6 h 38 m in)
|
||||
|
||||
## Headline
|
||||
|
||||
On **2026-10-31 at 18:06:41 AWST** the gateway received a **`SIGUSR1` reload** (`systemctl reload` /
|
||||
`hermes gateway restart`). It deferred its own stop for in-flight work — and this time it **waited the
|
||||
whole deferral cap**, hit it at **18:37:38** with *"drain timed out with interrupted agents"*, and
|
||||
force-restarted. The casualty was `ea79a2107e36` **Mnemosyne Health Monitor** (12-hourly `0 */12 * * *`),
|
||||
whose 12:01:06 slot had been running for **6 h 38 m 09 s**. It closed **`failed`** at 18:39:15 on
|
||||
**`RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership`**.
|
||||
|
||||
This is the **third variant of the orphan class** — after the 2026-10-20 *unclean crash* exit and the
|
||||
2026-10-26 *planned `SIGTERM` whose drain timed out after 1.35 s* — and the first where the drain
|
||||
mechanism behaved as designed and still ended in a lost run.
|
||||
|
||||
## Timeline (live gateway log, `~/.hermes/logs/agent.log`)
|
||||
|
||||
| Time (AWST) | Event |
|
||||
|---|---|
|
||||
| 18:06:41.772 | `SIGUSR1 received (systemctl reload / hermes gateway restart): performing a graceful gateway restart — drain active turns, exit, supervisor relaunches.` |
|
||||
| 18:06:41.788 | `Restart requested with 2 active work unit(s); deferring stop() until they finish (cap=1800s) so in-flight turns are not amputated (#77184)` |
|
||||
| 18:06:41.899 | Deferred on **1** unit: the cron unit `ea79a2107e36` (`elapsed_s: 21950.7`, `external: True`, `wedged: False`, **`restart_safe: True`** — *excluded* from the wait) plus `{'kind': 'managed', 'pid': 397}` |
|
||||
| 18:07:11 → 18:11:12 | Countdown logged every 30 s: 1770 s → 1529 s remaining, same single `managed pid 397` unit |
|
||||
| 18:11:23 | **Second `SIGUSR1`** arrives while the first restart is still deferred |
|
||||
| 18:20:13 | `Restart deferred: … (988s remaining before force drain): [{'kind': 'managed', 'pid': 397}]` |
|
||||
| **≈18:36:41** | **Cap reached** (18:06:41 + 1 800 s) |
|
||||
| 18:37:38.341 | `Skipping .clean_shutdown marker — drain timed out with interrupted agents; next startup will recover their interrupted turns.` |
|
||||
| 18:38:43.284 | New gateway starts; 18:39:14 reconnection watcher for 1 failed platform (`relay`) |
|
||||
| 18:39:15.208 | Failure **delivered** (delivery row `9e7da0fa…`, `for_failure=1`, status `delivered`) |
|
||||
| 18:39:15.563 | Execution `9e7da0fafd2847b69643c6f235cb8651` closes **`failed`** — 6 h 38 m 09 s |
|
||||
| 18:39:23 / 18:45:23 / 19:00:22 | Three derivative `96eb95ce12f2` (log-error watchdog) reds quote the cron error; the watchdog clears on its 19:15:21 slot |
|
||||
| 19:23:39 → 19:24:40 | **Second restart, clean**: `drain took 0.00s, timed_out=False`; `Previous gateway exited cleanly — skipping session suspension` |
|
||||
|
||||
Current gateway: `hermes-gateway.service` active since **19:23:56**, PID `175269` (4 h 43 m at the
|
||||
2026-11-01 00:07 probe). Nothing failed for the remaining ~5 h of 10-31 (every `96eb95ce12f2` and
|
||||
`0c771463c3cd` slot from 19:15:21 onward `completed`).
|
||||
|
||||
## Incident-ledger effect
|
||||
|
||||
| Row | Job | State | First seen | Last seen |
|
||||
|---|---|---|---|---|
|
||||
| `ea79a2_9197df32320b` | `ea79a2107e36` | **`alerted` (open)** | 2026-10-31T18:06:44 | 2026-10-31T18:06:44 |
|
||||
| `ea79a2_a4661554a3c2` | `ea79a2107e36` | **`alerted` (open, reused row)** | **2026-10-26**T14:35:59 | 2026-10-31T18:39:15 |
|
||||
| `0e753e_e7f8194a7d06` | `0e753e9720fd` | `detected` (open, unchanged) | 2026-10-01T09:00:55 | 2026-10-01T09:00:55 |
|
||||
|
||||
The `error_sig` match re-opened the **2026-10-26 row** rather than filing a fresh one, so two rows now
|
||||
describe one event: the earlier *"Cron execution unverified; reconcile `admissions/292056e5742c1ba4…`"*
|
||||
(`failure_type: unknown`) and the definitive *fire-claim* row. **Unclosed incident rows 1 → 3.**
|
||||
|
||||
## What is genuinely new here
|
||||
|
||||
1. **The deferral cap is reachable in normal operation, not just under a stuck process.** The 10-26
|
||||
incident drained in **1.35 s**; this one waited the full **1 800 s** because a `managed` unit
|
||||
(`pid 397`) stayed registered. Force drain is therefore a *routine* outcome of any reload that
|
||||
overlaps a long-running cron job.
|
||||
2. **`restart_safe: True` did not protect the execution.** The scheduler explicitly classified the cron
|
||||
unit as restart-safe and *excluded* it from the wait — and the run still died with a lost claim. The
|
||||
flag protects the *scheduler's* bookkeeping, not the running agent's lease.
|
||||
3. **A `SIGUSR1` reload is not a lighter-weight restart.** The pre-restart log shows plugin/update
|
||||
machinery active minutes earlier (`.update-<id>/plugin` capability checks at 18:05:41 and 18:19:53,
|
||||
`plugin scan passed` at 18:20:22) — the reload behaved exactly like a full stop/start for the
|
||||
in-flight job.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Why is a `managed` (non-cron) work unit allowed to hold the gateway for the entire 30-minute cap?
|
||||
What is `pid 397`, and should it be exempt from the "wait" set (or bounded separately)?
|
||||
- The 12:01 slot had already run **6 h 38 m** against **10 h 01 m** for its 10-30 catch-up. Should a
|
||||
reload deferral weight long cron runs differently — or should the cron lease survive a supervisor
|
||||
relaunch, as the `restart_safe` flag implies?
|
||||
- Two `SIGUSR1`s 4 m 42 s apart produced a **single** interruption and then a second clean restart at
|
||||
19:23 — is the double signal expected?
|
||||
|
||||
## Impact
|
||||
|
||||
- One job failure + three derivative watchdog reds = **all four** of 2026-10-31's failed executions
|
||||
(day closed **160 `completed` / 4 `failed`**).
|
||||
- `ea79a2107e36` now reads `last_status: error`, **`failure_streak: 2`** — the only non-zero streak in
|
||||
the fleet; non-clean `last_status` **3 → 4**, failing (run failed outright) **0 → 1**.
|
||||
- Its next slot (2026-11-01 **00:00:41**) was already `running` at audit time.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]] — fleet ledger and the 10-31 window numbers
|
||||
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — variant 2, the 1.35 s drain timeout
|
||||
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — variant 1, the unclean crash exit
|
||||
- [[systems/hermes-agent]] — gateway lifecycle record
|
||||
- [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — the same job, the day before
|
||||
Reference in New Issue
Block a user