Files
agent-estate-wiki/systems/conduit.md
T

23 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Conduit system active 2026-10-24 2026-11-17 2026-11-17 2026-12-08 high
conduit
plugin
phone
iphone
push
voice
memory
tool-calling
session-ttl
hook
timeout
pre-tool-call
live-probe
plugin-tree
live-sqlite-conduit
live-sqlite-cron
live-log
session-20261024_125329
session-20260908_031541_49c051

Conduit

Purpose

Hermes on Anthony's phone. A dashboard plugin + iOS app pair that pushes cards, prompts, location and health data between the Hermes gateway and the iPhone, and lets a voice call be started from the phone where the agent can speak and listen (conduit-voice sessions).

Related: systems/mnemosyne (memory backend the voice session reads), systems/hermes-ui (the browser desktop), systems/push-notifications (the older hermes-push bridge).

Runtime checks (2026-10-30 00:0x AWST)

  • conduit-calls.json → history entries: 2 — the first real calls ever recorded (2026-10-29 08:38:04 and 08:49:36). The path is no longer unused; the earlier "0 calls" reading is superseded. See incidents/2026-10-29-conduit-voice-silence.
  • Calls still unproven end-to-end: last_request_at remained null at the end of the 2026-10-29 window — no call is yet known to have run a real Hermes turn.
  • ends vs history: ends records any turn-end that fires a watch (including cron sessions) — never read it as a call log.
  • Device: iPhone app version now 2.2.0 (was 1.10.1 on 2026-10-28), 19 settings keys, no voice.local.* keys → Local Live cannot be configured host-side any more.
  • Voice providers were both dead and are now fixed — tts.provider: ollama_local, stt.provider: local (2026-10-29). Details below and in the 10-29 incident.

Where it lives

  • Plugin tree: /home/hermes/.hermes/plugins/conduit_push
    • Version (re-verified 2026-11-01): 0.122.2 (plugin.yaml) — the plugin updated itself on 2026-10-31 18:05 (plugin.yaml mtime) and fetched at 18:19 (FETCH_HEAD); git HEAD 9eac857, "Every profile that turns Conduit on runs the main install (#482)", upstream commit 2026-10-31 12:04:31 AWST. Working tree clean; +148 commits on from the 0.69.2 reading below (#342 → #482). See the 2026-11-01 section at the foot of this page.
  • ⚠ Superseded (2026-10-26 reading): 0.69.2 (plugin.yaml) / 0.69.2 (dashboard manifest), git HEAD 942ddb8 — Plugin and relay fixes left over from earlier reviews (0.69.2) (#342), committed 2026-10-24 15:35:42 +0000. 0 commits behind origin/main; 24 commits on from the 0.64.0 / eaa14be baseline of the 2026-10-24 audit (upstream #324–#342: Pulse cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway restart, channels.yaml, App Store builds following the archived plugin version).
  • ⚠ Superseded provenance: the 2026-10-24 audit read 0.69.0 / f000bf0; two further commits landed on 2026-10-24 15:35. Git HEAD was previously recorded as: - **Version:** 0.69.0 (plugin.yamlanddashboard/manifest.json), git HEAD f000bf0— *“Pulse: list candidates, decide in code, journal them (#337)”*. **22 commits on from the0.64.0/eaa14berecorded at the 2026-10-24 audit** (upstream#324–#338: Pulse cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway restart, channels.yaml`, App Store builds following the archived plugin version).
  • Upstream: https://github.com/kaishi00/hermes-conduit-notifier.git (origin, fetch+push)
  • ✅ The uncommitted local patch is GONE — and it landed upstream. git status --short on the 0.122.2 tree is clean, and dashboard/api/memory.py now ships MEMORY_TIMEOUT_S = 10.0 applied to every provider call with an asyncio.wait_for(…, MEMORY_TIMEOUT_S + 2.0) backstop, plus its own tests/test_memory.py. So the fix in incidents/2026-10-22-conduit-memory-start-timeout is upstream, not lost — the 10-23 in-tree edit is no longer needed. ⚠ Superseded: the working tree still carries the uncommitted local patch (M dashboard/plugin_api.py, the memory-timeout fix in incidents/2026-10-22-conduit-memory-start-timeout) — now five releases behind HEAD (0.64.0 → 0.69.0), so the chance a plugin update resets the tree and drops it is rising, not falling.
  • Skills shipped with the plugin (4, under plugins/conduit_push/skills/): calling-the-user, reading-health-data, using-calendar, using-location.
  • Data stores (~/.hermes/): conduit-health.sqlite3, conduit-location.sqlite3, conduit-phone.sqlite3, plus JSON state — conduit-push.json (credential / gateway id / installation id / relay url / e2e — no values recorded here), conduit-calls.json, conduit-settings.json, conduit-present.json, conduit-desktop-views.json, conduit-link-status.json, conduit-local-live.json (holds an api key — never copy its contents into the wiki).

Current state (re-verified 2026-10-30 00:0x AWST)

  • Dashboard gateway: listening on 0.0.0.0:9119; pid 176771, started 2026-10-31 19:35:02 (4 h 50 m at the 2026-11-01 00:2x probe; GET / → 302 to auth) — ⚠ superseded: pid 4042003, started 2026-10-29 08:03:26 — the restart that put the 30-day session TTL live (see below). The 0.64.0 → 0.69.0 upstream commits remain not live (the process predates nothing relevant; the tree still only carries the memory-timeout patch).
  • Device: one registered — iPhone, app version 2.2.0 (self-updated from 1.10.1 on 2026-10-29), seen_at 2026-10-29 18:56:44 AWST, mode: small, 19 settings keys. last_seen / last_sent / last_contact are null in the current store — those three fields track phone-status contact, not data flow (health rows and link pings keep arriving regardless: conduit-link-status.json app_seen_at 2026-10-29 23:53:15, push_ok_at 18:50:00). Paired 2026-10-22 12:54:39; sharing window 2026-10-16 → 2027-04-22.
  • Dashboard session TTL (raised 2026-10-29): dashboard.basic_auth.session_ttl_seconds = 2592000 (30 days), up from the 43200 (12 h) default. A lapsed session was making Conduit's call socket answer WS 4401 "Dashboard sign-in required" — silently. ⚠ dashboard.session_ttl_seconds is a separate key and still 43200; check both.
  • Call settings: enabled, when_asked, decides, alerts all true; rate limits min_gap_s 120 / per_hour 6 / per_day 20; avoid_unknown_place false.
  • ⚠ The call store was re-shaped by 0.122.x: history is now EMPTY (0 entries) and last_request_at is null. The file is v: 1 with keys outcomes / delivered / told / watches / waiting / seeded / ends / settings; ends holds 2 turn-end records (both outcome: stopped, at 21:06:05 and 21:10:48 on 10-31) and settings has gained quiet_at_screen: true (upstream #479 — alert calls wait while you're at a screen). The two 10-29 calls below are no longer in the store — do not cite them as live history. ⚠ Superseded — recent calls, as recorded on 2026-10-30: two, both 2026-10-29. conduit-calls.json → history now reads 2 (08:38:04, 08:49:36). The 08:38:04 call produced no Mac connection and no Hermes turn, and the iPhone re-authenticated at 08:39:05 — the expired-session fault. The 08:49:36 call was placed after the classic mode switch and the STT/TTS fixes, but the session ended one second earlier, so its outcome is unrecorded. The memory-timeout patch therefore still has no confirmed call behind it → incidents/2026-10-29-conduit-voice-silence.
  • Data volume (2026-11-01 00:2x): conduit-health.sqlite3 938.2 MB (983,793,664 B) holding 2,623,879 events, 4,384 buckets, 504 windows — +8.2 MB and +22,314 events / +279 buckets / +93 windows since the 10-30 reading: growth is steady at ≈8 MB/day with no retention policy. (conduit-location.sqlite3 80 KB, mtime 10-31 12:15; conduit-phone.sqlite3 296 KB, mtime 10-31 14:54.) The per-kind event split was not re-measured this pass. ⚠ Superseded (2026-10-30): 930.0 MB (+360 MB in 5 days; +8.2 MB/25.7 h as measured on 10-25 — the rate has not held, so treat growth as ingestion-paced, unbounded), holding 2,601,565 events (2,592,189 sample / 6,146 sleep / 3,230 workout), 4,105 buckets and 411 windows; newest health row 2026-10-29 23:12:01 AWST. conduit-location.sqlite3 80 KB; conduit-phone.sqlite3 284 KB (last measured 10-25). ⚠ The 10-24 page recorded 308 buckets / 62 windows; a direct COUNT(*) reads 4,105 / 411, so the earlier pair was a narrower grouping — quote these. No retention or compaction has been observed, and the store is now closing on 1 GB — watch it against the CT460 disk budget.

Calls from Hermes (the conduit_call_user tool)

The plugin registers a conduit_call_user tool (toolset conduit, 📞) for every session and ships the matching skill calling-the-user. Hermes exposes it through lazy tool discovery (tool_search), so it is not in a session's headline tool list — it has to be looked up. Gate conditions (call_tool.py): a real session (subagents are refused), a paired device, settings.enabled, and either when_asked (with the ask recorded as user-originated) or decides; the call rings when the asking turn ends and plays the reason first. Limits: min_gap_s 120 / 6 per hour / 20 per day.

⚠ Known failure, 2026-10-24: three requests to be phoned were refused by the agent, which asserted the capability did not exist without calling the tool or loading the skill → incidents/2026-10-24-conduit-call-request-refused. Positive counter-example 2026-10-29: in that window the agent did look the tool up and did place calls (two entries in history), so the failures were behavioural, not a missing capability.

Voice pipeline (verified 2026-10-30)

Phone voice sessions ride the Mac mini speech server, which is also the Hermes TTS/STT target. All three settings below were wrong or dead as of 2026-10-28 and were fixed on 2026-10-29 — see incidents/2026-10-29-conduit-voice-silence.

Setting Value (now) Was Failure when wrong
voice.speech_server.url wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime …:8443/v1/ 403 — the plugin dials the value verbatim, it does not append /realtime
tts.provider ollama_local (~/.hermes/scripts/ollama-tts.sh, model phonon) openai → Mac /v1/speech 502
stt.provider local (faster_whisper, model base) groq 403 / Cloudflare error 1010 — a block, not a bad key (fails identically with a bogus key)
  • voice.mode = classic (switched from local 2026-10-29). Classic relays everything through the plugin and needs no phone-side server address — which is why both host providers above had to work. A Local Live path cannot be configured from the host at all: app 2.2.0's schema exposes 19 keys, none of them voice.local.*.
  • voice.auto_tts: false — assistant replies are not auto-spoken; audio appears only when requested.
  • wake_word.enabled: true.
  • Health check: GET https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/pool returned {"size": 2, "in_use": 0} with both units idle on 2026-10-28 00:2x — a 2-unit pool. /v1/models is not implemented ({"detail":"Not Found"}); probe /v1/pool, not /v1/models.
  • End-to-end proof (2026-10-29): driving the real call path produced session.created → transcription of live speech → response.done, with the Mac counters moving 0 → 10 connections / 1 completed response / 2.98 s audio / 0 errors and the plugin handshaking in 41 ms.
  • ⚠ Fragile URL: the Conduit settings tool writes Hermes config (it had flipped tts.provider to openai). If it or an app update ever writes the bare /v1/ back, every call silently 403s again.

Memory integration

The voice session (conduit-voice) reaches Mnemosyne through the plugin's own memory layer, not through Mnemosyne's system_prompt_block() — recall is wrapped in a per-request deadline (MEMORY_TIMEOUT_S, default 60 s as of 2026-10-23, overridable with CONDUIT_MEMORY_TIMEOUT_S; it was a hard-coded 10 s before that, which was too short for Mnemosyne's 10–45 s cold start). If the provider fails to start, the plugin logs a warning and proceeds without recall rather than abandoning the call. See incidents/2026-10-22-conduit-memory-start-timeout and systems/mnemosyne.

Maintenance commands

# what version is installed / is the tree patched?
git -C ~/.hermes/plugins/conduit_push log --oneline -1 && \
git -C ~/.hermes/plugins/conduit_push status --short

# is the gateway up, and since when? (the restart that applies a session-TTL change)
ss -tlnp | grep 9119

# session TTLs — BOTH keys, they are different consumers
python3 -c "import yaml;c=yaml.safe_load(open('/home/hermes/.hermes/config.yaml'));\
print('basic_auth',c['dashboard']['basic_auth'].get('session_ttl_seconds'),\
'| dashboard',c['dashboard'].get('session_ttl_seconds'))"

# voice transport + providers
python3 -c "import yaml;c=yaml.safe_load(open('/home/hermes/.hermes/config.yaml'));\
print(c['voice']['speech_server']['url']);print(c['tts']['provider'],c['stt']['provider'])"

# did a voice call lose its memory context?
grep -h "Conduit" ~/.hermes/logs/agent.log | grep -i "memory" | tail

# did a real call happen? (history = calls; ends = turn-ends, NOT calls)
python3 -c "import json,os;d=json.load(open(os.path.expanduser('~/.hermes/conduit-calls.json')));print(len(d['history']),'calls;',len(d['ends']),'turn-ends')"

# is the phone actually reaching the relay?
cat ~/.hermes/conduit-link-status.json

# state files (no secrets in the wiki)
ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3

2026-11-01 00:2x AWST — plugin 0.122.2, a re-shaped call store, and the Local Live fault

One night, three changes. The plugin self-updated to 0.122.2 at 18:05 on 2026-10-31 and the gateway reloaded for it at 18:06:41; the dashboard gateway came back as pid 176771 at 19:35:02; and a voice conversation opened at 19:37:09 reported that the app's "local" voice mode never applies despite a passing connection test.

  • The memory-timeout patch is resolved, not lost. The tree at 0.122.2 is clean (git status --short empty) and dashboard/api/memory.py carries MEMORY_TIMEOUT_S = 10.0 with a wait_for backstop plus tests/test_memory.py. The 2026-10-23 in-tree edit that incidents/2026-10-22-conduit-memory-start-timeout depended on is no longer required.
  • The call store changed shape. conduit-calls.json is v: 1 (outcomes / delivered / told / watches / waiting / seeded / ends / settings); history is empty (0) and last_request_at is null. ends = 2 turn-end records, both outcome: stopped, at 21:06:05 and 21:10:48. settings gained quiet_at_screen: true (#479). ⚠ The two 2026-10-29 calls the page used to cite as live history are no longer in the store.
  • New fault — local voice mode. The Mac mini speech server's LLM proxy is disabled (its LLM endpoint answers HTTP 501 "Start the server with --enable_llm_proxy"), /v1/usage reads llm_proxy.requests: 0, and its realtime socket accepts the connection but rejects session.update as an unknown event. The app's "test connection" only checks socket reachability, which is why it reports success. voice.local_live.front_model was changed during the session to {base_url: http://100.78.63.20:20128/v1, model: mistral/voxtral-small-latest} (was mistral/ministral-14b-latest); config.yaml mtime 21:02:30. Nothing was changed on the Mac. → incidents/2026-10-31-conduit-voice-local-mode-not-applying
  • Misread, corrected mid-session (keep the lesson): local_live.py:882 is the only writer of _local_live_last_request, inside the chat-completions route — so a "recent last_request_at" was the investigation's own probe, not phone traffic. last_request_at proves an HTTP request, never a call.
  • Environment: the 18:06:41 gateway reload waited the full 1,800 s cap and force-drained at 18:37:38, orphaning the Mnemosyne Health Monitor's 12:01:06 slot. → incidents/2026-10-31-gateway-reload-force-drain-orphaned-run
  • Data volume: conduit-health.sqlite3 938.2 MB (983,793,664 B) — 2,623,879 events / 4,384 buckets / 504 windows; ≈8 MB/day, no retention policy.
  • Not re-measured: per-kind event split, health-table freshness, device pairing state.

2026-11-08 00:4x AWST — self-updated again to 0.134.4, and its last step failed

The plugin updated itself in-window and its own ledger records the final step as failed:

~/.hermes/conduit-self-update.json
{"id": "916d72daeb644b06b5ae1359ddfdd992", "step": "failed", "from_version": "0.122.2",
 "version": "0.134.4", "failed_step": "restarting_gateway",
 "detail": "The plugin is updated, but the gateway didn't restart. Run hermes gateway status on the host.",
 "started_at": 1793996875.92, "finished_at": 1793997152.39, "pid": 176771, "boot_id": "f292ee44712be8b3"}
  • Window: 2026-11-07 04:27:55 → 04:32:32 AWST (276 s). from_version 0.122.2 (the version this page recorded on 2026-11-01) → 0.134.4; hermes plugins list and the plugin tree agree.
  • failed_step: restarting_gateway is literal: the gateway was not restarted at 04:32 — pid 176771 was still the live gateway at 04:59 and only restarted at 05:01:00 (systemd), then again at 05:43 as part of the Hermes 0.22.5 → 0.23.0 update (systems/hermes-agent).
  • Consequence, and the reason this is recorded here: a restart was requested at ~04:29:52 and held for the full 1800 s after-turn window, then drained — which killed two in-flight cron runs (incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain). Working hypothesis, not proven: the requester was this self-update. Either way, a plugin under ~/.hermes/plugins/ cannot restart a systemd-managed gateway itself, and it reported failure rather than retrying.
  • The plugin remained enabled and functional throughout; no user-facing conduit fault was recorded on 2026-11-07.

2026-11-08 13:2x–13:3x AWST — its pre_tool_call hook was jamming every tool call

The plugin is the only pre_tool_call registrant on the host (conduit_push/__init__.py:119), and for ~18 hours that callback overran Hermes's 30 s hook budget and was skipped:

~/.hermes/logs/errors.log
2026-11-07 14:14:03,075 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
...                                                                          (58 lines total)
2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
  • 58 timeout lines in errors.log (12 on 2026-11-07, 46 on 2026-11-08) plus 89 "skipped after previous timeout or while still running" suppression lines across the logs. The last occurrence is 2026-11-08 08:21:03 — nothing since the fix below.
  • The plugin's caching was NOT the fault — this was measured, not assumed. session_titles.py:21 CACHE_SECONDS = 30.0; timed in-process, a hook call costs 3.1 s cold and 0.00 s warm. The "re-opens a 1.87 GB state.db per call, ~9 s each" theory was wrong. The timeouts track load spikes on the 2-core CT460 (load average 5.89 at 13:27:50; state.db with a 40 MB WAL checkpointing on the same spindle).
  • Fix: plugins.hook_callback_timeout 30 → 120 in ~/.hermes/config.yaml (2026-11-08 13:35 AWST) — verified live in config, and verified effective (no timeout logged after 08:21:03).
  • Cost of the fault: Mnemosyne Health Monitor ran 8 h 27 m and 5 h 59 m 51 s on its two 2026-11-08 slots; wiki-stocktake-maintenance failed at 04:36:08 on "No reply: the turn stopped while a tool result was still pending"; FreshRSS Midday took 2 h 09 m 53 s. → incidents/2026-11-08-pre-tool-call-hook-timeout-cascade

2026-11-15 — 474 registrations dropped across eight failed loads

errors.log for 2026-11-15 holds 9 × Failed to load plugin 'conduit_push': load timed out after 10s (import + register() never returned) and 474 × Plugin 'conduit_push' called register_{tool,hook,skill}() after its load timed out; ignored, grouped into eight load events (01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47). The loader discards late registrations, so for each of those processes Conduit's tools, skills and hooks were absent — a silent loss the caller never sees as an error. That same day the pre_tool_call hook this plugin registers timed out 11× (10 tool calls refused). Zero such lines on 11-16 and 11-17. Full counts: incidents/2026-11-15-plugin-load-timeout-storm; the hook side: incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.

Open items

  • Hook budget is 120 s, not 30 s — watch it. A second overrun would eat two minutes per tool call. The binding constraint is state.db (1.89 GB, 2,901 sessions, live 40 MB WAL), not this plugin. → incidents/2026-11-08-pre-tool-call-hook-timeout-cascade
  • ✅ Resolved 2026-11-01 — the plugin_api.py timeout patch is no longer needed: the tree is clean at 0.122.2 and MEMORY_TIMEOUT_S ships upstream in dashboard/api/memory.py.
  • Verify a call end-to-end from the phone. The 2026-10-29 08:49:36 call has no recorded outcome, so MEMORY_TIMEOUT_S = 60 is still unexercised by a confirmed call, and last_request_at was still null → incidents/2026-10-29-conduit-voice-silence.
  • Local voice mode does not work — see incidents/2026-10-31-conduit-voice-local-mode-not-applying. The Mac mini speech server's LLM proxy is disabled (LLM endpoint → 501, /v1/usage shows llm_proxy.requests: 0) and its realtime socket rejects session.update as an unknown event, while the app's "test connection" checks socket reachability only. Nothing changed on the Mac yet.
  • voice.mode is a phone-side selector: neither config.yaml → voice nor conduit-settings.json carries a mode key, so a "mode change did not apply" cannot be diagnosed on the host config.
  • Re-check voice.speech_server.url after any Conduit/app update or settings sync — a bare /v1/ silently 403s every call.
  • No retention policy on conduit-health.sqlite3 (938.2 MB, ≈8 MB/day, growing unbounded).
  • The Mnemosyne cold start that motivated the patch is still slow (10–45 s); only the budget was raised. → systems/mnemosyne