wiki(audit): 2026-10-29 window — Conduit voice fault documented and repaired; radar two-stage split resolved both incidents (20 pages, 2 letters, Notion 455->475); cron 176/0

This commit is contained in:
Tony0410
2026-10-30 00:21:21 +08:00
parent cc95dcc54d
commit 1d335ca8e6
10 changed files with 404 additions and 82 deletions
+95 -50
View File
@@ -3,12 +3,12 @@ title: Conduit
type: system
status: active
created: 2026-10-24
updated: 2026-10-28
verified_on: 2026-10-28
updated: 2026-10-30
verified_on: 2026-10-30
review_after: 2026-11-08
confidence: high
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling]
sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329]
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl]
sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329, session-20260908_031541_49c051]
---
# Conduit
@@ -21,11 +21,18 @@ be started from the phone where the agent can speak and listen (`conduit-voice`
Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[systems/hermes-ui]]
(the browser desktop), [[systems/push-notifications]] (the older `hermes-push` bridge).
## Runtime checks (2026-10-26 00:1x AWST)
- **Working tree:** M dashboard/plugin_api.py — the uncommitted memory-timeout patch is **still present**.
- **`conduit-calls.json` → `history` entries:** **0** — the call-request path remains unused
→ [[incidents/2026-10-24-conduit-call-request-refused]].
- **Tool use in-window:** `conduit_show` ×1 — the plugin works for cards; the open incident concerns *placing calls*.
## Runtime checks (2026-10-30 00:0x AWST)
- **`conduit-calls.json` → `history` entries: 2** — the first real calls ever recorded
(2026-10-29 08:38:04 and 08:49:36). The path is no longer unused; the earlier
*"0 calls"* reading is superseded. See [[incidents/2026-10-29-conduit-voice-silence]].
- **Calls still unproven end-to-end:** `last_request_at` remained `null` at the end of the
2026-10-29 window — no call is yet known to have run a real Hermes turn.
- **`ends` vs `history`:** `ends` records *any turn-end that fires a watch* (including cron
sessions) — never read it as a call log.
- **Device:** iPhone app version now **2.2.0** (was 1.10.1 on 2026-10-28), **19 settings keys**,
**no `voice.local.*` keys** → Local Live cannot be configured host-side any more.
- **Voice providers were both dead and are now fixed** — `tts.provider: ollama_local`,
`stt.provider: local` (2026-10-29). Details below and in the 10-29 incident.
## Where it lives
- **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push`
@@ -46,36 +53,42 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste
- **Data stores** (`~/.hermes/`): `conduit-health.sqlite3`, `conduit-location.sqlite3`,
`conduit-phone.sqlite3`, plus JSON state — `conduit-push.json` (credential / gateway id /
installation id / relay url / e2e — **no values recorded here**), `conduit-calls.json`,
`conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`.
`conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`,
`conduit-link-status.json`, `conduit-local-live.json` (**holds an api key — never copy its
contents into the wiki**).
## Current state (verified 2026-10-25 00:2x–00:5x AWST)
- **Dashboard gateway (re-verified 2026-10-26):** listening on `0.0.0.0:9119`; process started **2026-10-25
18:42:33** — **restarted since the 2026-10-24 entry, so the upstream commits are now live** (the recorded
*"not restarted since / not live"* note is superseded, not wrong at the time) — so it is still the process that picked up the memory-timeout patch, and
the `0.64.0` → `0.69.0` upstream commits are **not** live.
- **Device:** one registered — **iPhone** (`id 70aa63f3…`, `kid c24e8215…`), app version `2.0.0`,
features `chat-cards`, `info-cards`, `live-cards`, `maps-navigate`, `media-cards`, `show-cards`;
tz `Australia/Perth`; sharing window 2026-10-16 → 2027-04-22. Paired **2026-10-22 12:54:39**.
`last_seen` / `last_sent` / `last_contact` all read **2026-10-23 19:10:30 AWST** (~29 h stale at
this audit) while health rows kept arriving until **2026-10-24 09:59:59** and window rollups were
recomputed at **13:56:35** — so those three fields track *phone-status* contact, not data flow.
(The 10-24 page recorded 19:02:16 from the same store; the epoch reads 19:10:35.)
## Current state (re-verified 2026-10-30 00:0x AWST)
- **Dashboard gateway:** listening on `0.0.0.0:9119`; **pid 4042003, started 2026-10-29 08:03:26**
— the restart that put the 30-day session TTL live (see below). The `0.64.0` → `0.69.0` upstream
commits remain **not** live (the process predates nothing relevant; the tree still only carries the
memory-timeout patch).
- **Device:** one registered — **iPhone**, app version **`2.2.0`** (self-updated from `1.10.1` on
2026-10-29), `seen_at` **2026-10-29 18:56:44 AWST**, `mode: small`, **19 settings keys**.
`last_seen` / `last_sent` / `last_contact` are `null` in the current store — those three fields
track *phone-status* contact, not data flow (health rows and link pings keep arriving regardless:
`conduit-link-status.json` `app_seen_at` **2026-10-29 23:53:15**, `push_ok_at` **18:50:00**).
Paired **2026-10-22 12:54:39**; sharing window 2026-10-16 → 2027-04-22.
- **Dashboard session TTL (raised 2026-10-29):** `dashboard.basic_auth.session_ttl_seconds` =
**2592000 (30 days)**, up from the **43200 (12 h)** default. A lapsed session was making Conduit's
call socket answer **WS 4401 "Dashboard sign-in required"** — silently. ⚠ `dashboard.session_ttl_seconds`
is a **separate key and still 43200**; check both.
- **Call settings:** `enabled`, `when_asked`, `decides`, `alerts` all true; rate limits
`min_gap_s 120` / `per_hour 6` / `per_day 20`; `avoid_unknown_place` false.
- **Recent calls — none.** `conduit-calls.json → history` is **empty**: no call has been recorded
since the memory-timeout fix landed, so that fix still has **no positive evidence** behind it. The
“two call sessions closed cleanly 00:03:23 / 00:06:57” claim previously on this page was a **false
positive** — those `ends` entries are the two **cron** sessions of that night, and `ends` records
any turn-end that fires a watch, not calls →
[[incidents/2026-10-24-conduit-call-request-refused]].
- **Data volume (2026-10-25 00:2x):** `conduit-health.sqlite3` **569.9 MB** (+8.2 MB in 25.7 h),
holding **1,520,229 `events`** (1,510,929 `sample` / 6,070 `sleep` / 3,230 `workout`),
**3,028 `buckets`** and **124 `windows`** for its single device; newest row **2026-10-24 09:59:59**,
newest `buckets.date` 2026-10-24; `conduit-location.sqlite3` 80 KB; `conduit-phone.sqlite3` 284 KB.
- **Recent calls — two, both 2026-10-29.** `conduit-calls.json → history` now reads **2**
(08:38:04, 08:49:36). The 08:38:04 call produced no Mac connection and no Hermes turn, and the
iPhone re-authenticated at 08:39:05 — the expired-session fault. The 08:49:36 call was placed after
the `classic` mode switch and the STT/TTS fixes, but the session ended one second earlier, so its
outcome is **unrecorded**. The memory-timeout patch therefore still has no *confirmed* call behind
it → [[incidents/2026-10-29-conduit-voice-silence]].
- **Data volume (2026-10-30 00:0x):** `conduit-health.sqlite3` **930.0 MB** (+360 MB in 5 days;
+8.2 MB/25.7 h as measured on 10-25 — the rate has not held, so treat growth as **ingestion-paced,
unbounded**), holding **2,601,565 `events`** (2,592,189 `sample` / 6,146 `sleep` / 3,230 `workout`),
**4,105 `buckets`** and **411 `windows`**; newest health row **2026-10-29 23:12:01 AWST**.
`conduit-location.sqlite3` 80 KB; `conduit-phone.sqlite3` 284 KB (last measured 10-25).
⚠ The 10-24 page recorded **308 `buckets` / 62 `windows`**; a direct `COUNT(*)` reads
**3,028 / 124**, so the earlier pair was a narrower grouping — quote these. **No retention or
compaction has been observed** — the store grows from ingestion alone and is worth watching against
the CT460 disk budget.
**4,105 / 411**, so the earlier pair was a narrower grouping — quote these. **No retention or
compaction has been observed**, and the store is now closing on 1 GB — watch it against the CT460
disk budget.
## Calls from Hermes (the `conduit_call_user` tool)
The plugin registers a **`conduit_call_user`** tool (toolset `conduit`, 📞) for every session and
@@ -88,20 +101,36 @@ the call rings when the asking turn ends and plays the reason first. Limits: `mi
⚠ **Known failure, 2026-10-24:** three requests to be phoned were refused by the agent, which
asserted the capability did not exist **without calling the tool or loading the skill** →
[[incidents/2026-10-24-conduit-call-request-refused]].
[[incidents/2026-10-24-conduit-call-request-refused]]. **Positive counter-example 2026-10-29:** in
that window the agent did look the tool up and did place calls (two entries in `history`), so the
failures were behavioural, not a missing capability.
## Voice pipeline (probed 2026-10-28)
## Voice pipeline (verified 2026-10-30)
Phone voice sessions ride the Mac mini speech server, which is also the Hermes TTS target:
Phone voice sessions ride the Mac mini speech server, which is also the Hermes TTS/STT target.
**All three settings below were wrong or dead as of 2026-10-28 and were fixed on 2026-10-29** —
see [[incidents/2026-10-29-conduit-voice-silence]].
- `voice.speech_server.url` = `wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/` — live-voice transport,
`wake_word.enabled: true`.
- `tts.provider: openai` → `https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech` (model `phonon`,
voice `alloy`) — see [[systems/hermes-agent]].
- **Health check:** `GET https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/pool` returned
`{"size": 2, "in_use": 0}` with both units `idle` on 2026-10-28 00:2x — a **2-unit pool**. `/v1/models` is
**not implemented** (`{"detail":"Not Found"}`); probe `/v1/pool`, not `/v1/models`.
| Setting | Value (now) | Was | Failure when wrong |
|---|---|---|---|
| `voice.speech_server.url` | `wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime` | `…:8443/v1/` | **403** — the plugin dials the value **verbatim**, it does not append `/realtime` |
| `tts.provider` | `ollama_local` (`~/.hermes/scripts/ollama-tts.sh`, model `phonon`) | `openai` → Mac `/v1/speech` | **502** |
| `stt.provider` | `local` (`faster_whisper`, model `base`) | `groq` | **403 / Cloudflare error 1010** — a *block*, not a bad key (fails identically with a bogus key) |
- `voice.mode` = **`classic`** (switched from `local` 2026-10-29). Classic relays everything through
the plugin and needs **no phone-side server address** — which is why both host providers above had
to work. A Local Live path cannot be configured from the host at all: app **2.2.0**'s schema exposes
**19 keys, none of them `voice.local.*`**.
- `voice.auto_tts: false` — assistant replies are not auto-spoken; audio appears only when requested.
- `wake_word.enabled: true`.
- **Health check:** `GET https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/pool` returned
`{"size": 2, "in_use": 0}` with both units `idle` on 2026-10-28 00:2x — a **2-unit pool**. `/v1/models`
is **not implemented** (`{"detail":"Not Found"}`); probe `/v1/pool`, not `/v1/models`.
- **End-to-end proof (2026-10-29):** driving the real call path produced `session.created` →
transcription of live speech → `response.done`, with the Mac counters moving **0 → 10 connections
/ 1 completed response / 2.98 s audio / 0 errors** and the plugin handshaking in **41 ms**.
- ⚠ **Fragile URL:** the Conduit settings tool writes Hermes config (it had flipped `tts.provider` to
`openai`). If it or an app update ever writes the bare `/v1/` back, every call silently 403s again.
## Memory integration
The voice session (`conduit-voice`) reaches Mnemosyne through the plugin's own memory layer, not
@@ -118,15 +147,27 @@ See [[incidents/2026-10-22-conduit-memory-start-timeout]] and [[systems/mnemosyn
git -C ~/.hermes/plugins/conduit_push log --oneline -1 && \
git -C ~/.hermes/plugins/conduit_push status --short
# is the gateway up, and since when?
# is the gateway up, and since when? (the restart that applies a session-TTL change)
ss -tlnp | grep 9119
# session TTLs — BOTH keys, they are different consumers
python3 -c "import yaml;c=yaml.safe_load(open('/home/hermes/.hermes/config.yaml'));\
print('basic_auth',c['dashboard']['basic_auth'].get('session_ttl_seconds'),\
'| dashboard',c['dashboard'].get('session_ttl_seconds'))"
# voice transport + providers
python3 -c "import yaml;c=yaml.safe_load(open('/home/hermes/.hermes/config.yaml'));\
print(c['voice']['speech_server']['url']);print(c['tts']['provider'],c['stt']['provider'])"
# did a voice call lose its memory context?
grep -h "Conduit" ~/.hermes/logs/agent.log | grep -i "memory" | tail
# did a real call happen? (history = calls; ends = turn-ends, NOT calls)
python3 -c "import json,os;d=json.load(open(os.path.expanduser('~/.hermes/conduit-calls.json')));print(len(d['history']),'calls;',len(d['ends']),'turn-ends')"
# is the phone actually reaching the relay?
cat ~/.hermes/conduit-link-status.json
# state files (no secrets in the wiki)
ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3
```
@@ -134,16 +175,20 @@ ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3
## Open items
- Commit or upstream the `plugin_api.py` timeout patch so a plugin update cannot revert it — it is
now five releases behind HEAD.
- **Verify a call end-to-end.** No call has been placed since the fix, so `MEMORY_TIMEOUT_S = 60` is
still unexercised by a real call, and the 10-24 refusal means the chat route has never worked →
[[incidents/2026-10-24-conduit-call-request-refused]].
- No retention policy on `conduit-health.sqlite3` (569.9 MB and growing).
- **Verify a call end-to-end from the phone.** The 2026-10-29 08:49:36 call has no recorded outcome,
so `MEMORY_TIMEOUT_S = 60` is still unexercised by a *confirmed* call, and `last_request_at` was
still `null` → [[incidents/2026-10-29-conduit-voice-silence]].
- Re-check `voice.speech_server.url` after any Conduit/app update or settings sync — a bare `/v1/`
silently 403s every call.
- No retention policy on `conduit-health.sqlite3` (**930 MB** and growing unbounded).
- The Mnemosyne cold start that motivated the patch is still slow (10–45 s); only the *budget* was
raised. → [[systems/mnemosyne]]
## Related
- [[incidents/2026-10-29-conduit-voice-silence]]
- [[incidents/2026-10-24-conduit-call-request-refused]]
- [[incidents/2026-10-22-conduit-memory-start-timeout]]
- [[systems/hermes-agent]]
- [[systems/mnemosyne]]
- [[systems/scheduled-tasks]]
- [[current-state]]
+34 -2
View File
@@ -3,8 +3,8 @@ title: Hermes Agent
type: system
status: active
created: 2026-07-22
updated: 2026-10-28
verified_on: 2026-10-28
updated: 2026-10-30
verified_on: 2026-10-30
confidence: high
tags: [system, hermes, agent, llm]
sources: [live-hermes-version, live-config, live-skill-inventory]
@@ -166,6 +166,10 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
- **TTS:** ElevenLabs is the configured provider; its package was missing from the live managed env
and was reinstalled 2026-10-10. `voice.auto_tts: false` - auto-speak is off by design, so TTS
output appears only when explicitly requested.
- ⚠ **Superseded twice — live config checked 2026-10-30:** `tts.provider` is now **`ollama_local`**
(`~/.hermes/scripts/ollama-tts.sh`, model `phonon`) and `stt.provider` is now **`local`** (faster-whisper
`base`); `voice.speech_server.url` is now `wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime`
(the bare `/v1/` answers **403**, and the plugin dials the value verbatim). See the 2026-10-30 section below.
- ⚠ **Superseded — live config checked 2026-10-28** (change date not established): `tts.provider` is
**`openai`** pointed at the **Mac mini speech server** — `tts.providers.openai.base_url` =
`https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech`, `model: phonon`, `voice: alloy`. Live voice
@@ -179,6 +183,34 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
cadence - [[systems/messaging-integrations]].
- **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]].
## 2026-10-30 00:0x AWST — the Conduit voice path: a truncated URL, a 12-hour session TTL, and two dead providers
Anthony asked why Conduit voice calls never got an answer. **Three faults sat in series** — each one masked the next —
and all three were fixed on 2026-10-29 between 08:02 and 08:4x AWST:
| Fault | Was | Symptom | Now |
|---|---|---|---|
| `voice.speech_server.url` | `wss://…:8443/v1/` | dialled **verbatim** → HTTP **403**, reported as *"the speech server refused the key"* | `wss://…:8443/v1/realtime` |
| `dashboard.basic_auth.session_ttl_seconds` | `43200` (12 h default) | expired phone session → **WS 4401 "Dashboard sign-in required"**, no Hermes turn, no error on the device | `2592000` (30 d, matched to the refresh-token life) |
| `tts.provider` / `stt.provider` | `openai` / `groq` | Mac `/v1/speech` **502** / **403 Cloudflare 1010** | `ollama_local` / `local` |
- **Why it stayed invisible:** the Mac recorded **zero connections**, `conduit-calls.json → history` was empty, and each
attempt logged a reason string pointing at the *next* fault — so one broken chain looked like three unrelated problems.
`last_request_at` was `null`: no call had ever reached Hermes' turn logic.
- **The 12 h TTL hid behind the others.** The dashboard had stopped issuing long-enough sessions, and the iPhone only
re-authenticated when the dashboard happened to prompt it — an audit-log entry shows a re-auth at **08:00:59**, minutes
*after* a silent call.
- **Two TTL keys exist.** `dashboard.basic_auth.session_ttl_seconds` was raised; the similarly named
`dashboard.session_ttl_seconds` **still reads 43200**. Check **both** on any future 4401.
- **Groq was blocked, not mis-keyed** — it answers 403/1010 with a bogus key too, so this was Cloudflare bot-blocking
rather than a credential problem.
- **Verified for real:** the whole path was driven (mint HMAC dashboard session → WS ticket → let the plugin dial the Mac),
then two live calls landed at **2026-10-29 08:38:04** and **08:49:36** — the first entries ever in
`conduit-calls.json → history`. `ollama_local` produced **6.94 s** of real 64 kbps speech; `local` STT returned
`success: True`.
- Backups: `config.yaml.bak.url-081525`, `config.yaml.bak.ttl-20261029080258`, `config.yaml.bak.stt-084536`; the dashboard
restarted **08:03:26** → [[incidents/2026-10-29-conduit-voice-silence]] · [[systems/conduit]].
## 2026-10-26 00:1x AWST — gateway restart record and the dead Nous credential
- **Three gateway starts on 2026-10-25**: `00:32:20`, `18:30:04`, `19:27:13` AWST (after `2026-10-24 12:24:39`).
+12 -6
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-29
verified_on: 2026-10-29
review_after: 2026-11-03
updated: 2026-10-30
verified_on: 2026-10-30
review_after: 2026-11-06
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks
## Count
- **Current as of 2026-10-29 00:3x AWST (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real and the discovery stage still could not finish: stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.**
- **Current as of 2026-10-30 00:0x AWST (live `jobs.json` + `executions.db`) — the radar's two-stage split worked end to end for the first time, and the fleet had its first clean day in a week.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3**, and every `failure_streak` reads **0** — the three survivors are all stale re-runs of long-standing rows (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`). Window **2026-10-29 settled 176 `completed` / 0 `failed`** (+ this audit `running`) — the first zero-failure day since 2026-10-24; 2026-10-28 stands at **168 / 2**. **Unclosed incident rows 4 → 1:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving only the Facebook monthly's `0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report). `dca8482e4f76` (stage 1) ran **07:00:59 → 10:16:12** (**3 h 15 m 13 s**), closed **`completed`**, and wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok` — opening **no** new incident row (contrast the two consecutive stage-1 failures on 10-27 and 10-28). `5fc292ff0f3b` (stage 2) fired **10:13:46**, consumed the handoff **10:40:59**, closed **10:44:48 `completed`** and delivered the same second: **20 pages / 20 `page_ids` / 2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went **455 → 475**. Both later slots no-oped as designed (**11:30 → 10 m 46 s**, **14:30 → 1 m 51 s**). Stage 1 correctly no longer delivers. See [[incidents/2026-10-27-job-radar-budget-exhausted]] and [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], now both **resolved**. **Pins unchanged** — 19 of the 26 enabled jobs pin `custom:omni-101` (12 `smart-route`, 6 `free-stack`, 1 explicit).
- **Prior window — as of 2026-10-29 00:3x AWST *(superseded by the 2026-10-30 entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real, but stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.**
- **Prior window — as of 2026-10-28 00:2x AWST** (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. *(superseded by the 2026-10-29 entry above)*
- **Prior window — as of 2026-10-27 00:1x AWST *(superseded by the 2026-10-28 entry above)* (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`).
- **Current as of 2026-10-26 00:1x AWST** *(superseded by the 2026-10-27 entry above)* **(live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`).
@@ -51,8 +52,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) |
| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 |
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00 |
| Daily Marketing Job Radar — Write and Letters (stage 2) | `5fc292ff0f3b` | `30 11,14 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Created 2026-10-27 17:34** — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). `custom:omni-101` / `smart-route`; first fires 2026-10-27 17:35 and 17:38 both `completed`. See [[incidents/2026-10-27-job-radar-budget-exhausted]] |
| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | `local` (failures → topic 1030) | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00. ✅ **First full split run 2026-10-29:** 07:00:59 → 10:16:12 (3 h 15 m 13 s), `completed`, wrote `handoff/2026-10-29.json` (170 discovered / 33 duplicates avoided / 37 verified); both of its ledger rows closed `resolved` at 10:16:12. **Delivery is now `local`** — stage 2 owns the report |
| Daily Marketing Job Radar — Write and Letters (stage 2) | `5fc292ff0f3b` | `30 11,14 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Created 2026-10-27 17:34** — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). `custom:omni-101` / `smart-route`; first fires 2026-10-27 17:35 and 17:38 both `completed`. ✅ **First full split run 2026-10-29:** fired 10:13:46, consumed the handoff at 10:40:59, closed `completed` 10:44:48 and delivered the same second — **20 pages + 2 cover letters, `missing_cover_letter_count 0`**, Notion board 455 → 475; the 11:30 and 14:30 slots then no-oped correctly. See [[incidents/2026-10-27-job-radar-budget-exhausted]] |
| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant |
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] |
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
@@ -97,6 +98,11 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
## Known Issues
### Resolved 2026-10-29
- **The Job Radar's two-stage split finished end to end for the first time (`dca8482e4f76` + `5fc292ff0f3b`):** stage 1 ran **07:00:59 → 10:16:12** (3 h 15 m 13 s), closed **`completed`** and wrote `handoff/2026-10-29.json` (**170 discovered / 33 duplicates avoided / 37 verified**) — the first post-split run to write a handoff, after 10-27 died on the **tool-call budget** and 10-28 on the **iteration cap**. Stage 2 consumed it at **10:40:59**, closed `completed` at **10:44:48** and **delivered** to topic 1030 in the same second: **20 pages, 2 cover letters, `missing_cover_letter_count 0`**, Notion **455 → 475**. Both stage-1 ledger rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12**, and the run opened no new row → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], [[runbooks/job-radar]]. ⚠ The discovery stage is still **slow** (3 h 15 m) and both ceilings remain reachable — a `completed` stage-1 row with **no handoff file** is the same fault in new clothing.
- **Conduit voice calls — three faults in series, all fixed and verified by two real calls:** the trunk speech-server URL was truncated (`…/v1/` → **403**, blamed on the key), the dashboard session TTL was the **12 h default** (expired sessions answer **WS 4401** with no error on the phone), and **both** host providers were dead (`tts.provider: openai` → Mac `/v1/speech` **502**; `stt.provider: groq` → **403/Cloudflare 1010**). Repaired 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL → **2 592 000 s**, `stt.provider: local`, `tts.provider: ollama_local`; dashboard restarted **08:03:26**. The path was driven end to end, then **calls at 08:38:04 and 08:49:36** — the first entries in `conduit-calls.json → history`. ⚠ Two TTL keys exist and only one was raised → [[incidents/2026-10-29-conduit-voice-silence]], [[systems/conduit]].
### Resolved 2026-10-12
- **Both `deepseek-v4-flash` incidents closed by a clean scheduled run (`fad5a86979d4`, 03:00):** the Facebook Birthday Weekly Registry Sync completed and delivered on 2026-10-12 (03:00:45 → 03:05:47), and incidents `fad5a8_a13bbc837e1c` (open since 2026-09-28, every fallback provider exhausted) and `fad5a8_c80b59a907f0` (since 2026-09-30, `deepseek-v4-flash` HTTP 400) **both auto-closed at 03:05:39**. Its stale pre-repin `last_status: error` cleared with them, after carrying it for two weeks — the repin is now verified by a real scheduled run, not only by the manual one.
- **`dca8482e4f76` timeout incident closed (`dca848_23fcd2b18aaa`):** the Daily Marketing Job Radar ran on schedule 2026-10-12 07:00:47 → 07:33:50 and delivered to topic 1030; the 2026-10-09 timeout incident that belonged to the provider storm **auto-closed at 07:33:41**. Of the ten storm incidents, nine have now closed and **one remains**: `1b23c0_c35703f637b8` (`1b23c042e17d`, still `alerted`, next slot Fri 2026-10-16 08:00). See [[incidents/2026-10-09-cron-provider-timeout-storm]].