wiki(audit): 2026-10-29 window — Conduit voice fault documented and repaired; radar two-stage split resolved both incidents (20 pages, 2 letters, Notion 455->475); cron 176/0
This commit is contained in:
+35
-4
@@ -3,16 +3,47 @@ title: Current State
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-29
|
updated: 2026-10-30
|
||||||
verified_on: 2026-10-29
|
verified_on: 2026-10-30
|
||||||
review_after: 2026-11-03
|
review_after: 2026-11-07
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system]
|
tags: [system]
|
||||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
|
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
|
||||||
---
|
---
|
||||||
|
|
||||||
# Current State
|
# Current State
|
||||||
Updated: 2026-10-29
|
Updated: 2026-10-30
|
||||||
|
|
||||||
|
## Recent Changes (2026-10-30, nightly audit; live probes 2026-10-30 00:0x AWST)
|
||||||
|
|
||||||
|
- **Conduit voice calls were repaired — after three faults in series — and the first two real calls in the
|
||||||
|
product's history are now on record.** Anthony asked why Conduit never answered when he called. In order:
|
||||||
|
(A) `voice.speech_server.url` held the truncated `wss://…:8443/v1/`, which the plugin dials **verbatim**, so
|
||||||
|
the Mac answered **403** and the plugin blamed the key; (B) `dashboard.basic_auth.session_ttl_seconds` was the
|
||||||
|
**12-hour default**, and an expired phone session gets **WS 4401** with no Hermes turn and no error visible on
|
||||||
|
the phone (the iPhone re-authenticated at **08:00:59**, minutes *after* a silent call); (C) **both** host
|
||||||
|
providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq` → **403 /
|
||||||
|
Cloudflare 1010**. Fixed 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL → **2 592 000 s (30 d)**, `stt.provider:
|
||||||
|
local` (faster-whisper `base`), `tts.provider: ollama_local` (6.94 s of real speech verified); dashboard
|
||||||
|
restarted **08:03:26**. Verified end to end by driving the whole call path, then by **calls at 08:38:04 and
|
||||||
|
08:49:36** — `conduit-calls.json → history` is no longer empty.
|
||||||
|
→ [[incidents/2026-10-29-conduit-voice-silence]], [[systems/conduit]], [[systems/hermes-agent]]
|
||||||
|
- **The Job Radar's two-stage split worked end to end for the first time.** Stage 1 (`dca8482e4f76`,
|
||||||
|
`deliver: local`) ran **07:00:59 → 10:16:12** (3 h 15 m 13 s), closed **`completed`**, and wrote
|
||||||
|
`~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**,
|
||||||
|
sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Stage 2 (`5fc292ff0f3b`) consumed the
|
||||||
|
handoff at **10:40:59**, closed **10:44:48 `completed`** and delivered to topic 1030 in the same second:
|
||||||
|
**20 pages / 20 `page_ids` / 2 cover letters / `verified_cover_letter_count 2` /
|
||||||
|
`missing_cover_letter_count 0`**, Notion board **455 → 475**. Both of stage 1's incident rows
|
||||||
|
(`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12**; the 11:30 and 14:30 slots
|
||||||
|
then no-oped as designed (10 m 46 s / 1 m 51 s).
|
||||||
|
→ [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]],
|
||||||
|
[[runbooks/job-radar]]
|
||||||
|
- **Cron: 2026-10-29 settled 176 `completed` / 0 `failed`** — the first zero-failure day since 2026-10-24 — with
|
||||||
|
**10 deliveries, all 10 `delivered`, 0 failures**. Non-clean `last_status` **5 → 3**; every `failure_streak`
|
||||||
|
reads **0**; unclosed incident rows **4 → 1**. Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**, pins
|
||||||
|
unchanged (19 of the 26 enabled jobs on `custom:omni-101`: 12 `smart-route`, 6 `free-stack`, 1 explicit; 7
|
||||||
|
unpinned on the global default).
|
||||||
|
|
||||||
## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 00:2x AWST)
|
## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 00:2x AWST)
|
||||||
|
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
---
|
---
|
||||||
title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response
|
title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response
|
||||||
type: incident
|
type: incident
|
||||||
status: active
|
status: resolved
|
||||||
created: 2026-10-28
|
created: 2026-10-28
|
||||||
updated: 2026-10-29
|
updated: 2026-10-30
|
||||||
verified_on: 2026-10-29
|
verified_on: 2026-10-30
|
||||||
review_after: 2026-11-04
|
review_after: 2026-11-13
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
||||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api]
|
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api]
|
||||||
@@ -101,6 +101,26 @@ The **first real two-stage cycle ran on 2026-10-28** and answered the open quest
|
|||||||
- **Open question 4 is answered: the row did not close.** `dca848_e93a538ade6b` stayed open because the 10-28
|
- **Open question 4 is answered: the row did not close.** `dca848_e93a538ade6b` stayed open because the 10-28
|
||||||
slot was not clean, and a second row (`dca848_0a2f03b5952b`) opened — open incident rows **2 → 4**.
|
slot was not clean, and a second row (`dca848_0a2f03b5952b`) opened — open incident rows **2 → 4**.
|
||||||
|
|
||||||
|
## Follow-up — the 2026-10-29 slot: the split held (recorded 2026-10-30)
|
||||||
|
|
||||||
|
**Resolved.** Stage 1 (`dca8482e4f76`) ran **2026-10-29 07:00:59 → 10:16:12** — 3 h 15 m 13 s — and
|
||||||
|
closed **`completed`**. It wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` (24.8 KB):
|
||||||
|
`run_date 2026-10-29T08:23:51+08:00`, **170 discovered / 33 duplicates avoided / 37 verified**,
|
||||||
|
sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Both ledger rows this incident opened —
|
||||||
|
`dca848_e93a538ade6b` and `dca848_0a2f03b5952b` — closed **`resolved` at 10:16:12**, and the run
|
||||||
|
opened no new row. Open cron incident rows went **4 → 1**; the survivor is the Facebook monthly's
|
||||||
|
`0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot.
|
||||||
|
|
||||||
|
Stage 2 (`5fc292ff0f3b`) fired at **10:13:46**, consumed the handoff at **10:40:59**, finished
|
||||||
|
**10:44:48 `completed`** and **delivered** to topic 1030 in the same second (delivery `delivered`).
|
||||||
|
Its `stage2_result`: **20 pages written / 20 `page_ids` / 2 cover letters / 2 same-day strong matches
|
||||||
|
/ `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went
|
||||||
|
**455 → 475**. The later slots no-oped as designed (11:30 → 10 m 46 s, 14:30 → 1 m 51 s, already-consumed
|
||||||
|
handoff). Stage 1 itself no longer delivers — correct, because stage 2 owns the report.
|
||||||
|
|
||||||
|
The questions this page left open are answered for this run: the handoff **was** written inside budget and
|
||||||
|
consumed, and **no cover letter went missing on the first real run of the split**.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun
|
- [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
---
|
---
|
||||||
title: Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped
|
title: Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped
|
||||||
type: incident
|
type: incident
|
||||||
status: active
|
status: resolved
|
||||||
created: 2026-10-29
|
created: 2026-10-29
|
||||||
updated: 2026-10-29
|
updated: 2026-10-30
|
||||||
verified_on: 2026-10-29
|
verified_on: 2026-10-30
|
||||||
review_after: 2026-11-05
|
review_after: 2026-11-13
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
||||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir]
|
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir]
|
||||||
@@ -76,6 +76,21 @@ slot was not clean. Open `cron_incidents` rows went **2 → 4**.
|
|||||||
4. Both radar incident rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) close only when a **clean** stage-1
|
4. Both radar incident rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) close only when a **clean** stage-1
|
||||||
slot runs — watch the next 07:00 weekday slot.
|
slot runs — watch the next 07:00 weekday slot.
|
||||||
|
|
||||||
|
## Follow-up — the 2026-10-29 slot: the cap was not hit (recorded 2026-10-30)
|
||||||
|
|
||||||
|
**Resolved for this run.** The next stage-1 run (`dca8482e4f76`, **2026-10-29 07:00:59 → 10:16:12**,
|
||||||
|
3 h 15 m 13 s) closed **`completed`** *and* wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — the
|
||||||
|
first run of the split to do both. The handoff records **170 discovered / 33 duplicates avoided /
|
||||||
|
37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`,
|
||||||
|
`indeed blocked`, `wa_gov ok`. Stage 2 consumed it at **10:40:59**, delivered at **10:44:48** with
|
||||||
|
**20 Notion pages and 2 cover letters** (`missing_cover_letter_count 0`; board 455 → 475). Its two
|
||||||
|
ledger rows closed `resolved` at 10:16:12.
|
||||||
|
|
||||||
|
⚠ **Flaky, not solved.** A 3 h 15 m stage-1 run is still long enough for the cap to be reachable, and
|
||||||
|
the previous two slots in the split both died on it (2026-10-27 on the tool-call budget, 2026-10-28 on
|
||||||
|
the iteration cap). Treat a `completed` run with **no handoff file** as the same fault, whatever the
|
||||||
|
execution status says → [[incidents/2026-10-27-job-radar-budget-exhausted]].
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up
|
- [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up
|
||||||
|
|||||||
@@ -0,0 +1,106 @@
|
|||||||
|
---
|
||||||
|
title: Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hour Session TTL, Dead STT/TTS Providers
|
||||||
|
type: incident
|
||||||
|
status: resolved
|
||||||
|
created: 2026-10-30
|
||||||
|
updated: 2026-10-30
|
||||||
|
verified_on: 2026-10-30
|
||||||
|
confidence: high
|
||||||
|
tags: [conduit, voice, tts, stt, config, session-ttl, incident]
|
||||||
|
sources:
|
||||||
|
- live-probe-2026-10-30
|
||||||
|
- live-config-2026-10-30
|
||||||
|
- live-log-2026-10-30
|
||||||
|
- session-20260908_031541_49c051
|
||||||
|
- plugin-tree-conduit_push
|
||||||
|
---
|
||||||
|
|
||||||
|
# Conduit Voice Calls Silently Died — Truncated Speech-Server URL, 12-Hour Session TTL, Dead STT/TTS Providers
|
||||||
|
|
||||||
|
**Window:** 2026-10-29 07:54 – 09:0x AWST · **Fixed:** 2026-10-29 08:02–08:4x AWST (dashboard restarted 08:03:26)
|
||||||
|
**Symptom:** Anthony asked *"why do you never answer when I call you on Conduit"* — every Conduit voice call connected and then died silently. No Mac connection, no Hermes turn, no error surfaced on the phone.
|
||||||
|
|
||||||
|
Three independent faults sat in series. Fixing any one alone would not have made a call work; each one masked the next.
|
||||||
|
|
||||||
|
## Evidence trail (why it was invisible)
|
||||||
|
|
||||||
|
- The Mac Mini speech server recorded **zero connections all day.** `last_request_at` on the plugin was `null` — meaning no call had *ever* reached the Hermes turn logic.
|
||||||
|
- `conduit-calls.json → history` was empty: no successful call had ever been recorded.
|
||||||
|
- The only traces were in the plugin log: a wrong-looking reason string per attempt, which is what made each fault look like the *next* fault.
|
||||||
|
|
||||||
|
## Fault A — Truncated speech-server URL (403)
|
||||||
|
|
||||||
|
`voice.speech_server.url` was set to the bare host path:
|
||||||
|
|
||||||
|
```
|
||||||
|
wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/ ⛔ (truncated)
|
||||||
|
wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime ✅
|
||||||
|
```
|
||||||
|
|
||||||
|
`plugin_api.py` (~line 2646) dials the configured value **verbatim** — it does not append `/realtime`. The bare `/v1/` returns HTTP **403**, the socket closes, and the plugin reports it as *"the speech server refused the key."* The key was fine; the path was wrong.
|
||||||
|
|
||||||
|
**Fix:** set `voice.speech_server.url` to `…:8443/v1/realtime`. Backup: `~/.hermes/config.yaml.bak.url-081525` (08:15:25).
|
||||||
|
|
||||||
|
## Fault B — Dashboard session TTL of 12 hours (WS 4401)
|
||||||
|
|
||||||
|
`dashboard.basic_auth.session_ttl_seconds` held the **12-hour default (43 200 s)**. The dashboard issues the phone its session, and when that session lapses the Conduit call socket answers **WS 4401 "Dashboard sign-in required"** — with no Hermes turn and no user-visible error.
|
||||||
|
|
||||||
|
Audit-log corroboration: the iPhone **re-authenticated at 08:00:59**, *minutes after* the silent call — i.e. it was walking around with an expired session and only logging back in when the dashboard happened to prompt.
|
||||||
|
|
||||||
|
**Fix:** raised to **30 days (2 592 000 s)** — matched to the refresh-token life so the phone's session no longer expires under it. Dashboard restarted **2026-10-29 08:03:26** (pid 4042003). Backup: `~/.hermes/config.yaml.bak.ttl-20261029080258`.
|
||||||
|
|
||||||
|
> ⚠ Two keys exist and only one was changed: `dashboard.basic_auth.session_ttl_seconds` = **2592000** (raised) and `dashboard.session_ttl_seconds` = **43200** (still 12 h, different key, different consumer). Check *both* when diagnosing a 4401 again.
|
||||||
|
|
||||||
|
## Fault C — Host STT and TTS providers both dead
|
||||||
|
|
||||||
|
Both halves of the voice loop were broken on the host, so even a correctly-signed call would have had no ears and no mouth.
|
||||||
|
|
||||||
|
| Setting | Was | Failed with | Now | Verified |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `tts.provider` | `openai` | Mac `/v1/speech` → **502** | `ollama_local` | 6.94 s of real 64 kbps speech produced |
|
||||||
|
| `stt.provider` | `groq` | **403 / Cloudflare error 1010** | `local` | Hermes' own transcription path returned `success: True, provider: local` |
|
||||||
|
|
||||||
|
- Groq was **blocked, not mis-keyed** — it returns 403/1010 identically with a bogus key, so the failure was Cloudflare bot-blocking, not credentials.
|
||||||
|
- `tts.provider` had been flipped to `openai` by the **Conduit settings tool** earlier (that tool writes Hermes config). The app holds no `tts.provider` setting, so this fix will not be silently overwritten.
|
||||||
|
- `ollama_local` runs `~/.hermes/scripts/ollama-tts.sh` (model `phonon`); `local` STT runs `faster_whisper` (`base`).
|
||||||
|
- Backup: `~/.hermes/config.yaml.bak.stt-084536` (08:45:36); the TTS flip landed between the 08:15 and 08:45 backups.
|
||||||
|
|
||||||
|
## Verification (end-to-end, not just config)
|
||||||
|
|
||||||
|
Rather than trusting three config edits, the whole call path was driven for real: minted a valid HMAC dashboard session → got a WS ticket → let the plugin dial the Mac. Result:
|
||||||
|
|
||||||
|
- auth gate passed → `session.created` / `session.updated`
|
||||||
|
- speech in transcribed correctly ("Hello Conduit, can you hear me right now?")
|
||||||
|
- turn committed → audio streamed back → `response.done`
|
||||||
|
- **Mac counters 0 → 10 connections, 1 completed response, 2.98 s audio, 0 errors**
|
||||||
|
- plugin `connected: true`, handshake **41 ms**
|
||||||
|
- the iPhone minted its own WS tickets at **08:07** and **08:19** — i.e. against the new 30-day session
|
||||||
|
|
||||||
|
## App update to 2.2.0 changed the settings schema
|
||||||
|
|
||||||
|
The iPhone app self-updated to **2.2.0** (from 1.10.1) mid-window, and its settings schema changed with it: the phone now exposes **19 keys, none of them `voice.local.*`**. There is **no phone-side Local Live server address**, so the Local Live path cannot be configured from the host side at all — a prior assumption that it could is now obsolete.
|
||||||
|
|
||||||
|
Consequence: `voice.mode` was moved **`local` → `classic`** (request queued 08:47, approved in the app; `conduit-settings.json` now reads `classic`). Classic needs no phone-side server config because the plugin relays the whole call — which is precisely why Fault C (both host providers dead) had to be fixed for it to work at all.
|
||||||
|
|
||||||
|
## Call record
|
||||||
|
|
||||||
|
`conduit-calls.json → history` now holds **2 entries — the first real calls ever recorded**:
|
||||||
|
|
||||||
|
| Time (AWST) | Outcome |
|
||||||
|
|---|---|
|
||||||
|
| 2026-10-29 08:38:04 | No Mac connection, no Hermes turn. iPhone had to re-login at 08:39:05, a minute later. |
|
||||||
|
| 2026-10-29 08:49:36 | Placed *after* the Classic switch and the STT/TTS fixes. Session ended 08:49:35 — Anthony's verdict is unrecorded. |
|
||||||
|
|
||||||
|
## Open items
|
||||||
|
|
||||||
|
- The 08:49:36 Classic call has **no recorded outcome**. Nothing in the window proves a phone-originated call ran a real Hermes turn.
|
||||||
|
- `last_request_at` was still `null` at the end of the window: the call→Hermes-turn bridge remains **unproven** even though the transport is now healthy. Classic mode is the recommended path.
|
||||||
|
- If an app update or the settings sync ever writes the bare `/v1/` back into `voice.speech_server.url`, every call silently 403s again. Re-check it after any Conduit update.
|
||||||
|
- Watch for the Conduit settings tool re-flipping `tts.provider` should a `tts.provider` key ever appear in the app's schema.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [Conduit](../systems/conduit.md) — service page (voice pipeline, call path, pairing)
|
||||||
|
- [Hermes Agent](../systems/hermes-agent.md) — TTS/STT provider block
|
||||||
|
- [Conduit call request refused (2026-10-24)](2026-10-24-conduit-call-request-refused.md) — the earlier, behavioural half of the "calls don't arrive" problem
|
||||||
|
- [Conduit memory-start timeout (2026-10-22)](2026-10-22-conduit-memory-start-timeout.md)
|
||||||
@@ -1543,3 +1543,44 @@ not pushed.
|
|||||||
- **Not changed, flagged for a human:** the vault mirror at `/home/hermes/wiki/agent-estate-wiki/` has drifted on **both** conventions — top-level **35 match / 3 stale / 72 missing**, `wiki/` **41 match / 53 stale / 16 missing**. A full mirror pass is larger than a stocktake's mandate, so it is recorded here rather than performed.
|
- **Not changed, flagged for a human:** the vault mirror at `/home/hermes/wiki/agent-estate-wiki/` has drifted on **both** conventions — top-level **35 match / 3 stale / 72 missing**, `wiki/` **41 match / 53 stale / 16 missing**. A full mirror pass is larger than a stocktake's mandate, so it is recorded here rather than performed.
|
||||||
- **Removed/archived:** none. **Secrets written:** none (credential names and HTTP status codes only — no key or token values).
|
- **Removed/archived:** none. **Secrets written:** none (credential names and HTTP status codes only — no key or token values).
|
||||||
- **Unresolved:** `cliprox/terra` remains **401 on CT101** — the CT101-side connection still needs a dashboard reconnect if anything is ever to target that pin from that node.
|
- **Unresolved:** `cliprox/terra` remains **401 on CT101** — the CT101-side connection still needs a dashboard reconnect if anything is ever to target that pin from that node.
|
||||||
|
|
||||||
|
## [2026-10-30] audit | Nightly wiki audit — window 2026-10-29 00:00 → 2026-10-30 00:0x AWST: the radar's split finished end to end for the first time, and Conduit voice calls work after three faults in series were fixed
|
||||||
|
|
||||||
|
- **New page — [[incidents/2026-10-29-conduit-voice-silence]]:** every Conduit voice call connected and then died
|
||||||
|
silently. Three faults sat in series, each one masking the next: (A) `voice.speech_server.url` was the truncated
|
||||||
|
`wss://…:8443/v1/`, which the plugin dials **verbatim**, so the Mac answered **403** and the plugin blamed the key;
|
||||||
|
(B) `dashboard.basic_auth.session_ttl_seconds` was the **12 h default**, so an expired phone session got **WS 4401**
|
||||||
|
with no Hermes turn and no error on the device (the iPhone re-authenticated at **08:00:59**, minutes *after* a silent
|
||||||
|
call); (C) **both** host providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq`
|
||||||
|
→ **403/Cloudflare 1010** (blocked, not mis-keyed). Fixed 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL →
|
||||||
|
**2 592 000 s (30 d)**, `stt.provider: local` (faster-whisper `base`), `tts.provider: ollama_local`; dashboard
|
||||||
|
restarted **08:03:26**. Verified by driving the whole call path for real, then by **two live calls at 08:38:04 and
|
||||||
|
08:49:36** — the first entries ever in `conduit-calls.json → history`, so the memory fix is no longer evidence-free.
|
||||||
|
- **Updated [[systems/conduit]]** — new runtime-checks section, the two real calls on record, the incident cross-reference,
|
||||||
|
and `verified_on: 2026-10-30`.
|
||||||
|
- **Updated [[systems/hermes-agent]]** — new dated section **2026-10-30 00:0x AWST** for the voice path (fault table +
|
||||||
|
why it stayed invisible + the two-TTL-keys warning + backups), and the `tts.provider` block now carries a second
|
||||||
|
*superseded* note recording the live values (`ollama_local` / `local`, `/v1/realtime`).
|
||||||
|
- **Updated [[systems/scheduled-tasks]]** — new Count bullet for the 2026-10-30 window, the previous bullet demoted to
|
||||||
|
*superseded*, stage 1's delivery column corrected to `local` (verified live: stage 1 no longer delivers), both radar
|
||||||
|
rows annotated with their 2026-10-29 result, and a new **Resolved 2026-10-29** block under Known Issues.
|
||||||
|
- **Updated [[runbooks/job-radar]]** — new dated slot section **2026-10-29 07:00** (handoff written → consumed →
|
||||||
|
delivered: 170 discovered / 33 duplicates avoided / 37 verified; **20 pages + 2 cover letters**,
|
||||||
|
`missing_cover_letter_count 0`, Notion board **455 → 475**), plus the note that the second real cycle completed.
|
||||||
|
- **Updated [[incidents/2026-10-27-job-radar-budget-exhausted]]** and
|
||||||
|
**[[incidents/2026-10-28-job-radar-stage1-iteration-cap]]** — both **resolved**. Stage 1 ran 07:00:59 → 10:16:12
|
||||||
|
(3 h 15 m 13 s), closed `completed`, and both ledger rows it had opened (`dca848_e93a538ade6b`,
|
||||||
|
`dca848_0a2f03b5952b`) closed `resolved` at 10:16:12. Stage 2 consumed the handoff at 10:40:59, closed `completed`
|
||||||
|
at 10:44:48 and delivered the same second; the 11:30 and 14:30 slots no-oped as designed.
|
||||||
|
- **Updated [[current-state]]**, **[[index]]** — new Recent Changes block and the refreshed page descriptors.
|
||||||
|
- **Cron health:** window **2026-10-29 settled 176 `completed` / 0 `failed`** — the first zero-failure day since
|
||||||
|
2026-10-24 — with **10 deliveries, all 10 `delivered`**. Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**;
|
||||||
|
non-clean `last_status` **5 → 3** (all three are stale re-runs of long-standing rows) and every `failure_streak`
|
||||||
|
reads **0**. **Unclosed incident rows 4 → 1**, the lone survivor being the Facebook monthly's
|
||||||
|
`0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Pins unchanged.
|
||||||
|
- **Vault mirror catch-up:** the drift flagged by the 2026-10-29 stocktake is partly cleared — the pages changed by this
|
||||||
|
audit were mirrored into `/home/hermes/wiki/agent-estate-wiki/`, including `systems/conduit.md`,
|
||||||
|
`runbooks/job-radar.md`, `systems/scheduled-tasks.md`, `systems/hermes-agent.md`, `current-state.md`, `index.md` and
|
||||||
|
`log.md`. The wiki-maintained pages under `infrastructure/` remain a separate, known gap.
|
||||||
|
- **Removed/archived:** none. **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a
|
||||||
|
key or token value.
|
||||||
|
|||||||
+28
-3
@@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-27
|
created: 2026-07-27
|
||||||
updated: 2026-10-29
|
updated: 2026-10-30
|
||||||
review_after: 2026-11-10
|
review_after: 2026-11-13
|
||||||
verified_on: 2026-10-29
|
verified_on: 2026-10-30
|
||||||
version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2)
|
version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2)
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [automation, cron, jobs, notion, dashboard, runbook]
|
tags: [automation, cron, jobs, notion, dashboard, runbook]
|
||||||
@@ -328,6 +328,8 @@ now owns a job of its own and the writes own a second one.
|
|||||||
- First fires: two test runs of stage 2 on 2026-10-27 (17:35:00 → 17:36:52 and 17:38:38 → 17:43:42), both
|
- First fires: two test runs of stage 2 on 2026-10-27 (17:35:00 → 17:36:52 and 17:38:38 → 17:43:42), both
|
||||||
`completed` with delivery `suppressed`. **The first real 07:00 → 11:30 cycle
|
`completed` with delivery `suppressed`. **The first real 07:00 → 11:30 cycle
|
||||||
ran on 2026-10-28 and landed nothing** — stage 1 never wrote its handoff (see the 2026-10-28 slot below).
|
ran on 2026-10-28 and landed nothing** — stage 1 never wrote its handoff (see the 2026-10-28 slot below).
|
||||||
|
✅ **The second real cycle (2026-10-29) completed end to end** — handoff written, consumed and delivered; see the
|
||||||
|
2026-10-29 slot below.
|
||||||
|
|
||||||
**Recovery if stage 2 lands nothing:** check that stage 1 actually wrote its handoff file before running stage 2
|
**Recovery if stage 2 lands nothing:** check that stage 1 actually wrote its handoff file before running stage 2
|
||||||
by hand (`hermes cron run 5fc292ff0f3b`). A stage-2 run with no handoff is a no-op, not a failed search — do not
|
by hand (`hermes cron run 5fc292ff0f3b`). A stage-2 run with no handoff is a no-op, not a failed search — do not
|
||||||
@@ -351,6 +353,29 @@ discovery stage is the thing to bound, not the wall clock.
|
|||||||
|
|
||||||
Full ledger: [[incidents/2026-10-28-job-radar-stage1-iteration-cap]].
|
Full ledger: [[incidents/2026-10-28-job-radar-stage1-iteration-cap]].
|
||||||
|
|
||||||
|
### 2026-10-29 07:00 slot — the split finally finished: handoff written, consumed, and delivered
|
||||||
|
|
||||||
|
Stage 1 (`dca8482e4f76`) ran **07:00:59 → 10:16:12 AWST** (**3 h 15 m 13 s**) and closed **`completed`** — the first
|
||||||
|
post-split run to finish *and* write its handoff. `~/.hermes/radar-ref/handoff/2026-10-29.json` (24.8 KB) records
|
||||||
|
`run_date 2026-10-29T08:23:51+08:00`, **170 discovered / 33 duplicates avoided / 37 verified**, sources
|
||||||
|
`seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Both of the ledger rows the previous two days opened
|
||||||
|
(`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) **closed `resolved` at 10:16:12**, and this run opened no new row.
|
||||||
|
|
||||||
|
Stage 2 (`5fc292ff0f3b`) fired at **10:13:46**, consumed the handoff at **10:40:59**, closed **10:44:48 `completed`**
|
||||||
|
and **delivered to topic 1030 in the same second**. Its `stage2_result`: **20 pages written / 20 `page_ids` /
|
||||||
|
2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`** —
|
||||||
|
the Notion board went **455 → 475**. The 11:30 and 14:30 slots then no-oped exactly as designed
|
||||||
|
(**10 m 46 s** and **1 m 51 s**, handoff already consumed).
|
||||||
|
|
||||||
|
**What this does and does not settle:**
|
||||||
|
|
||||||
|
- The split's core promise holds in production — the writes can no longer be lost to a discovery overrun, and the first
|
||||||
|
full cycle landed 20 pages and 2 cover letters with **zero missing**.
|
||||||
|
- It does **not** mean discovery is fixed. 3 h 15 m is still long, and the 10-27 **tool-call budget** and 10-28
|
||||||
|
**iteration cap** remain reachable. Treat a `completed` stage-1 row with **no handoff file** as the same fault,
|
||||||
|
whatever the execution status says.
|
||||||
|
- Stage 1 no longer delivers (verified live: `deliver: local`); its failure route to topic 1030 is unchanged.
|
||||||
|
|
||||||
## Browser-layer hazard inside the run (2026-10-12)
|
## Browser-layer hazard inside the run (2026-10-12)
|
||||||
|
|
||||||
This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove
|
This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove
|
||||||
|
|||||||
+95
-50
@@ -3,12 +3,12 @@ title: Conduit
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-10-24
|
created: 2026-10-24
|
||||||
updated: 2026-10-28
|
updated: 2026-10-30
|
||||||
verified_on: 2026-10-28
|
verified_on: 2026-10-30
|
||||||
review_after: 2026-11-08
|
review_after: 2026-11-08
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling]
|
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl]
|
||||||
sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329]
|
sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329, session-20260908_031541_49c051]
|
||||||
---
|
---
|
||||||
|
|
||||||
# Conduit
|
# Conduit
|
||||||
@@ -21,11 +21,18 @@ be started from the phone where the agent can speak and listen (`conduit-voice`
|
|||||||
Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[systems/hermes-ui]]
|
Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[systems/hermes-ui]]
|
||||||
(the browser desktop), [[systems/push-notifications]] (the older `hermes-push` bridge).
|
(the browser desktop), [[systems/push-notifications]] (the older `hermes-push` bridge).
|
||||||
|
|
||||||
## Runtime checks (2026-10-26 00:1x AWST)
|
## Runtime checks (2026-10-30 00:0x AWST)
|
||||||
- **Working tree:** M dashboard/plugin_api.py — the uncommitted memory-timeout patch is **still present**.
|
- **`conduit-calls.json` → `history` entries: 2** — the first real calls ever recorded
|
||||||
- **`conduit-calls.json` → `history` entries:** **0** — the call-request path remains unused
|
(2026-10-29 08:38:04 and 08:49:36). The path is no longer unused; the earlier
|
||||||
→ [[incidents/2026-10-24-conduit-call-request-refused]].
|
*"0 calls"* reading is superseded. See [[incidents/2026-10-29-conduit-voice-silence]].
|
||||||
- **Tool use in-window:** `conduit_show` ×1 — the plugin works for cards; the open incident concerns *placing calls*.
|
- **Calls still unproven end-to-end:** `last_request_at` remained `null` at the end of the
|
||||||
|
2026-10-29 window — no call is yet known to have run a real Hermes turn.
|
||||||
|
- **`ends` vs `history`:** `ends` records *any turn-end that fires a watch* (including cron
|
||||||
|
sessions) — never read it as a call log.
|
||||||
|
- **Device:** iPhone app version now **2.2.0** (was 1.10.1 on 2026-10-28), **19 settings keys**,
|
||||||
|
**no `voice.local.*` keys** → Local Live cannot be configured host-side any more.
|
||||||
|
- **Voice providers were both dead and are now fixed** — `tts.provider: ollama_local`,
|
||||||
|
`stt.provider: local` (2026-10-29). Details below and in the 10-29 incident.
|
||||||
|
|
||||||
## Where it lives
|
## Where it lives
|
||||||
- **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push`
|
- **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push`
|
||||||
@@ -46,36 +53,42 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste
|
|||||||
- **Data stores** (`~/.hermes/`): `conduit-health.sqlite3`, `conduit-location.sqlite3`,
|
- **Data stores** (`~/.hermes/`): `conduit-health.sqlite3`, `conduit-location.sqlite3`,
|
||||||
`conduit-phone.sqlite3`, plus JSON state — `conduit-push.json` (credential / gateway id /
|
`conduit-phone.sqlite3`, plus JSON state — `conduit-push.json` (credential / gateway id /
|
||||||
installation id / relay url / e2e — **no values recorded here**), `conduit-calls.json`,
|
installation id / relay url / e2e — **no values recorded here**), `conduit-calls.json`,
|
||||||
`conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`.
|
`conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`,
|
||||||
|
`conduit-link-status.json`, `conduit-local-live.json` (**holds an api key — never copy its
|
||||||
|
contents into the wiki**).
|
||||||
|
|
||||||
## Current state (verified 2026-10-25 00:2x–00:5x AWST)
|
## Current state (re-verified 2026-10-30 00:0x AWST)
|
||||||
- **Dashboard gateway (re-verified 2026-10-26):** listening on `0.0.0.0:9119`; process started **2026-10-25
|
- **Dashboard gateway:** listening on `0.0.0.0:9119`; **pid 4042003, started 2026-10-29 08:03:26**
|
||||||
18:42:33** — **restarted since the 2026-10-24 entry, so the upstream commits are now live** (the recorded
|
— the restart that put the 30-day session TTL live (see below). The `0.64.0` → `0.69.0` upstream
|
||||||
*"not restarted since / not live"* note is superseded, not wrong at the time) — so it is still the process that picked up the memory-timeout patch, and
|
commits remain **not** live (the process predates nothing relevant; the tree still only carries the
|
||||||
the `0.64.0` → `0.69.0` upstream commits are **not** live.
|
memory-timeout patch).
|
||||||
- **Device:** one registered — **iPhone** (`id 70aa63f3…`, `kid c24e8215…`), app version `2.0.0`,
|
- **Device:** one registered — **iPhone**, app version **`2.2.0`** (self-updated from `1.10.1` on
|
||||||
features `chat-cards`, `info-cards`, `live-cards`, `maps-navigate`, `media-cards`, `show-cards`;
|
2026-10-29), `seen_at` **2026-10-29 18:56:44 AWST**, `mode: small`, **19 settings keys**.
|
||||||
tz `Australia/Perth`; sharing window 2026-10-16 → 2027-04-22. Paired **2026-10-22 12:54:39**.
|
`last_seen` / `last_sent` / `last_contact` are `null` in the current store — those three fields
|
||||||
`last_seen` / `last_sent` / `last_contact` all read **2026-10-23 19:10:30 AWST** (~29 h stale at
|
track *phone-status* contact, not data flow (health rows and link pings keep arriving regardless:
|
||||||
this audit) while health rows kept arriving until **2026-10-24 09:59:59** and window rollups were
|
`conduit-link-status.json` `app_seen_at` **2026-10-29 23:53:15**, `push_ok_at` **18:50:00**).
|
||||||
recomputed at **13:56:35** — so those three fields track *phone-status* contact, not data flow.
|
Paired **2026-10-22 12:54:39**; sharing window 2026-10-16 → 2027-04-22.
|
||||||
(The 10-24 page recorded 19:02:16 from the same store; the epoch reads 19:10:35.)
|
- **Dashboard session TTL (raised 2026-10-29):** `dashboard.basic_auth.session_ttl_seconds` =
|
||||||
|
**2592000 (30 days)**, up from the **43200 (12 h)** default. A lapsed session was making Conduit's
|
||||||
|
call socket answer **WS 4401 "Dashboard sign-in required"** — silently. ⚠ `dashboard.session_ttl_seconds`
|
||||||
|
is a **separate key and still 43200**; check both.
|
||||||
- **Call settings:** `enabled`, `when_asked`, `decides`, `alerts` all true; rate limits
|
- **Call settings:** `enabled`, `when_asked`, `decides`, `alerts` all true; rate limits
|
||||||
`min_gap_s 120` / `per_hour 6` / `per_day 20`; `avoid_unknown_place` false.
|
`min_gap_s 120` / `per_hour 6` / `per_day 20`; `avoid_unknown_place` false.
|
||||||
- **Recent calls — none.** `conduit-calls.json → history` is **empty**: no call has been recorded
|
- **Recent calls — two, both 2026-10-29.** `conduit-calls.json → history` now reads **2**
|
||||||
since the memory-timeout fix landed, so that fix still has **no positive evidence** behind it. The
|
(08:38:04, 08:49:36). The 08:38:04 call produced no Mac connection and no Hermes turn, and the
|
||||||
“two call sessions closed cleanly 00:03:23 / 00:06:57” claim previously on this page was a **false
|
iPhone re-authenticated at 08:39:05 — the expired-session fault. The 08:49:36 call was placed after
|
||||||
positive** — those `ends` entries are the two **cron** sessions of that night, and `ends` records
|
the `classic` mode switch and the STT/TTS fixes, but the session ended one second earlier, so its
|
||||||
any turn-end that fires a watch, not calls →
|
outcome is **unrecorded**. The memory-timeout patch therefore still has no *confirmed* call behind
|
||||||
[[incidents/2026-10-24-conduit-call-request-refused]].
|
it → [[incidents/2026-10-29-conduit-voice-silence]].
|
||||||
- **Data volume (2026-10-25 00:2x):** `conduit-health.sqlite3` **569.9 MB** (+8.2 MB in 25.7 h),
|
- **Data volume (2026-10-30 00:0x):** `conduit-health.sqlite3` **930.0 MB** (+360 MB in 5 days;
|
||||||
holding **1,520,229 `events`** (1,510,929 `sample` / 6,070 `sleep` / 3,230 `workout`),
|
+8.2 MB/25.7 h as measured on 10-25 — the rate has not held, so treat growth as **ingestion-paced,
|
||||||
**3,028 `buckets`** and **124 `windows`** for its single device; newest row **2026-10-24 09:59:59**,
|
unbounded**), holding **2,601,565 `events`** (2,592,189 `sample` / 6,146 `sleep` / 3,230 `workout`),
|
||||||
newest `buckets.date` 2026-10-24; `conduit-location.sqlite3` 80 KB; `conduit-phone.sqlite3` 284 KB.
|
**4,105 `buckets`** and **411 `windows`**; newest health row **2026-10-29 23:12:01 AWST**.
|
||||||
|
`conduit-location.sqlite3` 80 KB; `conduit-phone.sqlite3` 284 KB (last measured 10-25).
|
||||||
⚠ The 10-24 page recorded **308 `buckets` / 62 `windows`**; a direct `COUNT(*)` reads
|
⚠ The 10-24 page recorded **308 `buckets` / 62 `windows`**; a direct `COUNT(*)` reads
|
||||||
**3,028 / 124**, so the earlier pair was a narrower grouping — quote these. **No retention or
|
**4,105 / 411**, so the earlier pair was a narrower grouping — quote these. **No retention or
|
||||||
compaction has been observed** — the store grows from ingestion alone and is worth watching against
|
compaction has been observed**, and the store is now closing on 1 GB — watch it against the CT460
|
||||||
the CT460 disk budget.
|
disk budget.
|
||||||
|
|
||||||
## Calls from Hermes (the `conduit_call_user` tool)
|
## Calls from Hermes (the `conduit_call_user` tool)
|
||||||
The plugin registers a **`conduit_call_user`** tool (toolset `conduit`, 📞) for every session and
|
The plugin registers a **`conduit_call_user`** tool (toolset `conduit`, 📞) for every session and
|
||||||
@@ -88,20 +101,36 @@ the call rings when the asking turn ends and plays the reason first. Limits: `mi
|
|||||||
|
|
||||||
⚠ **Known failure, 2026-10-24:** three requests to be phoned were refused by the agent, which
|
⚠ **Known failure, 2026-10-24:** three requests to be phoned were refused by the agent, which
|
||||||
asserted the capability did not exist **without calling the tool or loading the skill** →
|
asserted the capability did not exist **without calling the tool or loading the skill** →
|
||||||
[[incidents/2026-10-24-conduit-call-request-refused]].
|
[[incidents/2026-10-24-conduit-call-request-refused]]. **Positive counter-example 2026-10-29:** in
|
||||||
|
that window the agent did look the tool up and did place calls (two entries in `history`), so the
|
||||||
|
failures were behavioural, not a missing capability.
|
||||||
|
|
||||||
## Voice pipeline (probed 2026-10-28)
|
## Voice pipeline (verified 2026-10-30)
|
||||||
|
|
||||||
Phone voice sessions ride the Mac mini speech server, which is also the Hermes TTS target:
|
Phone voice sessions ride the Mac mini speech server, which is also the Hermes TTS/STT target.
|
||||||
|
**All three settings below were wrong or dead as of 2026-10-28 and were fixed on 2026-10-29** —
|
||||||
|
see [[incidents/2026-10-29-conduit-voice-silence]].
|
||||||
|
|
||||||
- `voice.speech_server.url` = `wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/` — live-voice transport,
|
| Setting | Value (now) | Was | Failure when wrong |
|
||||||
`wake_word.enabled: true`.
|
|---|---|---|---|
|
||||||
- `tts.provider: openai` → `https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech` (model `phonon`,
|
| `voice.speech_server.url` | `wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime` | `…:8443/v1/` | **403** — the plugin dials the value **verbatim**, it does not append `/realtime` |
|
||||||
voice `alloy`) — see [[systems/hermes-agent]].
|
| `tts.provider` | `ollama_local` (`~/.hermes/scripts/ollama-tts.sh`, model `phonon`) | `openai` → Mac `/v1/speech` | **502** |
|
||||||
- **Health check:** `GET https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/pool` returned
|
| `stt.provider` | `local` (`faster_whisper`, model `base`) | `groq` | **403 / Cloudflare error 1010** — a *block*, not a bad key (fails identically with a bogus key) |
|
||||||
`{"size": 2, "in_use": 0}` with both units `idle` on 2026-10-28 00:2x — a **2-unit pool**. `/v1/models` is
|
|
||||||
**not implemented** (`{"detail":"Not Found"}`); probe `/v1/pool`, not `/v1/models`.
|
- `voice.mode` = **`classic`** (switched from `local` 2026-10-29). Classic relays everything through
|
||||||
|
the plugin and needs **no phone-side server address** — which is why both host providers above had
|
||||||
|
to work. A Local Live path cannot be configured from the host at all: app **2.2.0**'s schema exposes
|
||||||
|
**19 keys, none of them `voice.local.*`**.
|
||||||
- `voice.auto_tts: false` — assistant replies are not auto-spoken; audio appears only when requested.
|
- `voice.auto_tts: false` — assistant replies are not auto-spoken; audio appears only when requested.
|
||||||
|
- `wake_word.enabled: true`.
|
||||||
|
- **Health check:** `GET https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/pool` returned
|
||||||
|
`{"size": 2, "in_use": 0}` with both units `idle` on 2026-10-28 00:2x — a **2-unit pool**. `/v1/models`
|
||||||
|
is **not implemented** (`{"detail":"Not Found"}`); probe `/v1/pool`, not `/v1/models`.
|
||||||
|
- **End-to-end proof (2026-10-29):** driving the real call path produced `session.created` →
|
||||||
|
transcription of live speech → `response.done`, with the Mac counters moving **0 → 10 connections
|
||||||
|
/ 1 completed response / 2.98 s audio / 0 errors** and the plugin handshaking in **41 ms**.
|
||||||
|
- ⚠ **Fragile URL:** the Conduit settings tool writes Hermes config (it had flipped `tts.provider` to
|
||||||
|
`openai`). If it or an app update ever writes the bare `/v1/` back, every call silently 403s again.
|
||||||
|
|
||||||
## Memory integration
|
## Memory integration
|
||||||
The voice session (`conduit-voice`) reaches Mnemosyne through the plugin's own memory layer, not
|
The voice session (`conduit-voice`) reaches Mnemosyne through the plugin's own memory layer, not
|
||||||
@@ -118,15 +147,27 @@ See [[incidents/2026-10-22-conduit-memory-start-timeout]] and [[systems/mnemosyn
|
|||||||
git -C ~/.hermes/plugins/conduit_push log --oneline -1 && \
|
git -C ~/.hermes/plugins/conduit_push log --oneline -1 && \
|
||||||
git -C ~/.hermes/plugins/conduit_push status --short
|
git -C ~/.hermes/plugins/conduit_push status --short
|
||||||
|
|
||||||
# is the gateway up, and since when?
|
# is the gateway up, and since when? (the restart that applies a session-TTL change)
|
||||||
ss -tlnp | grep 9119
|
ss -tlnp | grep 9119
|
||||||
|
|
||||||
|
# session TTLs — BOTH keys, they are different consumers
|
||||||
|
python3 -c "import yaml;c=yaml.safe_load(open('/home/hermes/.hermes/config.yaml'));\
|
||||||
|
print('basic_auth',c['dashboard']['basic_auth'].get('session_ttl_seconds'),\
|
||||||
|
'| dashboard',c['dashboard'].get('session_ttl_seconds'))"
|
||||||
|
|
||||||
|
# voice transport + providers
|
||||||
|
python3 -c "import yaml;c=yaml.safe_load(open('/home/hermes/.hermes/config.yaml'));\
|
||||||
|
print(c['voice']['speech_server']['url']);print(c['tts']['provider'],c['stt']['provider'])"
|
||||||
|
|
||||||
# did a voice call lose its memory context?
|
# did a voice call lose its memory context?
|
||||||
grep -h "Conduit" ~/.hermes/logs/agent.log | grep -i "memory" | tail
|
grep -h "Conduit" ~/.hermes/logs/agent.log | grep -i "memory" | tail
|
||||||
|
|
||||||
# did a real call happen? (history = calls; ends = turn-ends, NOT calls)
|
# did a real call happen? (history = calls; ends = turn-ends, NOT calls)
|
||||||
python3 -c "import json,os;d=json.load(open(os.path.expanduser('~/.hermes/conduit-calls.json')));print(len(d['history']),'calls;',len(d['ends']),'turn-ends')"
|
python3 -c "import json,os;d=json.load(open(os.path.expanduser('~/.hermes/conduit-calls.json')));print(len(d['history']),'calls;',len(d['ends']),'turn-ends')"
|
||||||
|
|
||||||
|
# is the phone actually reaching the relay?
|
||||||
|
cat ~/.hermes/conduit-link-status.json
|
||||||
|
|
||||||
# state files (no secrets in the wiki)
|
# state files (no secrets in the wiki)
|
||||||
ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3
|
ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3
|
||||||
```
|
```
|
||||||
@@ -134,16 +175,20 @@ ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3
|
|||||||
## Open items
|
## Open items
|
||||||
- Commit or upstream the `plugin_api.py` timeout patch so a plugin update cannot revert it — it is
|
- Commit or upstream the `plugin_api.py` timeout patch so a plugin update cannot revert it — it is
|
||||||
now five releases behind HEAD.
|
now five releases behind HEAD.
|
||||||
- **Verify a call end-to-end.** No call has been placed since the fix, so `MEMORY_TIMEOUT_S = 60` is
|
- **Verify a call end-to-end from the phone.** The 2026-10-29 08:49:36 call has no recorded outcome,
|
||||||
still unexercised by a real call, and the 10-24 refusal means the chat route has never worked →
|
so `MEMORY_TIMEOUT_S = 60` is still unexercised by a *confirmed* call, and `last_request_at` was
|
||||||
[[incidents/2026-10-24-conduit-call-request-refused]].
|
still `null` → [[incidents/2026-10-29-conduit-voice-silence]].
|
||||||
- No retention policy on `conduit-health.sqlite3` (569.9 MB and growing).
|
- Re-check `voice.speech_server.url` after any Conduit/app update or settings sync — a bare `/v1/`
|
||||||
|
silently 403s every call.
|
||||||
|
- No retention policy on `conduit-health.sqlite3` (**930 MB** and growing unbounded).
|
||||||
- The Mnemosyne cold start that motivated the patch is still slow (10–45 s); only the *budget* was
|
- The Mnemosyne cold start that motivated the patch is still slow (10–45 s); only the *budget* was
|
||||||
raised. → [[systems/mnemosyne]]
|
raised. → [[systems/mnemosyne]]
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
- [[incidents/2026-10-29-conduit-voice-silence]]
|
||||||
- [[incidents/2026-10-24-conduit-call-request-refused]]
|
- [[incidents/2026-10-24-conduit-call-request-refused]]
|
||||||
- [[incidents/2026-10-22-conduit-memory-start-timeout]]
|
- [[incidents/2026-10-22-conduit-memory-start-timeout]]
|
||||||
|
- [[systems/hermes-agent]]
|
||||||
- [[systems/mnemosyne]]
|
- [[systems/mnemosyne]]
|
||||||
- [[systems/scheduled-tasks]]
|
- [[systems/scheduled-tasks]]
|
||||||
- [[current-state]]
|
- [[current-state]]
|
||||||
|
|||||||
+34
-2
@@ -3,8 +3,8 @@ title: Hermes Agent
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-28
|
updated: 2026-10-30
|
||||||
verified_on: 2026-10-28
|
verified_on: 2026-10-30
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system, hermes, agent, llm]
|
tags: [system, hermes, agent, llm]
|
||||||
sources: [live-hermes-version, live-config, live-skill-inventory]
|
sources: [live-hermes-version, live-config, live-skill-inventory]
|
||||||
@@ -166,6 +166,10 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
|
|||||||
- **TTS:** ElevenLabs is the configured provider; its package was missing from the live managed env
|
- **TTS:** ElevenLabs is the configured provider; its package was missing from the live managed env
|
||||||
and was reinstalled 2026-10-10. `voice.auto_tts: false` - auto-speak is off by design, so TTS
|
and was reinstalled 2026-10-10. `voice.auto_tts: false` - auto-speak is off by design, so TTS
|
||||||
output appears only when explicitly requested.
|
output appears only when explicitly requested.
|
||||||
|
- ⚠ **Superseded twice — live config checked 2026-10-30:** `tts.provider` is now **`ollama_local`**
|
||||||
|
(`~/.hermes/scripts/ollama-tts.sh`, model `phonon`) and `stt.provider` is now **`local`** (faster-whisper
|
||||||
|
`base`); `voice.speech_server.url` is now `wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime`
|
||||||
|
(the bare `/v1/` answers **403**, and the plugin dials the value verbatim). See the 2026-10-30 section below.
|
||||||
- ⚠ **Superseded — live config checked 2026-10-28** (change date not established): `tts.provider` is
|
- ⚠ **Superseded — live config checked 2026-10-28** (change date not established): `tts.provider` is
|
||||||
**`openai`** pointed at the **Mac mini speech server** — `tts.providers.openai.base_url` =
|
**`openai`** pointed at the **Mac mini speech server** — `tts.providers.openai.base_url` =
|
||||||
`https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech`, `model: phonon`, `voice: alloy`. Live voice
|
`https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech`, `model: phonon`, `voice: alloy`. Live voice
|
||||||
@@ -179,6 +183,34 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
|
|||||||
cadence - [[systems/messaging-integrations]].
|
cadence - [[systems/messaging-integrations]].
|
||||||
- **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]].
|
- **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]].
|
||||||
|
|
||||||
|
## 2026-10-30 00:0x AWST — the Conduit voice path: a truncated URL, a 12-hour session TTL, and two dead providers
|
||||||
|
|
||||||
|
Anthony asked why Conduit voice calls never got an answer. **Three faults sat in series** — each one masked the next —
|
||||||
|
and all three were fixed on 2026-10-29 between 08:02 and 08:4x AWST:
|
||||||
|
|
||||||
|
| Fault | Was | Symptom | Now |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `voice.speech_server.url` | `wss://…:8443/v1/` | dialled **verbatim** → HTTP **403**, reported as *"the speech server refused the key"* | `wss://…:8443/v1/realtime` |
|
||||||
|
| `dashboard.basic_auth.session_ttl_seconds` | `43200` (12 h default) | expired phone session → **WS 4401 "Dashboard sign-in required"**, no Hermes turn, no error on the device | `2592000` (30 d, matched to the refresh-token life) |
|
||||||
|
| `tts.provider` / `stt.provider` | `openai` / `groq` | Mac `/v1/speech` **502** / **403 Cloudflare 1010** | `ollama_local` / `local` |
|
||||||
|
|
||||||
|
- **Why it stayed invisible:** the Mac recorded **zero connections**, `conduit-calls.json → history` was empty, and each
|
||||||
|
attempt logged a reason string pointing at the *next* fault — so one broken chain looked like three unrelated problems.
|
||||||
|
`last_request_at` was `null`: no call had ever reached Hermes' turn logic.
|
||||||
|
- **The 12 h TTL hid behind the others.** The dashboard had stopped issuing long-enough sessions, and the iPhone only
|
||||||
|
re-authenticated when the dashboard happened to prompt it — an audit-log entry shows a re-auth at **08:00:59**, minutes
|
||||||
|
*after* a silent call.
|
||||||
|
- **Two TTL keys exist.** `dashboard.basic_auth.session_ttl_seconds` was raised; the similarly named
|
||||||
|
`dashboard.session_ttl_seconds` **still reads 43200**. Check **both** on any future 4401.
|
||||||
|
- **Groq was blocked, not mis-keyed** — it answers 403/1010 with a bogus key too, so this was Cloudflare bot-blocking
|
||||||
|
rather than a credential problem.
|
||||||
|
- **Verified for real:** the whole path was driven (mint HMAC dashboard session → WS ticket → let the plugin dial the Mac),
|
||||||
|
then two live calls landed at **2026-10-29 08:38:04** and **08:49:36** — the first entries ever in
|
||||||
|
`conduit-calls.json → history`. `ollama_local` produced **6.94 s** of real 64 kbps speech; `local` STT returned
|
||||||
|
`success: True`.
|
||||||
|
- Backups: `config.yaml.bak.url-081525`, `config.yaml.bak.ttl-20261029080258`, `config.yaml.bak.stt-084536`; the dashboard
|
||||||
|
restarted **08:03:26** → [[incidents/2026-10-29-conduit-voice-silence]] · [[systems/conduit]].
|
||||||
|
|
||||||
## 2026-10-26 00:1x AWST — gateway restart record and the dead Nous credential
|
## 2026-10-26 00:1x AWST — gateway restart record and the dead Nous credential
|
||||||
|
|
||||||
- **Three gateway starts on 2026-10-25**: `00:32:20`, `18:30:04`, `19:27:13` AWST (after `2026-10-24 12:24:39`).
|
- **Three gateway starts on 2026-10-25**: `00:32:20`, `18:30:04`, `19:27:13` AWST (after `2026-10-24 12:24:39`).
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-29
|
updated: 2026-10-30
|
||||||
verified_on: 2026-10-29
|
verified_on: 2026-10-30
|
||||||
review_after: 2026-11-03
|
review_after: 2026-11-06
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, scheduled, automation]
|
tags: [cron, scheduled, automation]
|
||||||
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||||
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
|||||||
# Scheduled Tasks
|
# Scheduled Tasks
|
||||||
|
|
||||||
## Count
|
## Count
|
||||||
- **Current as of 2026-10-29 00:3x AWST (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real and the discovery stage still could not finish: stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.**
|
- **Current as of 2026-10-30 00:0x AWST (live `jobs.json` + `executions.db`) — the radar's two-stage split worked end to end for the first time, and the fleet had its first clean day in a week.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3**, and every `failure_streak` reads **0** — the three survivors are all stale re-runs of long-standing rows (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`). Window **2026-10-29 settled 176 `completed` / 0 `failed`** (+ this audit `running`) — the first zero-failure day since 2026-10-24; 2026-10-28 stands at **168 / 2**. **Unclosed incident rows 4 → 1:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving only the Facebook monthly's `0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report). `dca8482e4f76` (stage 1) ran **07:00:59 → 10:16:12** (**3 h 15 m 13 s**), closed **`completed`**, and wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok` — opening **no** new incident row (contrast the two consecutive stage-1 failures on 10-27 and 10-28). `5fc292ff0f3b` (stage 2) fired **10:13:46**, consumed the handoff **10:40:59**, closed **10:44:48 `completed`** and delivered the same second: **20 pages / 20 `page_ids` / 2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went **455 → 475**. Both later slots no-oped as designed (**11:30 → 10 m 46 s**, **14:30 → 1 m 51 s**). Stage 1 correctly no longer delivers. See [[incidents/2026-10-27-job-radar-budget-exhausted]] and [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], now both **resolved**. **Pins unchanged** — 19 of the 26 enabled jobs pin `custom:omni-101` (12 `smart-route`, 6 `free-stack`, 1 explicit).
|
||||||
|
- **Prior window — as of 2026-10-29 00:3x AWST *(superseded by the 2026-10-30 entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real, but stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.**
|
||||||
- **Prior window — as of 2026-10-28 00:2x AWST** (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. *(superseded by the 2026-10-29 entry above)*
|
- **Prior window — as of 2026-10-28 00:2x AWST** (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. *(superseded by the 2026-10-29 entry above)*
|
||||||
- **Prior window — as of 2026-10-27 00:1x AWST *(superseded by the 2026-10-28 entry above)* (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`).
|
- **Prior window — as of 2026-10-27 00:1x AWST *(superseded by the 2026-10-28 entry above)* (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`).
|
||||||
- **Current as of 2026-10-26 00:1x AWST** *(superseded by the 2026-10-27 entry above)* **(live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`).
|
- **Current as of 2026-10-26 00:1x AWST** *(superseded by the 2026-10-27 entry above)* **(live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`).
|
||||||
@@ -51,8 +52,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
|||||||
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) |
|
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) |
|
||||||
| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 |
|
| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 |
|
||||||
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
|
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
|
||||||
| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00 |
|
| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | `local` (failures → topic 1030) | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00. ✅ **First full split run 2026-10-29:** 07:00:59 → 10:16:12 (3 h 15 m 13 s), `completed`, wrote `handoff/2026-10-29.json` (170 discovered / 33 duplicates avoided / 37 verified); both of its ledger rows closed `resolved` at 10:16:12. **Delivery is now `local`** — stage 2 owns the report |
|
||||||
| Daily Marketing Job Radar — Write and Letters (stage 2) | `5fc292ff0f3b` | `30 11,14 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Created 2026-10-27 17:34** — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). `custom:omni-101` / `smart-route`; first fires 2026-10-27 17:35 and 17:38 both `completed`. See [[incidents/2026-10-27-job-radar-budget-exhausted]] |
|
| Daily Marketing Job Radar — Write and Letters (stage 2) | `5fc292ff0f3b` | `30 11,14 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Created 2026-10-27 17:34** — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). `custom:omni-101` / `smart-route`; first fires 2026-10-27 17:35 and 17:38 both `completed`. ✅ **First full split run 2026-10-29:** fired 10:13:46, consumed the handoff at 10:40:59, closed `completed` 10:44:48 and delivered the same second — **20 pages + 2 cover letters, `missing_cover_letter_count 0`**, Notion board 455 → 475; the 11:30 and 14:30 slots then no-oped correctly. See [[incidents/2026-10-27-job-radar-budget-exhausted]] |
|
||||||
| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant |
|
| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant |
|
||||||
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] |
|
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] |
|
||||||
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
|
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
|
||||||
@@ -97,6 +98,11 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
|||||||
|
|
||||||
## Known Issues
|
## Known Issues
|
||||||
|
|
||||||
|
### Resolved 2026-10-29
|
||||||
|
|
||||||
|
- **The Job Radar's two-stage split finished end to end for the first time (`dca8482e4f76` + `5fc292ff0f3b`):** stage 1 ran **07:00:59 → 10:16:12** (3 h 15 m 13 s), closed **`completed`** and wrote `handoff/2026-10-29.json` (**170 discovered / 33 duplicates avoided / 37 verified**) — the first post-split run to write a handoff, after 10-27 died on the **tool-call budget** and 10-28 on the **iteration cap**. Stage 2 consumed it at **10:40:59**, closed `completed` at **10:44:48** and **delivered** to topic 1030 in the same second: **20 pages, 2 cover letters, `missing_cover_letter_count 0`**, Notion **455 → 475**. Both stage-1 ledger rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12**, and the run opened no new row → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], [[runbooks/job-radar]]. ⚠ The discovery stage is still **slow** (3 h 15 m) and both ceilings remain reachable — a `completed` stage-1 row with **no handoff file** is the same fault in new clothing.
|
||||||
|
- **Conduit voice calls — three faults in series, all fixed and verified by two real calls:** the trunk speech-server URL was truncated (`…/v1/` → **403**, blamed on the key), the dashboard session TTL was the **12 h default** (expired sessions answer **WS 4401** with no error on the phone), and **both** host providers were dead (`tts.provider: openai` → Mac `/v1/speech` **502**; `stt.provider: groq` → **403/Cloudflare 1010**). Repaired 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL → **2 592 000 s**, `stt.provider: local`, `tts.provider: ollama_local`; dashboard restarted **08:03:26**. The path was driven end to end, then **calls at 08:38:04 and 08:49:36** — the first entries in `conduit-calls.json → history`. ⚠ Two TTL keys exist and only one was raised → [[incidents/2026-10-29-conduit-voice-silence]], [[systems/conduit]].
|
||||||
|
|
||||||
### Resolved 2026-10-12
|
### Resolved 2026-10-12
|
||||||
- **Both `deepseek-v4-flash` incidents closed by a clean scheduled run (`fad5a86979d4`, 03:00):** the Facebook Birthday Weekly Registry Sync completed and delivered on 2026-10-12 (03:00:45 → 03:05:47), and incidents `fad5a8_a13bbc837e1c` (open since 2026-09-28, every fallback provider exhausted) and `fad5a8_c80b59a907f0` (since 2026-09-30, `deepseek-v4-flash` HTTP 400) **both auto-closed at 03:05:39**. Its stale pre-repin `last_status: error` cleared with them, after carrying it for two weeks — the repin is now verified by a real scheduled run, not only by the manual one.
|
- **Both `deepseek-v4-flash` incidents closed by a clean scheduled run (`fad5a86979d4`, 03:00):** the Facebook Birthday Weekly Registry Sync completed and delivered on 2026-10-12 (03:00:45 → 03:05:47), and incidents `fad5a8_a13bbc837e1c` (open since 2026-09-28, every fallback provider exhausted) and `fad5a8_c80b59a907f0` (since 2026-09-30, `deepseek-v4-flash` HTTP 400) **both auto-closed at 03:05:39**. Its stale pre-repin `last_status: error` cleared with them, after carrying it for two weeks — the repin is now verified by a real scheduled run, not only by the manual one.
|
||||||
- **`dca8482e4f76` timeout incident closed (`dca848_23fcd2b18aaa`):** the Daily Marketing Job Radar ran on schedule 2026-10-12 07:00:47 → 07:33:50 and delivered to topic 1030; the 2026-10-09 timeout incident that belonged to the provider storm **auto-closed at 07:33:41**. Of the ten storm incidents, nine have now closed and **one remains**: `1b23c0_c35703f637b8` (`1b23c042e17d`, still `alerted`, next slot Fri 2026-10-16 08:00). See [[incidents/2026-10-09-cron-provider-timeout-storm]].
|
- **`dca8482e4f76` timeout incident closed (`dca848_23fcd2b18aaa`):** the Daily Marketing Job Radar ran on schedule 2026-10-12 07:00:47 → 07:33:50 and delivered to topic 1030; the 2026-10-09 timeout incident that belonged to the provider storm **auto-closed at 07:33:41**. Of the ten storm incidents, nine have now closed and **one remains**: `1b23c0_c35703f637b8` (`1b23c042e17d`, still `alerted`, next slot Fri 2026-10-16 08:00). See [[incidents/2026-10-09-cron-provider-timeout-storm]].
|
||||||
|
|||||||
Reference in New Issue
Block a user