wiki(audit): 2026-10-29 window — Conduit voice fault documented and repaired; radar two-stage split resolved both incidents (20 pages, 2 letters, Notion 455->475); cron 176/0

This commit is contained in:
Tony0410
2026-10-30 00:21:21 +08:00
parent cc95dcc54d
commit 1d335ca8e6
10 changed files with 404 additions and 82 deletions
+12 -6
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-29
verified_on: 2026-10-29
review_after: 2026-11-03
updated: 2026-10-30
verified_on: 2026-10-30
review_after: 2026-11-06
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks
## Count
- **Current as of 2026-10-29 00:3x AWST (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real and the discovery stage still could not finish: stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.**
- **Current as of 2026-10-30 00:0x AWST (live `jobs.json` + `executions.db`) — the radar's two-stage split worked end to end for the first time, and the fleet had its first clean day in a week.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3**, and every `failure_streak` reads **0** — the three survivors are all stale re-runs of long-standing rows (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`). Window **2026-10-29 settled 176 `completed` / 0 `failed`** (+ this audit `running`) — the first zero-failure day since 2026-10-24; 2026-10-28 stands at **168 / 2**. **Unclosed incident rows 4 → 1:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving only the Facebook monthly's `0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report). `dca8482e4f76` (stage 1) ran **07:00:59 → 10:16:12** (**3 h 15 m 13 s**), closed **`completed`**, and wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok` — opening **no** new incident row (contrast the two consecutive stage-1 failures on 10-27 and 10-28). `5fc292ff0f3b` (stage 2) fired **10:13:46**, consumed the handoff **10:40:59**, closed **10:44:48 `completed`** and delivered the same second: **20 pages / 20 `page_ids` / 2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went **455 → 475**. Both later slots no-oped as designed (**11:30 → 10 m 46 s**, **14:30 → 1 m 51 s**). Stage 1 correctly no longer delivers. See [[incidents/2026-10-27-job-radar-budget-exhausted]] and [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], now both **resolved**. **Pins unchanged** — 19 of the 26 enabled jobs pin `custom:omni-101` (12 `smart-route`, 6 `free-stack`, 1 explicit).
- **Prior window — as of 2026-10-29 00:3x AWST *(superseded by the 2026-10-30 entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real, but stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.**
- **Prior window — as of 2026-10-28 00:2x AWST** (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. *(superseded by the 2026-10-29 entry above)*
- **Prior window — as of 2026-10-27 00:1x AWST *(superseded by the 2026-10-28 entry above)* (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`).
- **Current as of 2026-10-26 00:1x AWST** *(superseded by the 2026-10-27 entry above)* **(live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`).
@@ -51,8 +52,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) |
| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 |
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00 |
| Daily Marketing Job Radar — Write and Letters (stage 2) | `5fc292ff0f3b` | `30 11,14 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Created 2026-10-27 17:34** — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). `custom:omni-101` / `smart-route`; first fires 2026-10-27 17:35 and 17:38 both `completed`. See [[incidents/2026-10-27-job-radar-budget-exhausted]] |
| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | `local` (failures → topic 1030) | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00. ✅ **First full split run 2026-10-29:** 07:00:59 → 10:16:12 (3 h 15 m 13 s), `completed`, wrote `handoff/2026-10-29.json` (170 discovered / 33 duplicates avoided / 37 verified); both of its ledger rows closed `resolved` at 10:16:12. **Delivery is now `local`** — stage 2 owns the report |
| Daily Marketing Job Radar — Write and Letters (stage 2) | `5fc292ff0f3b` | `30 11,14 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Created 2026-10-27 17:34** — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). `custom:omni-101` / `smart-route`; first fires 2026-10-27 17:35 and 17:38 both `completed`. ✅ **First full split run 2026-10-29:** fired 10:13:46, consumed the handoff at 10:40:59, closed `completed` 10:44:48 and delivered the same second — **20 pages + 2 cover letters, `missing_cover_letter_count 0`**, Notion board 455 → 475; the 11:30 and 14:30 slots then no-oped correctly. See [[incidents/2026-10-27-job-radar-budget-exhausted]] |
| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant |
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] |
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
@@ -97,6 +98,11 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
## Known Issues
### Resolved 2026-10-29
- **The Job Radar's two-stage split finished end to end for the first time (`dca8482e4f76` + `5fc292ff0f3b`):** stage 1 ran **07:00:59 → 10:16:12** (3 h 15 m 13 s), closed **`completed`** and wrote `handoff/2026-10-29.json` (**170 discovered / 33 duplicates avoided / 37 verified**) — the first post-split run to write a handoff, after 10-27 died on the **tool-call budget** and 10-28 on the **iteration cap**. Stage 2 consumed it at **10:40:59**, closed `completed` at **10:44:48** and **delivered** to topic 1030 in the same second: **20 pages, 2 cover letters, `missing_cover_letter_count 0`**, Notion **455 → 475**. Both stage-1 ledger rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12**, and the run opened no new row → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], [[runbooks/job-radar]]. ⚠ The discovery stage is still **slow** (3 h 15 m) and both ceilings remain reachable — a `completed` stage-1 row with **no handoff file** is the same fault in new clothing.
- **Conduit voice calls — three faults in series, all fixed and verified by two real calls:** the trunk speech-server URL was truncated (`…/v1/` → **403**, blamed on the key), the dashboard session TTL was the **12 h default** (expired sessions answer **WS 4401** with no error on the phone), and **both** host providers were dead (`tts.provider: openai` → Mac `/v1/speech` **502**; `stt.provider: groq` → **403/Cloudflare 1010**). Repaired 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL → **2 592 000 s**, `stt.provider: local`, `tts.provider: ollama_local`; dashboard restarted **08:03:26**. The path was driven end to end, then **calls at 08:38:04 and 08:49:36** — the first entries in `conduit-calls.json → history`. ⚠ Two TTL keys exist and only one was raised → [[incidents/2026-10-29-conduit-voice-silence]], [[systems/conduit]].
### Resolved 2026-10-12
- **Both `deepseek-v4-flash` incidents closed by a clean scheduled run (`fad5a86979d4`, 03:00):** the Facebook Birthday Weekly Registry Sync completed and delivered on 2026-10-12 (03:00:45 → 03:05:47), and incidents `fad5a8_a13bbc837e1c` (open since 2026-09-28, every fallback provider exhausted) and `fad5a8_c80b59a907f0` (since 2026-09-30, `deepseek-v4-flash` HTTP 400) **both auto-closed at 03:05:39**. Its stale pre-repin `last_status: error` cleared with them, after carrying it for two weeks — the repin is now verified by a real scheduled run, not only by the manual one.
- **`dca8482e4f76` timeout incident closed (`dca848_23fcd2b18aaa`):** the Daily Marketing Job Radar ran on schedule 2026-10-12 07:00:47 → 07:33:50 and delivered to topic 1030; the 2026-10-09 timeout incident that belonged to the provider storm **auto-closed at 07:33:41**. Of the ten storm incidents, nine have now closed and **one remains**: `1b23c0_c35703f637b8` (`1b23c042e17d`, still `alerted`, next slot Fri 2026-10-16 08:00). See [[incidents/2026-10-09-cron-provider-timeout-storm]].