audit(2026-10-26): runaway Gmail digest + dead Nous credential; model default reverted to OmniRoute; conduit 0.69.2 live; freshness floor still 2026-10-08
This commit is contained in:
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-25
|
||||
verified_on: 2026-10-25
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
review_after: 2026-10-31
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page.
|
||||
- **Current as of 2026-10-26 00:1x AWST (live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`).
|
||||
- **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page. *(Superseded by the 2026-10-26 entry above.)*
|
||||
- **Current as of 2026-10-24 00:2x AWST *(superseded by the 2026-10-25 entry above)* (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]]
|
||||
- **Current as of 2026-10-23 00:1x AWST *(superseded by the 2026-10-24 entry above)* (`jobs.json.updated_at 2026-10-23T00:07:25+08:00`) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. **A DB recount by `started_at` and by `claimed_at` puts 2026-10-21 at 168 / 1**, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Five incidents closed in-window and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (`0e519874d280`, closed 08:01:40), `06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so **unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`: `1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, `c0a9fa_764558b5ae83`, `0e753e_e7f8194a7d06`). **Non-clean `last_status` falls 9 → 5:** `1b23c042e17d` (`error`, streak 2 — **next slot Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` (`delivery_failed` — Mon 10-26 03:00), `066bb5191280` (`delivery_failed` — 11-01). All four cleared rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its first slot of 10-22. **Pins:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (`provider`/`model` null → global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). ⚠ **Stocktake anomaly:** `1a8e90ea2271` ran **03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, finished `completed` with delivery `suppressed` — and **committed nothing**. It left its regenerated `systems/skills-index.md` uncommitted (mtime 10-22 03:00, with a `## .Archive (41)` section reading **347**) and its diagnosed fixes unapplied → [[incidents/2026-08-15-skills-index-drift]]. **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop` (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models `mimo-v2.5`, `or-inclusionai_ling-2_6-1t`, `minimax/minimax-m3.1-flash`), 2 `acp` (0 messages), 1 `oneshot` memory-capture probe. → [[systems/skills-index]], [[incidents/2026-08-15-skills-index-drift]]
|
||||
- **Current as of 2026-10-22 00:1x AWST *(superseded by the 2026-10-23 entry above)* (`jobs.json.updated_at 2026-10-22T00:11:31+08:00`) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen.** Window 2026-10-21: **171 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`), which died at 07:40:48 after **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string, which appears in no 2026-10-21 row → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` stays at 9** (7 `error` / 2 `delivery_failed`) but the composition moved: **`08eb86f02d37` cleared** (its 08:30 slot ran clean; incident `08eb86_472c7b609c80` **resolved 2026-10-21 08:37:21**) while **`dca8482e4f76` re-entered** (streak 1). Survivors, with next run: `dca8482e4f76` (1, **10-22 07:00**), `0e519874d280` (2, **10-22 08:00**), `06a0a1c8704f` (1, **10-22 10:00**), `c8d52904e1d6` (1, **10-22 18:00**), `1b23c042e17d` (2, Fri 10-23 08:00), `c0a9fa314542` (1, Mon 10-26 09:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 11-01 02:00). **Incident ledger: 1 opened / 1 closed, unclosed stays 9** (8 `alerted` / 1 `detected`). **Pins:** 17 of the 28 jobs pin `custom:omni-101` (10 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`), **7 carry no pin at all** (`provider`/`model` null → global default), 1 pins `custom:omni` / `smart-route`. **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. **12 sessions in the window, all `cron` — no interactive sessions.** ⚠ **Infrastructure changed in-window: Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]], [[infrastructure/backups]].
|
||||
|
||||
Reference in New Issue
Block a user