diff --git a/current-state.md b/current-state.md index 29cd0e8..74e1bbf 100644 --- a/current-state.md +++ b/current-state.md @@ -24,15 +24,20 @@ Updated: 2026-11-05 from Stage 1's candidate files**, re-verified every candidate live in Camofox, and wrote **5 roles (board 479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at 16:08:59 - see [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. -- **Fleet: 158 jobs (144 enabled / 14 disabled).** In-window ledger: **170 rows - 167 `completed` / 2 `failed` / - 1 `item_suppressed`**. **Non-clean `last_status` 2 -> 3**; **unclosed incident rows 1 -> 3** (all +- **Fleet: 29 jobs (26 enabled / 3 disabled).** In-window ledger: **170 rows - 167 `completed` / 2 `failed` / + 1 `running`**. *(fleet and last row corrected 2026-11-05 01:3x - the first draft read 158 jobs / 144 / 14 and a + closing `item_suppressed` row; live `jobs.json` + `hermes cron list` give 29 / 26 / 3, and the 170th 11-04 row is + this audit, `running`. 11-03 has since settled to 171 `completed` / 1 `failed`.)* **Non-clean `last_status` 2 -> 3**; **unclosed incident rows 1 -> 3** (all `state = alerted`, incl. the 11-03 `timeout` row `dca848_dfd4bea84f2a`). Deliveries on 11-04: **8 rows, 8 `delivered`** - the failure notices were delivered, not silent. - **The cover-letter route is only alive by override.** `regenerate_cover_letter.py` line 82 reads `CL_MODEL`, defaulting to **`deepseek-paid-smart`**, which returned **HTTP 402** during the 11-04 run; that process was pinned to **`gemini/gemini-3.5-flash`**. **No `CL_MODEL` exists in `~/.hermes/.env`**, so the override did not persist. Endpoint check 2026-11-05: `omniroute:20128` -> **401** and `100.78.63.20:20128` -> **401** (live, - key-gated), and the same model still serves other jobs - so re-test before calling the route dead. + key-gated - that is auth, not health). **Keyed re-probe 2026-11-05 01:33:12 AWST:** a completion request with + `deepseek-paid-smart` returns **503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`) on `omniroute:20128`, + but **200** on `100.78.63.20:20128` (`omniroute-1`), which resolved the alias to **`deepseek/deepseek-v4-flash`**. + So the model is **undispatchable on the CT600 node only** - an override or the `omniroute-1` route is required. - **CT299 froze a second time.** A manual `vzdump 299 ... --storage ssd0` started **2026-11-04 08:30:04** and wedged; the guest was frozen until **~00:30 on 11-05** (~16 h). Live probe **2026-11-05 00:54:21 AWST**: `LOCK: 0`, `cgroup.freeze=0`, `ns/cgroup.freeze=0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump processes; diff --git a/index.md b/index.md index b2fbb4a..577a825 100644 --- a/index.md +++ b/index.md @@ -11,9 +11,12 @@ tags: [index, infrastructure, systems, stream-failure, ct299] # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-11-05 (nightly audit - the 11-04 fleet settled 170 rows: 167 `completed` / 2 `failed` / 1 `item_suppressed`; non-clean `last_status` 2 -> 3; unclosed incident rows 1 -> 3; deliveries 8/8 `delivered`. Both 07:00 failures were one upstream stream fault - and the Job Radar's Stage 2 slot reconstructed the missing handoff and saved the day. CT299 froze a second time.) +> model routing and automation. Last updated: 2026-11-05 (nightly audit - the 11-04 fleet settled 170 rows: 167 `completed` / 2 `failed` / 1 `running`; non-clean `last_status` 2 -> 3; unclosed incident rows 1 -> 3; deliveries 8/8 `delivered`. Both 07:00 failures were one upstream stream fault - and the Job Radar's Stage 2 slot reconstructed the missing handoff and saved the day. CT299 froze a second time.) > -> **2026-11-05 (nightly audit) - a dead route, a rescued pipeline, and a second freeze:** the **11-04 fleet settled 170 rows - 167 `completed` / 2 `failed` / 1 `item_suppressed`**, **non-clean `last_status` 2 -> 3**, **unclosed incident rows 1 -> 3**, **deliveries 8 rows / 8 `delivered`**. Both failures are the *same* upstream fault: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**) died **36 seconds apart at 09:41-09:42**, each after **2 h 41 m**, on `RuntimeError: Upstream stream failed before completion.` - both pinned `custom:omni-101` / `smart-route`. **The day was not lost:** the 11:30 Stage 2 slot **reconstructed the missing handoff from Stage 1's own candidate files**, re-verified every candidate live, and still wrote **5 roles (board 479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at 16:08:59 -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. The cover-letter route survived only on an override: `regenerate_cover_letter.py` defaults to `deepseek-paid-smart`, which threw **HTTP 402** mid-run, so the letter was pinned to `gemini/gemini-3.5-flash` via `CL_MODEL` - and that override **was never written to `.env`**, so it did not persist. And **CT299 was frozen twice, not once**: a second, **manual** `vzdump 299 ... --storage ssd0` wedged from **11-04 08:30:04** until **11-05 ~00:30** (~16 h); the live probe at 00:54:21 AWST shows `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. +> **2026-11-05 (nightly audit) - a dead route, a rescued pipeline, and a second freeze:** the **11-04 fleet settled 170 rows - 167 `completed` / 2 `failed` / 1 `running`**, **non-clean `last_status` 2 -> 3**, **unclosed incident rows 1 -> 3**, **deliveries 8 rows / 8 `delivered`**. Both failures are the *same* upstream fault: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**) died **36 seconds apart at 09:41-09:42**, each after **2 h 41 m**, on `RuntimeError: Upstream stream failed before completion.` - both pinned `custom:omni-101` / `smart-route`. **The day was not lost:** the 11:30 Stage 2 slot **reconstructed the missing handoff from Stage 1's own candidate files**, re-verified every candidate live, and still wrote **5 roles (board 479 -> 484) + 1 canonical cover letter** (`MISSING_COVER_LETTER_COUNT 0`) before consuming the handoff at 16:08:59 -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. The cover-letter route survived only on an override: `regenerate_cover_letter.py` defaults to `deepseek-paid-smart`, which threw **HTTP 402** mid-run, so the letter was pinned to `gemini/gemini-3.5-flash` via `CL_MODEL` - and that override **was never written to `.env`**, so it did not persist. And **CT299 was frozen twice, not once**: a second, **manual** `vzdump 299 ... --storage ssd0` wedged from **11-04 08:30:04** until **11-05 ~00:30** (~16 h); the live probe at 00:54:21 AWST shows `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. ⚠ **Corrected the same night:** the first draft of this entry read the fleet as **158 jobs (144 / 14)** and the +> 11-04 ledger as closing `1 item_suppressed`; live reads give **29 jobs (26 / 3)** and a final row of **1 `running`** +> (this audit). Keyed probe **2026-11-05 01:33**: `deepseek-paid-smart` is **undispatchable on `omniroute:20128`** +> (HTTP **503**, all targets skipped) but **serves normally on `omniroute-1`** (HTTP **200** -> `deepseek/deepseek-v4-flash`). > **2026-11-04 (nightly audit) — the ledger raised its own red row:** the **11-03 fleet settled **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) — **unclosed incident rows 0 → 1**, non-clean `last_status` **1 → 2**, deliveries **7 rows / 7 `delivered`**. The single failure is the **Job Radar Stage 1** (`dca8482e4f76`), which ran **07:00:33 → 13:28:52 (6 h 28 m 18 s)** and died on **`RuntimeError: stream active timeout`** — a **new terminal signature**, not the iteration cap — after going **silent for 65 m 21 s**; its incident `dca848_dfd4bea84f2a` (`timeout`) is **`alerted` and open** and the notice **was delivered** (13:28:51.741724) → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. That answers the 11-02 page's open question: **output is suppressed when the run ends `completed`, delivered when it ends `failed`** — which is what let 11-02 vanish and 11-03 announce itself. Both Stage 2 slots declined correctly, so **2026-11-02's and 2026-11-03's candidate sets are both unrecovered**. ⚠ **The 03:00 stocktake overran for a second day** (`1a8e90ea2271`, 03:00:30 → 10:41:13 = **7 h 40 m 43 s**, `suppressed`, nothing delivered, artefacts uncommitted again). **New page:** [[systems/hermes-hq-push]] (v0.9.27, enabled; manifest says it **replaces `dispatch-push`**, and no trace of the old plugin survives). > diff --git a/log.md b/log.md index c0e03c7..880bb87 100644 --- a/log.md +++ b/log.md @@ -1978,7 +1978,7 @@ embedded credential was seen during verification and is not reproduced here ([RE 00:54:21 AWST: `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump procs. - **Updated [[infrastructure/tailscale]]** - the sftpgo row now records recovery from *two* freezes. - **Updated [[systems/scheduled-tasks]]** - new ledger bullet: 11-04 settled **170 rows - 167 `completed` / - 2 `failed` / 1 `item_suppressed`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3**; + 2 `failed` / 1 `running`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3**; deliveries **8/8 `delivered`**; the two failures are one upstream signature; stocktake overran for the third day (5 h 41 m 46 s, `free-stack` pin). - **Updated [[runbooks/job-radar]]** - cover-letter model route: `regenerate_cover_letter.py` defaults to @@ -1998,3 +1998,32 @@ embedded credential was seen during verification and is not reproduced here ([RE decision reversed. **No secrets, passwords, tokens or keys written or quoted.** - *network-map.md re-verified (live probe 2026-11-05):* corrected the ct460 port table — **9377 is Camofox browser REST** (`/home/hermes/services/camofox-browser`), not Stratos; `8790` and `5174` are no longer listening; added `9120` (wiki serve), `9222` (Camofox CDP/WS), `7777` (OpenChamber), `6767` (Paseo), `8085` (AgentMail webhook), `9900` (gateway). Added the full Tailscale Serve table (12 routes on `:443`, `/` on `:9443` → 5173, Funnel `:8443` `/webhook` → 8085). Two routes are dead: `/agent-estate` → 8769 and `/synergy-prep` → 9123 have no listener. +## [2026-11-05] correction + live probe | fleet count, 11-04 ledger composition, `deepseek-paid-smart` dispatch + +- **What was wrong.** The 2026-11-05 nightly-audit entry (written 00:5x AWST) recorded the fleet as **158 jobs + (144 enabled / 14 disabled)** and the 11-04 ledger as closing **1 `item_suppressed`**. Both were wrong. Live + re-reads - `~/.hermes/cron/jobs.json`, `hermes cron list`, and the in-flight job count - each give **29 jobs / + 26 enabled / 3 disabled**; `executions.db` shows 11-04 at **170 rows - 167 `completed` / 2 `failed` / 1 + `running`** (this audit), with **no `item_suppressed` status anywhere on 11-04**. +- **Where corrected:** `systems/scheduled-tasks.md` (Count bullet + correction clause), + `current-state.md` (11-05 block), `index.md` (11-05 ledger paragraph), and this entry. No other page carried + the bad number. +- **11-03 re-settle:** `171 completed / 1 failed` (172 rows) - its `running` row has since completed - so 11-03 + and 11-04 are the only consecutive days with a `failed` row since 10-31. +- **Live probe - model dispatch, 2026-11-05 01:33:12 AWST (first-hand, keyed):** + `POST http://omniroute:20128/v1/chat/completions` with `deepseek-paid-smart` -> **HTTP 503** + `ALL_TARGETS_SKIPPED` (`poolSize: 5`, `attempted: 0`). The same request to + `http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, `model: + deepseek/deepseek-v4-flash`. Unkeyed `/v1/models` on both hosts returns **401** - auth-gating, not health. The + fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101`, i.e. + the `omniroute-1` node), which is why the audit still completes while a direct CT600 probe 503s. +- **Re-verified and matching (no change needed):** `cron_incidents` - 3 unclosed rows, all `state = alerted`; + `deliveries.db` - 8 rows on 11-04, **8/8 `delivered`**, including both `for_failure` notices (09:41:43, + 09:42:20); non-clean `last_status` **3**; pin distribution **14 `smart-route` / 6 `free-stack` / 8 unpinned / + 1 `deepseek-paid-smart`**. +- **Files touched this pass:** `systems/scheduled-tasks.md`, `current-state.md`, `runbooks/job-radar.md` + (`CL_MODEL` caveat now carries the keyed probe), `index.md`, `log.md`. Frontmatter `updated` / `verified_on` + bumped to 2026-11-05 where stale. +- **Not done / unchanged:** no job added, removed, disabled or re-pinned; no page removed or archived; no + decision reversed. **No secrets, passwords, tokens or keys written or quoted** - the API key was read from + `~/.hermes/.env` into the shell for the probe only. diff --git a/runbooks/job-radar.md b/runbooks/job-radar.md index d15e439..332c75c 100644 --- a/runbooks/job-radar.md +++ b/runbooks/job-radar.md @@ -47,7 +47,11 @@ Cron jobs inherit the profile-level `fallback_providers` chain; Hermes does not > process and confirm the letters rendered - do **not** ship a run without its cover letters. Note the same model > name still serves other cron jobs successfully (e.g. the 23:59 nightly audit), and the endpoint itself is live: > `omniroute:20128/v1/models` -> **401** and `100.78.63.20:20128/v1/models` -> **401** on 2026-11-05 (key-gated, -> not down). Treat a 402 as **route/time-specific until re-tested**, not as proof the route is dead. +> not down). Treat a 402 as **route/time-specific until re-tested**, not as proof the route is dead. **Keyed re-test 2026-11-05 01:33:12 AWST:** `deepseek-paid-smart` -> **503 `ALL_TARGETS_SKIPPED`** +> (`poolSize 5`, `attempted 0`) on `omniroute:20128`, but **200** on `100.78.63.20:20128` (`omniroute-1`, resolved to +> `deepseek/deepseek-v4-flash`). As of that probe the default `CL_MODEL` **cannot dispatch on the CT600 node**, so a +> `CL_MODEL` override (or an `omniroute-1` base URL) is required for cover-letter runs. An unkeyed `/v1/models` 401 is +> auth-gating, not health. A successful radar run includes the cover-letter stage. diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 3930367..4685cdd 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -14,7 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db # Scheduled Tasks ## Count -- **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **158 jobs (144 enabled / 14 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `item_suppressed`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. +- **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. -> [[runbooks/job-radar]] `CL_MODEL` caveat. - **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since 10-31**, and