diff --git a/comparisons/steel-vs-camoufox.md b/comparisons/steel-vs-camoufox.md index 7a8c4b1..0523d95 100644 --- a/comparisons/steel-vs-camoufox.md +++ b/comparisons/steel-vs-camoufox.md @@ -3,8 +3,8 @@ title: Steel Browser vs Camoufox type: comparison status: active created: 2026-07-29 -updated: 2026-10-13 -verified_on: 2026-10-13 +updated: 2026-10-29 +verified_on: 2026-10-29 confidence: high tags: [comparison, browser, steel, camofox, decision] sources: [live-probe, session-history] @@ -26,7 +26,7 @@ Compare the two browser-automation backends available to Hermes, including the J | **Container** | Docker on CT201 | `systemd --user` unit on the Hermes host (CT460) | | **Stealth** | Standard Chromium | Anti-detection fingerprinting | | **Session viewer** | Built-in UI at `/steel` | VNC at `/vnc-camofox` | -| **Plugin** | `browser-steel` plugin (disabled) | Native Hermes `browser.cloud_provider: camofox` | +| **Plugin** | `browser-steel` — **removed 2026-10-29** | Native Hermes `browser.cloud_provider: camofox` | ## The Steel Experiment (Jul 29–30) @@ -48,7 +48,7 @@ Compare the two browser-automation backends available to Hermes, including the J ### Lesson learned Steel is a capable backend but adds a network hop and another plugin layer. Camofox runs locally on the -Hermes host (CT460) with fewer failure modes. The `browser-steel` plugin still exists (disabled) if needed in future. +Hermes host (CT460) with fewer failure modes. The `browser-steel` plugin was **removed on 2026-10-29** after the endpoint went dark, so returning to Steel means redeploying the plugin, not flipping a flag. ## Detail @@ -62,11 +62,11 @@ Hermes host (CT460) with fewer failure modes. The `browser-steel` plugin still e ### Steel Browser (disabled but available) - **Deployed:** 2026-07-29 on CT201, Docker, custom `steel-browser-api:fixed` image -- **Status:** Still running on CT201, Hermes not connected +- **Status:** **Offline** (verified 2026-10-29 00:3x) — the host answers ping in 2.5 ms, but CDP `:9223` and REST `:3000` refuse, the tunnel route `/steel` returns **502** and `steel.martinwa.org` no longer resolves - **CDP endpoint:** `http://100.96.244.39:9223` - **REST API:** `http://100.96.244.39:3000` - **UI:** `https://runtipi.kangaroo-eel.ts.net/steel` -- **Plugin:** `browser-steel` at `/home/hermes/.hermes/plugins/browser-steel/` — disabled, `plugin_steel` toolset flagged `requires_health=steel` +- **Plugin:** **removed 2026-10-29 00:13** — `~/.hermes/plugins/browser-steel/` is gone and `plugins.enabled` has no steel entry, so the `plugin_steel` toolset can no longer be selected ## Related - [[systems/browser-backend]] diff --git a/current-state.md b/current-state.md index 5251ea8..a5b823d 100644 --- a/current-state.md +++ b/current-state.md @@ -3,8 +3,8 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-28 -verified_on: 2026-10-28 +updated: 2026-10-29 +verified_on: 2026-10-29 review_after: 2026-11-03 confidence: high tags: [system] @@ -45,6 +45,60 @@ Updated: 2026-10-28 `systems/skills-index.md` (regenerated 10-27, committed), `index.md`, `log.md`. **Nothing removed or archived.** No secrets written. +## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 22:00 – 2026-10-29 00:3x AWST) + +### The radar's two-stage split ran for real, and landed nothing — stage 1 hit the agent iteration cap + +`dca8482e4f76` (stage 1, Discovery) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**) and closed **`failed`** on +*"hit the agent iteration cap before the handoff could be written"* — **`failure_streak: 2`**. Stage 1 never +produced a handoff (`~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`), so the *first* real two-stage +cycle delivered **no Notion writes and no cover letters** — the same outcome as 10-27, by a different route. + +Both stage-2 slots then did exactly what they were built to do: `5fc292ff0f3b` fired **11:30:27 → 11:31:31** and +**14:30:27 → 14:31:43**, both `completed` with **`delivery_outcome: suppressed`** — *a stage-2 run with no +handoff is a silent no-op, not a failure* (the recovery rule in [[runbooks/job-radar]]), now proven in +production. The two post-split failures also stopped at different ceilings: 10-27 at the **tool-call budget** +(4 h 2 m, writes not started), 10-28 at the **iteration cap** (3 h 32 m, handoff not written) — less of the +pipeline completed despite the shorter wall clock. Open radar incident rows: `dca848_e93a538ade6b` (10-27) and +`dca848_0a2f03b5952b` (10-28), both open. Detail: +[[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. + +### The 03:00 stocktake failed on a provider 400 — and its incident row never alerted + +`1a8e90ea2271` ran **03:00:28 → 03:37:58** (37 m 30 s) and closed **`failed`** (streak 1, delivery +`suppressed`) with `RuntimeError: [400] Failed to deserialize the JSON body into the target type: duplicate +field 'reasoning_content' at line 1 column 512203` against `custom:omni-101 / free-stack` — the *response* the +provider was asked to deserialize carried a duplicated `reasoning_content` field. It performed its rebuild +(302 skills → `systems/skills-index.md` mtime 03:01:40; freshness report `raw/stocktake/freshness-20261028.txt` +→ **2 stale / 103 fresh / 4 no-date of 109 pages**) and then died **before the commit**, leaving the wiki tree +dirty for the rest of the day. Its incident row `1a8e90_f57ab4cafab8` sits in state **`detected`** with **no +alert** — the only outward sign was the job's own `error` status. The 2026-10-29 audit committed the two +artefacts. + +### Steel left the stack: Camofox restored, plugin removed, endpoint dark + +`browser.cloud_provider` and `browser.camera_provider` are back to **`camofox`** (`engine: auto`, `cdp_url: ""`) +in `config.yaml` (mtime **2026-10-29 00:15:13**), with backups taken at every step +(`config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z` 22:00:57, and +`config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24). The **`browser-steel` plugin was +removed** from `~/.hermes/plugins/` and from `plugins.enabled`, so the `plugin_steel` toolset is no longer +selectable. Steel itself is **not serving**: the CT201/runtipi host answers ping (2.5 ms) but CDP `:9223` and +REST `:3000` refuse (also on the LAN address), the tunnel route `/steel` returns **502**, and +`steel.martinwa.org` no longer resolves — superseding the earlier "still running on CT201" notes. ⚠ +`STEEL_BASE_URL=http://100.96.244.39:3000` remains exported into **10 live Hermes processes** and clears only on +a gateway restart. Detail: [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], +[[comparisons/steel-vs-camoufox]]. + +### Fleet and tallies + +29 jobs (**26 enabled / 3 disabled**) — unchanged. Non-clean `last_status`: **5** (`0e753e9720fd` error, +`fad5a86979d4` delivery_failed, `dca8482e4f76` error **streak 2**, `066bb5191280` delivery_failed, +`1a8e90ea2271` error **streak 1**). Pins: **12 `smart-route` / 6 `free-stack` / 7 unpinned / +1 `deepseek-paid-smart`**. **2026-10-28 settled 168 completed / 2 failed / 1 running**; **2026-10-29** so far +6 completed / 1 running; **2026-10-27 settled 173 / 1** (revised up by one late run after the 2026-10-28 +audit). No Hindsight service is running on this box (`:8888` closed, no `hindsight` units) — the standing +memory-backend correction still holds and needs no change. + ## Recent Changes (2026-10-27, nightly audit; live probes 2026-10-27 00:1x–00:5x AWST) - **A *planned* gateway restart still orphaned a live cron run — and this time it alerted loudly.** A planned diff --git a/decisions/rollback-steel-to-camofox.md b/decisions/rollback-steel-to-camofox.md index 3204a01..247532c 100644 --- a/decisions/rollback-steel-to-camofox.md +++ b/decisions/rollback-steel-to-camofox.md @@ -3,8 +3,8 @@ title: Rollback Steel to Camoufox type: decision status: active created: 2026-08-02 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-10-29 +verified_on: 2026-10-29 confidence: high tags: [decision, browser, steel, camofox, rollback] sources: [session-history, live-config] @@ -59,6 +59,21 @@ Steel Browser was deployed on CT201 on 2026-07-29 as the primary browser automat - Uber Eats report (Aug 1) successfully scraped via Camofox REST API — 11 orders, $264.51 - Steel plugin `browser-steel` still present at `/home/hermes/.hermes/plugins/browser-steel/` — disabled +## Update (2026-10-29) — the plugin was removed and the endpoint went dark + +The 2026-07-30 rollback disabled the Steel plugin but left it on disk and left Steel running. Both of those +were closed on **2026-10-28 22:00 – 2026-10-29 00:15**: `browser.cloud_provider` / `camera_provider` were +returned to **`camofox`** (backups `config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z`, +`config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z`), and **`~/.hermes/plugins/browser-steel/` was +removed**, so the `plugin_steel` toolset is no longer selectable. + +Steel itself is **not serving**: CDP `:9223` and REST `:3000` on `100.96.244.39` refuse, the tunnel route +`/steel` returns **502**, and `steel.martinwa.org` does not resolve, while the host still answers ping. +`STEEL_BASE_URL=http://100.96.244.39:3000` remains exported in 10 live Hermes processes and clears on the next +gateway restart. The decision below is unchanged — Camofox is the backend — but there is now **no Steel plugin +fallback in the tree**, so returning to Steel would mean redeploying it. Detail: +[[systems/browser-backend]], [[comparisons/steel-vs-camoufox]]. + ## Related - [[comparisons/steel-vs-camoufox]] - [[systems/browser-backend]] diff --git a/incidents/2026-10-27-job-radar-budget-exhausted.md b/incidents/2026-10-27-job-radar-budget-exhausted.md index e5da7ec..dc22d75 100644 --- a/incidents/2026-10-27-job-radar-budget-exhausted.md +++ b/incidents/2026-10-27-job-radar-budget-exhausted.md @@ -3,8 +3,8 @@ title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion type: incident status: active created: 2026-10-28 -updated: 2026-10-28 -verified_on: 2026-10-28 +updated: 2026-10-29 +verified_on: 2026-10-29 review_after: 2026-11-04 confidence: high tags: [automation, cron, jobs, notion, silent-failure, scheduling] @@ -85,6 +85,22 @@ Fleet size went **28 → 29 jobs** (26 enabled / 3 disabled). 4. The incident row stays open until a clean stage-1 slot auto-closes it — confirm the 2026-10-28 07:00 slot closes `dca848_e93a538ade6b`. +## Follow-up — the 2026-10-28 slot (recorded 2026-10-29) + +The **first real two-stage cycle ran on 2026-10-28** and answered the open questions above — not favourably: + +- **Stage 1 overran again, on a new signature.** `dca8482e4f76` ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), + closed **`failed`**, `failure_streak` **2**, on *"hit the agent iteration cap before the handoff could be + written"* → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. +- **No handoff file was written** — `~/.hermes/radar-ref/handoff/` held only `2026-10-27.json` at the 2026-10-29 + audit, so **Notion writes and cover letters did not run again**. +- **Both stage-2 slots no-oped correctly** — `5fc292ff0f3b` fired **11:30:27 → 11:31:31** and + **14:30:27 → 14:31:43**, both `completed` with `delivery_outcome: suppressed`. Open question 1 is therefore + answered only in the negative: the recovery rule ("a stage-2 run with no handoff is a no-op") holds in + production, but nothing has yet proved the positive path end-to-end. +- **Open question 4 is answered: the row did not close.** `dca848_e93a538ade6b` stayed open because the 10-28 + slot was not clean, and a second row (`dca848_0a2f03b5952b`) opened — open incident rows **2 → 4**. + ## Related - [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun diff --git a/incidents/2026-10-28-job-radar-stage1-iteration-cap.md b/incidents/2026-10-28-job-radar-stage1-iteration-cap.md new file mode 100644 index 0000000..25c4994 --- /dev/null +++ b/incidents/2026-10-28-job-radar-stage1-iteration-cap.md @@ -0,0 +1,84 @@ +--- +title: Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped +type: incident +status: active +created: 2026-10-29 +updated: 2026-10-29 +verified_on: 2026-10-29 +review_after: 2026-11-05 +confidence: high +tags: [automation, cron, jobs, notion, silent-failure, scheduling] +sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir] +--- + +# Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped + +## Headline + +On **2026-10-28** the Discovery stage of the Daily Marketing Job Radar (`dca8482e4f76`) ran +**07:00:32 → 10:33:18 AWST** (**3 h 32 m 46 s**) and closed **`failed`** on *"hit the agent iteration cap before +the handoff could be written"* — its **second consecutive failure** (`failure_streak: 2`). + +This is the first real cycle of the **two-stage split** introduced on 2026-10-27 (see +[[incidents/2026-10-27-job-radar-budget-exhausted]] and [[runbooks/job-radar]]), and it produced a clean negative +result: **stage 1 never wrote its handoff file, so there was nothing for stage 2 to land.** Both stage-2 slots +ran anyway — **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed`, both +`delivery_outcome: suppressed` — which is the designed no-op path working correctly in production for the first +time. Notion ingestion and cover letters did not execute for the day. + +## Ledger state + +| Field | Value | +|---|---| +| Execution row | `dca8482e4f76`, started `2026-10-28T07:00:32.463309+08:00`, finished `2026-10-28T10:33:18.833788+08:00`, **`failed`** | +| Duration | **3 h 32 m 46 s** (10-27: 4 h 2 m 46 s; 10-26: 2 h 9 m 19 s) | +| `jobs.json` | `last_status: error`, **`failure_streak: 2`**; `last_error` opens *"Daily Marketing Job Radar — Stage 1 (Discovery) — 2026-10-28 — hit the agent iteration cap before the handoff could be…"*; pin unchanged (`custom:omni-101` / `smart-route`) | +| Incident row | `dca848_0a2f03b5952b` — `state: alerted`, `failure_type: timeout`, first_seen `2026-10-28T10:33:18`, **open** | +| Handoff | **not written** — `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json` (verified 2026-10-29 00:3x) | +| Stage 2 | `5fc292ff0f3b` fired `11:30:27 → 11:31:31` and `14:30:27 → 14:31:43`, both `completed` / `delivery_outcome: suppressed` | + +The 10-27 row (`dca848_e93a538ade6b`, classified `failure_type: auth`) did **not** close — the 2026-10-28 07:00 +slot was not clean. Open `cron_incidents` rows went **2 → 4**. + +## What the split proved, and what it did not + +1. **It proved the recovery rule.** *"A stage-2 run with no handoff is a no-op, not a failed search"* — exactly as + written in [[runbooks/job-radar]] — held in production: stage 2 invented no work, raised no error, and sent + nothing to topic 1030. +2. **It did not make discovery finish.** The two post-split runs died at *different* ceilings: + - **2026-10-27** — stopped at the **tool-call budget** after 4 h 2 m: discovery/verify/dedup/classify complete, writes never started. + - **2026-10-28** — stopped at the **agent iteration cap** after 3 h 32 m, *earlier in the pipeline*: no handoff, so nothing downstream could run at all. + The wall-clock number improving (4 h 2 m → 3 h 32 m) is **not** an improvement — less of the pipeline completed. +3. **The write stage is no longer the risk; the discovery stage still is.** Splitting removed the + "writes lost inside a green row" failure mode. What remains is that stage 1 cannot both finish discovery and + reach its handoff write inside its own limits. + +## Why it matters + +- **A lost day is now cheap to detect but easy to miss.** Stage 1 raises an incident row and flips `last_status` + to `error`; stage 2's *correct* silence (`suppressed`) means the only outward sign of a workless day is stage + 1's own failure report — not a "0 jobs added" report. +- **Two consecutive failures, two different classifiers.** 10-27 was tagged `failure_type: auth` (wrong for a + budget fault); 10-28 was tagged `timeout` (also only approximate — an agent-side iteration cap, not a network + timeout). Neither string names the actual cause. +- **Budget arithmetic is not the fix.** If the cap fires after ~3.5 h of discovery, the pipeline needs a smaller + discovery surface (fewer sources per slot), **staged/checkpointed handoff writes** (so a capped run still + leaves something landable), or an explicit per-phase iteration allowance. + +## Open questions + +1. **What is the iteration cap for this job, and can stage 1 reach its handoff write before it?** The 10-28 run + died *before* the write; the 10-27 run died *after* discovery but *before* writes. Neither produced a handoff. +2. **Should stage 1 checkpoint a partial handoff** (after DEDUP/SCORE) so a capped run gives stage 2 something to + land instead of a silent no-op day? +3. **Make the abort self-describing** — a distinct `iteration-cap` / `partial-run` failure type rather than + `auth` / `timeout`. +4. Both radar incident rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) close only when a **clean** stage-1 + slot runs — watch the next 07:00 weekday slot. + +## Related + +- [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up +- [[incidents/2026-10-27-job-radar-budget-exhausted]] — the prior day's budget exhaustion and the split +- [[systems/scheduled-tasks]] — fleet ledger and the 2026-10-28 window +- [[systems/browser-backend]] — the browser layer this run depends on diff --git a/index.md b/index.md index 017c5be..4928278 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,6 @@ # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-28 (nightly audit — 1 new incident page, 9 pages updated) +> model routing and automation. Last updated: 2026-10-29 (nightly audit — 1 new incident page, 9 pages updated) > > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. @@ -24,7 +24,7 @@ - [[systems/openrouter]] — OpenRouter current state and history - [[systems/opencode-go]] — OpenCode Go subscription, model limits, pricing - [[systems/nanobot]] — Nanobot current state and history -- [[systems/browser-backend]] — Camofox (primary, `systemd --user` on CT460, REST `:9377`) + Steel (disabled); memory-containment drop-in and the 2026-10-12 ceiling event documented +- [[systems/browser-backend]] — Camofox (primary, `systemd --user` on CT460, REST `:9377`); **Steel was removed from the stack on 2026-10-29** (plugin deleted, `cloud_provider` back to `camofox`, endpoint dark, `STEEL_BASE_URL` still exported in 10 live processes); memory-containment drop-in and the 2026-10-12 ceiling event documented - [[systems/hermes-ui]] — Browser Hermes Desktop (przbadu/hermes-ui) - [[systems/paseo]] — Paseo coding-agent daemon + web terminal workspace (CT460, :6767) - [[systems/ignis]] — Ignis browser-native Obsidian (wiki interface) @@ -35,10 +35,11 @@ - [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.69.2` (HEAD `942ddb8`, 2026-10-24 15:35 — *"Pulse: list candidates, decide in code, journal them (#342)"*; 24 commits on since the `0.64.0` baseline, **0 behind `origin/main`**)**, upstream `kaishi00/hermes-conduit-notifier`. **Gateway on `:9119` restarted 2026-10-25 18:42:33 — so the upstream commits are now LIVE** (the 10-24 "not live" note is superseded). One registered device (iPhone app `2.0.0`). **Still carries the uncommitted in-tree `dashboard/plugin_api.py` patch**; `conduit-calls.json → history` is **empty**, so the memory fix still has no positive call evidence, but `conduit_show` ran once on 10-25 14:37 (card delivered). ⚠ **Known failure: three 10-24 requests to be phoned were refused by the agent, which never called the tool; no recurrence since** → [[incidents/2026-10-24-conduit-call-request-refused]]. - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26** (peer offline **36 d**, `tx 312 / rx 0`; host answers neither `:8065` nor ICMP). **268** connect timeouts on 10-25 vs 264 on 10-24; the reconnect **attempt counter reset to 54** at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost. - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. -- [[systems/scheduled-tasks]] — **As of 2026-10-28 00:2x: 29 jobs (26 enabled, 3 disabled), 5 non-clean `last_status`, 2 unclosed incident rows; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** +- [[systems/scheduled-tasks]] — **As of 2026-10-29 00:3x: 29 jobs (26 enabled, 3 disabled), 5 non-clean `last_status`, 4 unclosed incident rows; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised)**. Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** - [[systems/skills-index]] — Installed skills inventory (**302 skills** (69 categories) as of the **2026-10-27** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at the 2026-10-27 run: **0 stale / 104 fresh / 4 no-date of 108 pages** (`freshness-20261027.txt`). The 2026-10-27 03:00 slot regenerated `systems/skills-index.md` but left it uncommitted in the authoring tree; the 2026-10-28 audit committed it. Note: `SKILL.md` files on disk still exceed the index (the generator de-dups one nested duplicate; see [[systems/scheduled-tasks]]). ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents +- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] — **The radar's two-stage split ran for real on 2026-10-28 and landed nothing: stage 1 stopped at the agent iteration cap before writing its handoff, so both stage-2 slots correctly no-oped (silent). Two consecutive stage-1 failures, two different ceilings (tool-call budget 10-27, iteration cap 10-28); no Notion writes for a second day.** `status: active`, review 2026-11-05. - [[incidents/2026-10-27-job-radar-budget-exhausted]] — **The Daily Marketing Job Radar exhausted its tool-call budget for the second day running — and this time the run is recorded as `failed`, not `completed`.** On 2026-10-27 the 07:00 slot ran **07:01:08 → 11:03:54** (4 h 2 m 46 s) and its delivered report opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"*: discovery/verify/dedup/classify finished, **Notion writes, cover letters and the final audit never ran** (Notion DB held 451 entries, 0 added for the day). Incident row `dca848_e93a538ade6b` opened 11:03:53 (`alerted` 11:03:59, `failure_type: auth` — wrong for a budget fault) and was still open at the 2026-10-28 audit. **Response the same day:** the radar was split into stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) and new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030), created 17:34:15, two `completed` test fires that evening → [[runbooks/job-radar]] - [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **A *planned* gateway restart orphaned a live cron run — the second instance of the orphan class, and the first that alerted.** On 2026-10-26 at **14:33:35 AWST** the gateway took a planned `SIGTERM` (`under_systemd=yes`); the shutdown drain reported **`timed_out=True` after just 1.35 s** of its 30 s budget and interrupted `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) **2 h 34 m** in. It closed **`failed`** on a **new** string — `lost its durable fire claim ownership` — raised **two** `cron_incidents` rows (14:33:37 *unverified / reconcile admission* + 14:35:59 *fire claim*), **delivered** the failure at 14:35:59, and left the job on **`failure_streak: 2`** from one interrupted fire. Two Mnemosyne Watchdog reds followed at 14:46 and 15:01 and self-resolved at **15:16:12**. Net = **169 / 3** for 10-26. Unresolved: why a **planned** stop bypasses the drain deadline, and whether `failure_streak` should count the unverified row. → also updates [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] (the class now recurs without a crash). - [[incidents/2026-10-25-gmail-digest-runaway-and-refire]] — **The Gmail Action Digest ran 1 h 55 m and fired six times on 2026-10-25.** The 08:30 slot started 08:31:14 and closed **10:26:37 `failed`** on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed)` — against **91 s** (10-23) and **104 s** (10-24). The same job then fired five more times (17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53); the 19:18:41 fire died on a **new signature** — `Cron admission missing: the owner never admitted this fire` (distinct from the 10-20 orphan) — and all six fires, failures included, carry `delivery_outcome: delivered`, so up to **six digests reached Anthony**. Neither failure raised a `cron_incidents` row; the job now reads `last_status: delivery_queued` with an **empty** `last_error`. Hypothesis (unconfirmed): the 08:31 fire **adopted an unrelated long-lived session** — its worker made **70 `terminal` calls on VM101 `/dev/sdb` backup work** with context growing 66 k → 148 k tokens, while **context compression was unavailable all day** (`invalid_grant`). → [[systems/scheduled-tasks]], [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] diff --git a/log.md b/log.md index 909cb82..16f26fb 100644 --- a/log.md +++ b/log.md @@ -1506,3 +1506,12 @@ not pushed. - **Unresolved:** whether the two-stage split actually fixes the budget problem — the first real 07:00 → 11:30 cycle (stage 1 handoff → stage 2 writes) had not run at audit time; why a budget exhaustion is classified `failure_type: auth`; and the 10-27 radar incident row will only close when a clean stage-1 slot runs. + +## [2026-10-29] audit | Nightly wiki audit — window 2026-10-28 00:00 → 2026-10-29 00:3x AWST: the radar's two-stage split ran for real and landed nothing, and the stocktake died on a 400 that never alerted + +- **New page:** [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] — stage 1 (`dca8482e4f76`) ran 07:00:32 → 10:33:18 (3 h 32 m 46 s), closed `failed`, `failure_streak: 2`, on *"hit the agent iteration cap before the handoff could be written"*. No handoff file existed (`~/.hermes/radar-ref/handoff/` held only `2026-10-27.json`), so **both stage-2 slots (`5fc292ff0f3b`, 11:30:27 → 11:31:31 and 14:30:27 → 14:31:43) `completed` with `delivery_outcome: suppressed`** — the no-op recovery rule proven in production. Second consecutive stage-1 failure at a *different* ceiling (tool-call budget on 10-27 vs iteration cap on 10-28). +- **Updated:** [[incidents/2026-10-27-job-radar-budget-exhausted]] (follow-up section answering its open questions: handoff never written, stage 2 no-oped correctly, its incident row did **not** close, unclosed rows 2 → 4) · [[runbooks/job-radar]] (10-28 slot write-up; "first real cycle" marked as run-and-empty) · [[systems/scheduled-tasks]] (new 2026-10-29 window bullet; prior bullet demoted to *Prior window*; the two in-window job rows refreshed) · [[systems/browser-backend]] (dated correction: Camofox restored, `browser-steel` plugin removed, endpoint dark, `STEEL_BASE_URL` residue in 10 live processes) · [[comparisons/steel-vs-camoufox]] · [[decisions/rollback-steel-to-camofox]] (2026-10-29 update) · [[systems/skills-index]] (stocktake-run note) · `current-state.md` (new 2026-10-28 section) · [[index]]. +- **Stocktake folded, not paged:** `1a8e90ea2271` ran 03:00:28 → 03:37:58, closed `failed` (streak 1, delivery `suppressed`) on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` after regenerating `systems/skills-index.md` (302 skills) and `raw/stocktake/freshness-20261028.txt` (**2 stale / 103 fresh / 4 no-date of 109 pages**); it left both uncommitted and filed incident `1a8e90_f57ab4cafab8` in state **`detected`** with no alert. The audit committed the two artefacts, clearing the dirty tree. +- **Live probes (2026-10-29 00:3x AWST):** `jobs.json` 29 jobs / 26 enabled / 3 disabled, 5 non-clean `last_status`, pins 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`; `executions.db` — 2026-10-28 settled **168 / 2 / 1 running**, 2026-10-27 now **173 / 1**, 2026-10-29 so far 6 / 1 running; Steel: ping 2.5 ms but CDP `:9223` and REST `:3000` refuse, tunnel `/steel` **502**, `steel.martinwa.org` unresolved; no Hindsight service on this box (`:8888` closed, no `hindsight` units) so [[systems/memory-backend]] and [[comparisons/memory-backends]] were left unchanged. +- **Removed/archived:** none. **Secrets written:** none. +- **Unresolved:** the radar's discovery stage still cannot reach its own handoff write (two ceilings in two days); `1a8e90_f57ab4cafab8` sits `detected` without an alert; both radar incident rows close only on a clean 07:00 weekday slot. diff --git a/raw/stocktake/freshness-20261028.txt b/raw/stocktake/freshness-20261028.txt new file mode 100644 index 0000000..9cd5938 --- /dev/null +++ b/raw/stocktake/freshness-20261028.txt @@ -0,0 +1,122 @@ +# Wiki freshness audit — 2026-10-28 +Threshold: >21 days since `updated` + +## STALE (2) + +- [ 22d] incidents/2026-10-01-cliproxy-connection-auth-expired.md (updated 2026-10-06) +- [ 22d] incidents/2026-10-05-dead-model-pin-repin.md (updated 2026-10-06) + +## FRESH (103) + +- [ 8d] CONTRIBUTING.md +- [ 2d] agents/hermes-chaos.md +- [ 2d] agents/hermes-production.md +- [ 8d] agents/index.md +- [ 17d] agents/nanobot.md +- [ 8d] agents/open-claw.md +- [ 17d] agents/qwenpaw.md +- [ 8d] agents/zeroclaw.md +- [ 8d] comparisons/camoufox-vs-chrome.md +- [ 8d] comparisons/hermes-vs-nanobot.md +- [ 8d] comparisons/memory-backends.md +- [ 8d] comparisons/model-routers.md +- [ 15d] comparisons/steel-vs-camoufox.md +- [ 8d] concepts/agent-memory.md +- [ 15d] concepts/browser-automation.md +- [ 8d] concepts/model-routing.md +- [ 8d] concepts/persistent-shells.md +- [ 8d] concepts/tool-calling.md +- [ 0d] current-state.md +- [ 8d] decisions/index.md +- [ 8d] decisions/keep-openrouter-as-fallback.md +- [ 8d] decisions/persistent-shell-enabled.md +- [ 8d] decisions/rollback-steel-to-camofox.md +- [ 8d] experiments/index.md +- [ 4d] incidents/2026-07-29-mnemosyne-relay.md +- [ 8d] incidents/2026-08-01-mnemosyne-capture-failure.md +- [ 4d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- [ 6d] incidents/2026-08-08-gitea-config-backup-push-dead.md +- [ 5d] incidents/2026-08-15-skills-index-drift.md +- [ 16d] incidents/2026-08-18-mnemosyne-capture-pipeline.md +- [ 3d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 8d] incidents/2026-08-29-gmail-read-state.md +- [ 7d] incidents/2026-09-02-sftpgo-outage.md +- [ 17d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 2d] incidents/2026-10-03-ct450-hermes-update.md +- [ 2d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 4d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 19d] incidents/2026-10-09-memory-recall-relevance.md +- [ 12d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 1d] incidents/2026-10-16-omniroute-503-resource-pressure.md +- [ 9d] incidents/2026-10-18-degenerate-stream-canned-substitute.md +- [ 1d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md +- [ 6d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md +- [ 3d] incidents/2026-10-22-conduit-memory-start-timeout.md +- [ 2d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md +- [ 2d] incidents/2026-10-24-conduit-call-request-refused.md +- [ 2d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md +- [ 0d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md +- [ 0d] incidents/2026-10-27-job-radar-budget-exhausted.md +- [ 6d] infrastructure/backups.md +- [ 8d] infrastructure/credentials-map.md +- [ 17d] infrastructure/docker-services.md +- [ 8d] infrastructure/domains-and-tunnels.md +- [ 2d] infrastructure/hosts.md +- [ 14d] infrastructure/network-map.md +- [ 14d] infrastructure/proxmox.md +- [ 0d] infrastructure/tailscale.md +- [ 16d] queries/retained-research-answers.md +- [ 3d] reports/communities-wa-social-media-audit-2026.md +- [ 7d] reports/uber-eats/2026-07.md +- [ 7d] reports/uber-eats/2026-08.md +- [ 7d] reports/uber-eats/2026-09.md +- [ 7d] reports/uber-eats/index.md +- [ 8d] runbooks/backup-wiki.md +- [ 8d] runbooks/diagnose-docker.md +- [ 8d] runbooks/gateway-resource-resilience.md +- [ 0d] runbooks/job-radar.md +- [ 8d] runbooks/multiplexer-setup.md +- [ 2d] runbooks/provider-health.md +- [ 8d] runbooks/recover-docker-service.md +- [ 8d] runbooks/restart-browser.md +- [ 8d] runbooks/restart-hermes.md +- [ 8d] runbooks/restore-openrouter.md +- [ 8d] runbooks/test-openrouter.md +- [ 8d] runbooks/troubleshoot-tool-loop.md +- [ 2d] runbooks/update-hermes-safely.md +- [ 15d] systems/browser-backend.md +- [ 8d] systems/cloudflare-tunnel.md +- [ 0d] systems/conduit.md +- [ 2d] systems/current-profile.md +- [ 8d] systems/guanaco.md +- [ 8d] systems/headless-server.md +- [ 0d] systems/hermes-agent.md +- [ 8d] systems/hermes-ui.md +- [ 16d] systems/home-assistant.md +- [ 3d] systems/homepage-dashboard.md +- [ 17d] systems/ignis.md +- [ 2d] systems/memory-backend.md +- [ 2d] systems/messaging-integrations.md +- [ 4d] systems/mnemosyne.md +- [ 2d] systems/model-providers.md +- [ 16d] systems/nanobot.md +- [ 8d] systems/nextcloud.md +- [ 8d] systems/obsidian.md +- [ 6d] systems/omniroute.md +- [ 8d] systems/opencode-go.md +- [ 8d] systems/openrouter.md +- [ 17d] systems/paseo.md +- [ 9d] systems/push-notifications.md +- [ 0d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md +- [ 8d] systems/terminal-backend.md +- [ 8d] systems/tool-search.md + +## NO updated DATE (4) + +- SCHEMA.md +- experiments/active/example.md +- index.md +- log.md + +TOTAL: 2 stale, 103 fresh, 4 no-date of 109 pages \ No newline at end of file diff --git a/runbooks/job-radar.md b/runbooks/job-radar.md index b54ecf1..1564163 100644 --- a/runbooks/job-radar.md +++ b/runbooks/job-radar.md @@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery type: runbook status: active created: 2026-07-27 -updated: 2026-10-28 +updated: 2026-10-29 review_after: 2026-11-10 -verified_on: 2026-10-28 +verified_on: 2026-10-29 version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2) confidence: high tags: [automation, cron, jobs, notion, dashboard, runbook] @@ -326,13 +326,31 @@ now owns a job of its own and the writes own a second one. - Stage 1 keeps the `custom:omni-101` / `smart-route` pin. Both jobs still inherit the profile-level `fallback_providers` chain; Hermes stores no per-job fallback list. - First fires: two test runs of stage 2 on 2026-10-27 (17:35:00 → 17:36:52 and 17:38:38 → 17:43:42), both - `completed` with delivery `suppressed`. **The first real 07:00 → 11:30 cycle had not yet run at the - 2026-10-28 audit.** + `completed` with delivery `suppressed`. **The first real 07:00 → 11:30 cycle + ran on 2026-10-28 and landed nothing** — stage 1 never wrote its handoff (see the 2026-10-28 slot below). **Recovery if stage 2 lands nothing:** check that stage 1 actually wrote its handoff file before running stage 2 by hand (`hermes cron run 5fc292ff0f3b`). A stage-2 run with no handoff is a no-op, not a failed search — do not re-run stage 1's discovery just to satisfy it. +### 2026-10-28 07:00 slot — stage 1 hit the iteration cap, so stage 2 had nothing to land + +Stage 1 (`dca8482e4f76`) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**) and closed **`failed`** on +*"hit the agent iteration cap before the handoff could be written"* — its **second** consecutive failure +(`failure_streak: 2`). No handoff file was written: `~/.hermes/radar-ref/handoff/` held only `2026-10-27.json` +at the 2026-10-29 audit. + +Both stage-2 slots then behaved exactly as designed — **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both +`completed`, both `delivery_outcome: suppressed`. **A stage-2 run with no handoff is a silent no-op, not a +failure**, which is the recovery rule below holding in production for the first time. + +Read with 10-27, the picture is: *the split stops the writes from being lost, but it does not make discovery +finish.* Two runs, two different ceilings — 10-27 stopped at the **tool-call budget** after 4 h 2 m (writes not +started), 10-28 stopped at the **agent iteration cap** after 3 h 32 m, earlier still (no handoff at all). The +discovery stage is the thing to bound, not the wall clock. + +Full ledger: [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. + ## Browser-layer hazard inside the run (2026-10-12) This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove diff --git a/systems/browser-backend.md b/systems/browser-backend.md index 205fcd5..d21aeff 100644 --- a/systems/browser-backend.md +++ b/systems/browser-backend.md @@ -3,8 +3,8 @@ title: Browser Backend type: system status: active created: 2026-07-22 -updated: 2026-10-13 -verified_on: 2026-10-13 +updated: 2026-10-29 +verified_on: 2026-10-29 confidence: high tags: [browser, camofox, automation] sources: [live-probe, config, session-history] @@ -20,7 +20,7 @@ Browser automation for Hermes — web search, form filling, screenshot, CDP cont - **Provider config:** `browser.cloud_provider: camofox`, `camera_provider: camofox` - **REST API:** **`:9377`** — live-verified 2026-10-13 (`node server.js`, `CAMOFOX_PORT=9377`; `LISTEN *:9377`). **The `:9093` figure in earlier revisions of this page is stale** — nothing was listening on 9093 on 2026-10-13. - **VNC viewer:** NovNC `websockify` on `127.0.0.1:6080` → `127.0.0.1:5900`, published as `https://ct460.kangaroo-eel.ts.net/vnc-camofox` -- **Steel Browser (CT201):** Still running but Hermes not connected — plugin disabled +- **Steel Browser (CT201):** **Not reachable as of 2026-10-29 00:3x** — host answers ping (2.5 ms) but CDP `:9223` and REST `:3000` refuse; plugin removed. See the 2026-10-29 correction below. ## Camofox (local `systemd --user` service) @@ -109,6 +109,29 @@ until the next session starts. the Hermes host (CT460). Containment drop-in documented. Same day, the 07:01 Job Radar session was audited end-to-end and its memory-pressure evidence added above. +## Correction (2026-10-29) — Steel is out of the stack, not just disabled + +Verified against live state at **2026-10-29 00:3x AWST**: + +- **Hermes is back on Camofox.** `config.yaml` (mtime **2026-10-29 00:15:13**) reads + `browser.cloud_provider: camofox`, `browser.camera_provider: camofox`, `engine: auto`, `cdp_url: ""`. +- **The `browser-steel` plugin is gone** — `~/.hermes/plugins/browser-steel/` no longer exists and + `plugins.enabled` carries no steel entry, so the `plugin_steel` toolset cannot be selected. Backup taken + immediately before the change: `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` (written + **2026-10-29 00:13:24**). A second backup, `config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z`, records + the 2026-10-28 22:00 attempt. +- **The Steel endpoint is not serving.** `ping 100.96.244.39` (runtipi/CT201) answers **2.5 ms**, but + `http://100.96.244.39:9223/json/version` (CDP) and `http://100.96.244.39:3000/v1/health` (REST) both return + **000** — as does the LAN address `192.168.178.77:9223`. The tunnel route + `https://runtipi.kangaroo-eel.ts.net/steel` returns **502**, and `steel.martinwa.org` no longer resolves. + Earlier pages describing Steel as "still running on CT201" are superseded by this probe. +- ⚠ **Stale environment variable still exported:** `STEEL_BASE_URL=http://100.96.244.39:3000` is present in + **10 live Hermes processes** — the gateway tree (`3371780`, `3372125`, `3372912`, `3844013`, `3880214`, + `3880663`), `agent-browser` 0.26.0 (`3843841`), two chromium crashpad handlers (`3843878`, `3843880`) and + `agentmail-webhook.py` (`402`). It now points at a dead endpoint and clears only when the gateway is + restarted. The config is explicit (`cloud_provider: camofox`), so it does not currently misroute browser + calls — but it is a trap for anything that reads the variable. + ## Related - [[comparisons/steel-vs-camoufox]] - [[concepts/browser-automation]] diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 29dc8bd..72790ee 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,8 +3,8 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-28 -verified_on: 2026-10-28 +updated: 2026-10-29 +verified_on: 2026-10-29 review_after: 2026-11-03 confidence: high tags: [cron, scheduled, automation] @@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count -- **Current as of 2026-10-28 00:2x AWST (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. +- **Current as of 2026-10-29 00:3x AWST (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real and the discovery stage still could not finish: stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **removed** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and the endpoint is **not serving** — CDP `:9223`/REST `:3000` refuse, tunnel `/steel` **502**, `steel.martinwa.org` unresolved — while **`STEEL_BASE_URL=http://100.96.244.39:3000` is still exported into 10 live Hermes processes** and clears only on a gateway restart → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.** +- **Prior window — as of 2026-10-28 00:2x AWST** (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. *(superseded by the 2026-10-29 entry above)* - **Prior window — as of 2026-10-27 00:1x AWST *(superseded by the 2026-10-28 entry above)* (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`). - **Current as of 2026-10-26 00:1x AWST** *(superseded by the 2026-10-27 entry above)* **(live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`). - **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page. *(Superseded by the 2026-10-26 entry above.)* diff --git a/systems/skills-index.md b/systems/skills-index.md index ae18c8a..a7c3060 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-10-27 -verified_on: 2026-10-27 +updated: 2026-10-29 +verified_on: 2026-10-29 confidence: high tags: [skills, hermes] sources: [] @@ -531,3 +531,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit. ## Yuanbao (1) - `yuanbao` — # Yuanbao Group Interaction + +## Recent stocktake runs + +- **2026-10-28 03:00** — regenerated this index (**302 skills**, mtime 03:01:40) and wrote `raw/stocktake/freshness-20261028.txt` (**2 stale / 103 fresh / 4 no-date of 109 pages**), then **failed** before committing: `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` (provider `custom:omni-101 / free-stack`). Incident `1a8e90_f57ab4cafab8` (`detected`, no alert). This index and the freshness file were left uncommitted and were committed by the 2026-10-29 wiki audit — see [[systems/scheduled-tasks]] and [[incidents/2026-10-28-job-radar-stage1-iteration-cap]].