wiki(audit): 2026-10-28 window — radar stage-1 iteration cap (no handoff, stage-2 no-oped), stocktake 400 never alerted, Steel removed from the stack
This commit is contained in:
+56
-2
@@ -3,8 +3,8 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-28
|
||||
verified_on: 2026-10-28
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
review_after: 2026-11-03
|
||||
confidence: high
|
||||
tags: [system]
|
||||
@@ -45,6 +45,60 @@ Updated: 2026-10-28
|
||||
`systems/skills-index.md` (regenerated 10-27, committed), `index.md`, `log.md`.
|
||||
**Nothing removed or archived.** No secrets written.
|
||||
|
||||
## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 22:00 – 2026-10-29 00:3x AWST)
|
||||
|
||||
### The radar's two-stage split ran for real, and landed nothing — stage 1 hit the agent iteration cap
|
||||
|
||||
`dca8482e4f76` (stage 1, Discovery) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**) and closed **`failed`** on
|
||||
*"hit the agent iteration cap before the handoff could be written"* — **`failure_streak: 2`**. Stage 1 never
|
||||
produced a handoff (`~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`), so the *first* real two-stage
|
||||
cycle delivered **no Notion writes and no cover letters** — the same outcome as 10-27, by a different route.
|
||||
|
||||
Both stage-2 slots then did exactly what they were built to do: `5fc292ff0f3b` fired **11:30:27 → 11:31:31** and
|
||||
**14:30:27 → 14:31:43**, both `completed` with **`delivery_outcome: suppressed`** — *a stage-2 run with no
|
||||
handoff is a silent no-op, not a failure* (the recovery rule in [[runbooks/job-radar]]), now proven in
|
||||
production. The two post-split failures also stopped at different ceilings: 10-27 at the **tool-call budget**
|
||||
(4 h 2 m, writes not started), 10-28 at the **iteration cap** (3 h 32 m, handoff not written) — less of the
|
||||
pipeline completed despite the shorter wall clock. Open radar incident rows: `dca848_e93a538ade6b` (10-27) and
|
||||
`dca848_0a2f03b5952b` (10-28), both open. Detail:
|
||||
[[incidents/2026-10-28-job-radar-stage1-iteration-cap]].
|
||||
|
||||
### The 03:00 stocktake failed on a provider 400 — and its incident row never alerted
|
||||
|
||||
`1a8e90ea2271` ran **03:00:28 → 03:37:58** (37 m 30 s) and closed **`failed`** (streak 1, delivery
|
||||
`suppressed`) with `RuntimeError: [400] Failed to deserialize the JSON body into the target type: duplicate
|
||||
field 'reasoning_content' at line 1 column 512203` against `custom:omni-101 / free-stack` — the *response* the
|
||||
provider was asked to deserialize carried a duplicated `reasoning_content` field. It performed its rebuild
|
||||
(302 skills → `systems/skills-index.md` mtime 03:01:40; freshness report `raw/stocktake/freshness-20261028.txt`
|
||||
→ **2 stale / 103 fresh / 4 no-date of 109 pages**) and then died **before the commit**, leaving the wiki tree
|
||||
dirty for the rest of the day. Its incident row `1a8e90_f57ab4cafab8` sits in state **`detected`** with **no
|
||||
alert** — the only outward sign was the job's own `error` status. The 2026-10-29 audit committed the two
|
||||
artefacts.
|
||||
|
||||
### Steel left the stack: Camofox restored, plugin removed, endpoint dark
|
||||
|
||||
`browser.cloud_provider` and `browser.camera_provider` are back to **`camofox`** (`engine: auto`, `cdp_url: ""`)
|
||||
in `config.yaml` (mtime **2026-10-29 00:15:13**), with backups taken at every step
|
||||
(`config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z` 22:00:57, and
|
||||
`config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24). The **`browser-steel` plugin was
|
||||
removed** from `~/.hermes/plugins/` and from `plugins.enabled`, so the `plugin_steel` toolset is no longer
|
||||
selectable. Steel itself is **not serving**: the CT201/runtipi host answers ping (2.5 ms) but CDP `:9223` and
|
||||
REST `:3000` refuse (also on the LAN address), the tunnel route `/steel` returns **502**, and
|
||||
`steel.martinwa.org` no longer resolves — superseding the earlier "still running on CT201" notes. ⚠
|
||||
`STEEL_BASE_URL=http://100.96.244.39:3000` remains exported into **10 live Hermes processes** and clears only on
|
||||
a gateway restart. Detail: [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]],
|
||||
[[comparisons/steel-vs-camoufox]].
|
||||
|
||||
### Fleet and tallies
|
||||
|
||||
29 jobs (**26 enabled / 3 disabled**) — unchanged. Non-clean `last_status`: **5** (`0e753e9720fd` error,
|
||||
`fad5a86979d4` delivery_failed, `dca8482e4f76` error **streak 2**, `066bb5191280` delivery_failed,
|
||||
`1a8e90ea2271` error **streak 1**). Pins: **12 `smart-route` / 6 `free-stack` / 7 unpinned /
|
||||
1 `deepseek-paid-smart`**. **2026-10-28 settled 168 completed / 2 failed / 1 running**; **2026-10-29** so far
|
||||
6 completed / 1 running; **2026-10-27 settled 173 / 1** (revised up by one late run after the 2026-10-28
|
||||
audit). No Hindsight service is running on this box (`:8888` closed, no `hindsight` units) — the standing
|
||||
memory-backend correction still holds and needs no change.
|
||||
|
||||
## Recent Changes (2026-10-27, nightly audit; live probes 2026-10-27 00:1x–00:5x AWST)
|
||||
|
||||
- **A *planned* gateway restart still orphaned a live cron run — and this time it alerted loudly.** A planned
|
||||
|
||||
Reference in New Issue
Block a user