wiki(audit): 2026-10-28 window — radar stage-1 iteration cap (no handoff, stage-2 no-oped), stocktake 400 never alerted, Steel removed from the stack
This commit is contained in:
@@ -3,8 +3,8 @@ title: Steel Browser vs Camoufox
|
||||
type: comparison
|
||||
status: active
|
||||
created: 2026-07-29
|
||||
updated: 2026-10-13
|
||||
verified_on: 2026-10-13
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
confidence: high
|
||||
tags: [comparison, browser, steel, camofox, decision]
|
||||
sources: [live-probe, session-history]
|
||||
@@ -26,7 +26,7 @@ Compare the two browser-automation backends available to Hermes, including the J
|
||||
| **Container** | Docker on CT201 | `systemd --user` unit on the Hermes host (CT460) |
|
||||
| **Stealth** | Standard Chromium | Anti-detection fingerprinting |
|
||||
| **Session viewer** | Built-in UI at `/steel` | VNC at `/vnc-camofox` |
|
||||
| **Plugin** | `browser-steel` plugin (disabled) | Native Hermes `browser.cloud_provider: camofox` |
|
||||
| **Plugin** | `browser-steel` — **removed 2026-10-29** | Native Hermes `browser.cloud_provider: camofox` |
|
||||
|
||||
## The Steel Experiment (Jul 29–30)
|
||||
|
||||
@@ -48,7 +48,7 @@ Compare the two browser-automation backends available to Hermes, including the J
|
||||
|
||||
### Lesson learned
|
||||
Steel is a capable backend but adds a network hop and another plugin layer. Camofox runs locally on the
|
||||
Hermes host (CT460) with fewer failure modes. The `browser-steel` plugin still exists (disabled) if needed in future.
|
||||
Hermes host (CT460) with fewer failure modes. The `browser-steel` plugin was **removed on 2026-10-29** after the endpoint went dark, so returning to Steel means redeploying the plugin, not flipping a flag.
|
||||
|
||||
## Detail
|
||||
|
||||
@@ -62,11 +62,11 @@ Hermes host (CT460) with fewer failure modes. The `browser-steel` plugin still e
|
||||
|
||||
### Steel Browser (disabled but available)
|
||||
- **Deployed:** 2026-07-29 on CT201, Docker, custom `steel-browser-api:fixed` image
|
||||
- **Status:** Still running on CT201, Hermes not connected
|
||||
- **Status:** **Offline** (verified 2026-10-29 00:3x) — the host answers ping in 2.5 ms, but CDP `:9223` and REST `:3000` refuse, the tunnel route `/steel` returns **502** and `steel.martinwa.org` no longer resolves
|
||||
- **CDP endpoint:** `http://100.96.244.39:9223`
|
||||
- **REST API:** `http://100.96.244.39:3000`
|
||||
- **UI:** `https://runtipi.kangaroo-eel.ts.net/steel`
|
||||
- **Plugin:** `browser-steel` at `/home/hermes/.hermes/plugins/browser-steel/` — disabled, `plugin_steel` toolset flagged `requires_health=steel`
|
||||
- **Plugin:** **removed 2026-10-29 00:13** — `~/.hermes/plugins/browser-steel/` is gone and `plugins.enabled` has no steel entry, so the `plugin_steel` toolset can no longer be selected
|
||||
|
||||
## Related
|
||||
- [[systems/browser-backend]]
|
||||
|
||||
+56
-2
@@ -3,8 +3,8 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-28
|
||||
verified_on: 2026-10-28
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
review_after: 2026-11-03
|
||||
confidence: high
|
||||
tags: [system]
|
||||
@@ -45,6 +45,60 @@ Updated: 2026-10-28
|
||||
`systems/skills-index.md` (regenerated 10-27, committed), `index.md`, `log.md`.
|
||||
**Nothing removed or archived.** No secrets written.
|
||||
|
||||
## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 22:00 – 2026-10-29 00:3x AWST)
|
||||
|
||||
### The radar's two-stage split ran for real, and landed nothing — stage 1 hit the agent iteration cap
|
||||
|
||||
`dca8482e4f76` (stage 1, Discovery) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**) and closed **`failed`** on
|
||||
*"hit the agent iteration cap before the handoff could be written"* — **`failure_streak: 2`**. Stage 1 never
|
||||
produced a handoff (`~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`), so the *first* real two-stage
|
||||
cycle delivered **no Notion writes and no cover letters** — the same outcome as 10-27, by a different route.
|
||||
|
||||
Both stage-2 slots then did exactly what they were built to do: `5fc292ff0f3b` fired **11:30:27 → 11:31:31** and
|
||||
**14:30:27 → 14:31:43**, both `completed` with **`delivery_outcome: suppressed`** — *a stage-2 run with no
|
||||
handoff is a silent no-op, not a failure* (the recovery rule in [[runbooks/job-radar]]), now proven in
|
||||
production. The two post-split failures also stopped at different ceilings: 10-27 at the **tool-call budget**
|
||||
(4 h 2 m, writes not started), 10-28 at the **iteration cap** (3 h 32 m, handoff not written) — less of the
|
||||
pipeline completed despite the shorter wall clock. Open radar incident rows: `dca848_e93a538ade6b` (10-27) and
|
||||
`dca848_0a2f03b5952b` (10-28), both open. Detail:
|
||||
[[incidents/2026-10-28-job-radar-stage1-iteration-cap]].
|
||||
|
||||
### The 03:00 stocktake failed on a provider 400 — and its incident row never alerted
|
||||
|
||||
`1a8e90ea2271` ran **03:00:28 → 03:37:58** (37 m 30 s) and closed **`failed`** (streak 1, delivery
|
||||
`suppressed`) with `RuntimeError: [400] Failed to deserialize the JSON body into the target type: duplicate
|
||||
field 'reasoning_content' at line 1 column 512203` against `custom:omni-101 / free-stack` — the *response* the
|
||||
provider was asked to deserialize carried a duplicated `reasoning_content` field. It performed its rebuild
|
||||
(302 skills → `systems/skills-index.md` mtime 03:01:40; freshness report `raw/stocktake/freshness-20261028.txt`
|
||||
→ **2 stale / 103 fresh / 4 no-date of 109 pages**) and then died **before the commit**, leaving the wiki tree
|
||||
dirty for the rest of the day. Its incident row `1a8e90_f57ab4cafab8` sits in state **`detected`** with **no
|
||||
alert** — the only outward sign was the job's own `error` status. The 2026-10-29 audit committed the two
|
||||
artefacts.
|
||||
|
||||
### Steel left the stack: Camofox restored, plugin removed, endpoint dark
|
||||
|
||||
`browser.cloud_provider` and `browser.camera_provider` are back to **`camofox`** (`engine: auto`, `cdp_url: ""`)
|
||||
in `config.yaml` (mtime **2026-10-29 00:15:13**), with backups taken at every step
|
||||
(`config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z` 22:00:57, and
|
||||
`config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24). The **`browser-steel` plugin was
|
||||
removed** from `~/.hermes/plugins/` and from `plugins.enabled`, so the `plugin_steel` toolset is no longer
|
||||
selectable. Steel itself is **not serving**: the CT201/runtipi host answers ping (2.5 ms) but CDP `:9223` and
|
||||
REST `:3000` refuse (also on the LAN address), the tunnel route `/steel` returns **502**, and
|
||||
`steel.martinwa.org` no longer resolves — superseding the earlier "still running on CT201" notes. ⚠
|
||||
`STEEL_BASE_URL=http://100.96.244.39:3000` remains exported into **10 live Hermes processes** and clears only on
|
||||
a gateway restart. Detail: [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]],
|
||||
[[comparisons/steel-vs-camoufox]].
|
||||
|
||||
### Fleet and tallies
|
||||
|
||||
29 jobs (**26 enabled / 3 disabled**) — unchanged. Non-clean `last_status`: **5** (`0e753e9720fd` error,
|
||||
`fad5a86979d4` delivery_failed, `dca8482e4f76` error **streak 2**, `066bb5191280` delivery_failed,
|
||||
`1a8e90ea2271` error **streak 1**). Pins: **12 `smart-route` / 6 `free-stack` / 7 unpinned /
|
||||
1 `deepseek-paid-smart`**. **2026-10-28 settled 168 completed / 2 failed / 1 running**; **2026-10-29** so far
|
||||
6 completed / 1 running; **2026-10-27 settled 173 / 1** (revised up by one late run after the 2026-10-28
|
||||
audit). No Hindsight service is running on this box (`:8888` closed, no `hindsight` units) — the standing
|
||||
memory-backend correction still holds and needs no change.
|
||||
|
||||
## Recent Changes (2026-10-27, nightly audit; live probes 2026-10-27 00:1x–00:5x AWST)
|
||||
|
||||
- **A *planned* gateway restart still orphaned a live cron run — and this time it alerted loudly.** A planned
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Rollback Steel to Camoufox
|
||||
type: decision
|
||||
status: active
|
||||
created: 2026-08-02
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
confidence: high
|
||||
tags: [decision, browser, steel, camofox, rollback]
|
||||
sources: [session-history, live-config]
|
||||
@@ -59,6 +59,21 @@ Steel Browser was deployed on CT201 on 2026-07-29 as the primary browser automat
|
||||
- Uber Eats report (Aug 1) successfully scraped via Camofox REST API — 11 orders, $264.51
|
||||
- Steel plugin `browser-steel` still present at `/home/hermes/.hermes/plugins/browser-steel/` — disabled
|
||||
|
||||
## Update (2026-10-29) — the plugin was removed and the endpoint went dark
|
||||
|
||||
The 2026-07-30 rollback disabled the Steel plugin but left it on disk and left Steel running. Both of those
|
||||
were closed on **2026-10-28 22:00 – 2026-10-29 00:15**: `browser.cloud_provider` / `camera_provider` were
|
||||
returned to **`camofox`** (backups `config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z`,
|
||||
`config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z`), and **`~/.hermes/plugins/browser-steel/` was
|
||||
removed**, so the `plugin_steel` toolset is no longer selectable.
|
||||
|
||||
Steel itself is **not serving**: CDP `:9223` and REST `:3000` on `100.96.244.39` refuse, the tunnel route
|
||||
`/steel` returns **502**, and `steel.martinwa.org` does not resolve, while the host still answers ping.
|
||||
`STEEL_BASE_URL=http://100.96.244.39:3000` remains exported in 10 live Hermes processes and clears on the next
|
||||
gateway restart. The decision below is unchanged — Camofox is the backend — but there is now **no Steel plugin
|
||||
fallback in the tree**, so returning to Steel would mean redeploying it. Detail:
|
||||
[[systems/browser-backend]], [[comparisons/steel-vs-camoufox]].
|
||||
|
||||
## Related
|
||||
- [[comparisons/steel-vs-camoufox]]
|
||||
- [[systems/browser-backend]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-10-28
|
||||
updated: 2026-10-28
|
||||
verified_on: 2026-10-28
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
review_after: 2026-11-04
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
||||
@@ -85,6 +85,22 @@ Fleet size went **28 → 29 jobs** (26 enabled / 3 disabled).
|
||||
4. The incident row stays open until a clean stage-1 slot auto-closes it — confirm the 2026-10-28 07:00 slot
|
||||
closes `dca848_e93a538ade6b`.
|
||||
|
||||
## Follow-up — the 2026-10-28 slot (recorded 2026-10-29)
|
||||
|
||||
The **first real two-stage cycle ran on 2026-10-28** and answered the open questions above — not favourably:
|
||||
|
||||
- **Stage 1 overran again, on a new signature.** `dca8482e4f76` ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**),
|
||||
closed **`failed`**, `failure_streak` **2**, on *"hit the agent iteration cap before the handoff could be
|
||||
written"* → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]].
|
||||
- **No handoff file was written** — `~/.hermes/radar-ref/handoff/` held only `2026-10-27.json` at the 2026-10-29
|
||||
audit, so **Notion writes and cover letters did not run again**.
|
||||
- **Both stage-2 slots no-oped correctly** — `5fc292ff0f3b` fired **11:30:27 → 11:31:31** and
|
||||
**14:30:27 → 14:31:43**, both `completed` with `delivery_outcome: suppressed`. Open question 1 is therefore
|
||||
answered only in the negative: the recovery rule ("a stage-2 run with no handoff is a no-op") holds in
|
||||
production, but nothing has yet proved the positive path end-to-end.
|
||||
- **Open question 4 is answered: the row did not close.** `dca848_e93a538ade6b` stayed open because the 10-28
|
||||
slot was not clean, and a second row (`dca848_0a2f03b5952b`) opened — open incident rows **2 → 4**.
|
||||
|
||||
## Related
|
||||
|
||||
- [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
---
|
||||
title: Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-10-29
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
review_after: 2026-11-05
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-radar-handoff-dir]
|
||||
---
|
||||
|
||||
# Job Radar Stage 1 Hit the Agent Iteration Cap on 2026-10-28 — No Handoff Was Written, So Both Stage-2 Slots Correctly No-Oped
|
||||
|
||||
## Headline
|
||||
|
||||
On **2026-10-28** the Discovery stage of the Daily Marketing Job Radar (`dca8482e4f76`) ran
|
||||
**07:00:32 → 10:33:18 AWST** (**3 h 32 m 46 s**) and closed **`failed`** on *"hit the agent iteration cap before
|
||||
the handoff could be written"* — its **second consecutive failure** (`failure_streak: 2`).
|
||||
|
||||
This is the first real cycle of the **two-stage split** introduced on 2026-10-27 (see
|
||||
[[incidents/2026-10-27-job-radar-budget-exhausted]] and [[runbooks/job-radar]]), and it produced a clean negative
|
||||
result: **stage 1 never wrote its handoff file, so there was nothing for stage 2 to land.** Both stage-2 slots
|
||||
ran anyway — **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed`, both
|
||||
`delivery_outcome: suppressed` — which is the designed no-op path working correctly in production for the first
|
||||
time. Notion ingestion and cover letters did not execute for the day.
|
||||
|
||||
## Ledger state
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Execution row | `dca8482e4f76`, started `2026-10-28T07:00:32.463309+08:00`, finished `2026-10-28T10:33:18.833788+08:00`, **`failed`** |
|
||||
| Duration | **3 h 32 m 46 s** (10-27: 4 h 2 m 46 s; 10-26: 2 h 9 m 19 s) |
|
||||
| `jobs.json` | `last_status: error`, **`failure_streak: 2`**; `last_error` opens *"Daily Marketing Job Radar — Stage 1 (Discovery) — 2026-10-28 — hit the agent iteration cap before the handoff could be…"*; pin unchanged (`custom:omni-101` / `smart-route`) |
|
||||
| Incident row | `dca848_0a2f03b5952b` — `state: alerted`, `failure_type: timeout`, first_seen `2026-10-28T10:33:18`, **open** |
|
||||
| Handoff | **not written** — `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json` (verified 2026-10-29 00:3x) |
|
||||
| Stage 2 | `5fc292ff0f3b` fired `11:30:27 → 11:31:31` and `14:30:27 → 14:31:43`, both `completed` / `delivery_outcome: suppressed` |
|
||||
|
||||
The 10-27 row (`dca848_e93a538ade6b`, classified `failure_type: auth`) did **not** close — the 2026-10-28 07:00
|
||||
slot was not clean. Open `cron_incidents` rows went **2 → 4**.
|
||||
|
||||
## What the split proved, and what it did not
|
||||
|
||||
1. **It proved the recovery rule.** *"A stage-2 run with no handoff is a no-op, not a failed search"* — exactly as
|
||||
written in [[runbooks/job-radar]] — held in production: stage 2 invented no work, raised no error, and sent
|
||||
nothing to topic 1030.
|
||||
2. **It did not make discovery finish.** The two post-split runs died at *different* ceilings:
|
||||
- **2026-10-27** — stopped at the **tool-call budget** after 4 h 2 m: discovery/verify/dedup/classify complete, writes never started.
|
||||
- **2026-10-28** — stopped at the **agent iteration cap** after 3 h 32 m, *earlier in the pipeline*: no handoff, so nothing downstream could run at all.
|
||||
The wall-clock number improving (4 h 2 m → 3 h 32 m) is **not** an improvement — less of the pipeline completed.
|
||||
3. **The write stage is no longer the risk; the discovery stage still is.** Splitting removed the
|
||||
"writes lost inside a green row" failure mode. What remains is that stage 1 cannot both finish discovery and
|
||||
reach its handoff write inside its own limits.
|
||||
|
||||
## Why it matters
|
||||
|
||||
- **A lost day is now cheap to detect but easy to miss.** Stage 1 raises an incident row and flips `last_status`
|
||||
to `error`; stage 2's *correct* silence (`suppressed`) means the only outward sign of a workless day is stage
|
||||
1's own failure report — not a "0 jobs added" report.
|
||||
- **Two consecutive failures, two different classifiers.** 10-27 was tagged `failure_type: auth` (wrong for a
|
||||
budget fault); 10-28 was tagged `timeout` (also only approximate — an agent-side iteration cap, not a network
|
||||
timeout). Neither string names the actual cause.
|
||||
- **Budget arithmetic is not the fix.** If the cap fires after ~3.5 h of discovery, the pipeline needs a smaller
|
||||
discovery surface (fewer sources per slot), **staged/checkpointed handoff writes** (so a capped run still
|
||||
leaves something landable), or an explicit per-phase iteration allowance.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **What is the iteration cap for this job, and can stage 1 reach its handoff write before it?** The 10-28 run
|
||||
died *before* the write; the 10-27 run died *after* discovery but *before* writes. Neither produced a handoff.
|
||||
2. **Should stage 1 checkpoint a partial handoff** (after DEDUP/SCORE) so a capped run gives stage 2 something to
|
||||
land instead of a silent no-op day?
|
||||
3. **Make the abort self-describing** — a distinct `iteration-cap` / `partial-run` failure type rather than
|
||||
`auth` / `timeout`.
|
||||
4. Both radar incident rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) close only when a **clean** stage-1
|
||||
slot runs — watch the next 07:00 weekday slot.
|
||||
|
||||
## Related
|
||||
|
||||
- [[runbooks/job-radar]] — the two-stage contract, recovery rule and the 2026-10-28 slot write-up
|
||||
- [[incidents/2026-10-27-job-radar-budget-exhausted]] — the prior day's budget exhaustion and the split
|
||||
- [[systems/scheduled-tasks]] — fleet ledger and the 2026-10-28 window
|
||||
- [[systems/browser-backend]] — the browser layer this run depends on
|
||||
@@ -1506,3 +1506,12 @@ not pushed.
|
||||
- **Unresolved:** whether the two-stage split actually fixes the budget problem — the first real 07:00 → 11:30
|
||||
cycle (stage 1 handoff → stage 2 writes) had not run at audit time; why a budget exhaustion is classified
|
||||
`failure_type: auth`; and the 10-27 radar incident row will only close when a clean stage-1 slot runs.
|
||||
|
||||
## [2026-10-29] audit | Nightly wiki audit — window 2026-10-28 00:00 → 2026-10-29 00:3x AWST: the radar's two-stage split ran for real and landed nothing, and the stocktake died on a 400 that never alerted
|
||||
|
||||
- **New page:** [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] — stage 1 (`dca8482e4f76`) ran 07:00:32 → 10:33:18 (3 h 32 m 46 s), closed `failed`, `failure_streak: 2`, on *"hit the agent iteration cap before the handoff could be written"*. No handoff file existed (`~/.hermes/radar-ref/handoff/` held only `2026-10-27.json`), so **both stage-2 slots (`5fc292ff0f3b`, 11:30:27 → 11:31:31 and 14:30:27 → 14:31:43) `completed` with `delivery_outcome: suppressed`** — the no-op recovery rule proven in production. Second consecutive stage-1 failure at a *different* ceiling (tool-call budget on 10-27 vs iteration cap on 10-28).
|
||||
- **Updated:** [[incidents/2026-10-27-job-radar-budget-exhausted]] (follow-up section answering its open questions: handoff never written, stage 2 no-oped correctly, its incident row did **not** close, unclosed rows 2 → 4) · [[runbooks/job-radar]] (10-28 slot write-up; "first real cycle" marked as run-and-empty) · [[systems/scheduled-tasks]] (new 2026-10-29 window bullet; prior bullet demoted to *Prior window*; the two in-window job rows refreshed) · [[systems/browser-backend]] (dated correction: Camofox restored, `browser-steel` plugin removed, endpoint dark, `STEEL_BASE_URL` residue in 10 live processes) · [[comparisons/steel-vs-camoufox]] · [[decisions/rollback-steel-to-camofox]] (2026-10-29 update) · [[systems/skills-index]] (stocktake-run note) · `current-state.md` (new 2026-10-28 section) · [[index]].
|
||||
- **Stocktake folded, not paged:** `1a8e90ea2271` ran 03:00:28 → 03:37:58, closed `failed` (streak 1, delivery `suppressed`) on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` after regenerating `systems/skills-index.md` (302 skills) and `raw/stocktake/freshness-20261028.txt` (**2 stale / 103 fresh / 4 no-date of 109 pages**); it left both uncommitted and filed incident `1a8e90_f57ab4cafab8` in state **`detected`** with no alert. The audit committed the two artefacts, clearing the dirty tree.
|
||||
- **Live probes (2026-10-29 00:3x AWST):** `jobs.json` 29 jobs / 26 enabled / 3 disabled, 5 non-clean `last_status`, pins 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`; `executions.db` — 2026-10-28 settled **168 / 2 / 1 running**, 2026-10-27 now **173 / 1**, 2026-10-29 so far 6 / 1 running; Steel: ping 2.5 ms but CDP `:9223` and REST `:3000` refuse, tunnel `/steel` **502**, `steel.martinwa.org` unresolved; no Hindsight service on this box (`:8888` closed, no `hindsight` units) so [[systems/memory-backend]] and [[comparisons/memory-backends]] were left unchanged.
|
||||
- **Removed/archived:** none. **Secrets written:** none.
|
||||
- **Unresolved:** the radar's discovery stage still cannot reach its own handoff write (two ceilings in two days); `1a8e90_f57ab4cafab8` sits `detected` without an alert; both radar incident rows close only on a clean 07:00 weekday slot.
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
# Wiki freshness audit — 2026-10-28
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (2)
|
||||
|
||||
- [ 22d] incidents/2026-10-01-cliproxy-connection-auth-expired.md (updated 2026-10-06)
|
||||
- [ 22d] incidents/2026-10-05-dead-model-pin-repin.md (updated 2026-10-06)
|
||||
|
||||
## FRESH (103)
|
||||
|
||||
- [ 8d] CONTRIBUTING.md
|
||||
- [ 2d] agents/hermes-chaos.md
|
||||
- [ 2d] agents/hermes-production.md
|
||||
- [ 8d] agents/index.md
|
||||
- [ 17d] agents/nanobot.md
|
||||
- [ 8d] agents/open-claw.md
|
||||
- [ 17d] agents/qwenpaw.md
|
||||
- [ 8d] agents/zeroclaw.md
|
||||
- [ 8d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 8d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 8d] comparisons/memory-backends.md
|
||||
- [ 8d] comparisons/model-routers.md
|
||||
- [ 15d] comparisons/steel-vs-camoufox.md
|
||||
- [ 8d] concepts/agent-memory.md
|
||||
- [ 15d] concepts/browser-automation.md
|
||||
- [ 8d] concepts/model-routing.md
|
||||
- [ 8d] concepts/persistent-shells.md
|
||||
- [ 8d] concepts/tool-calling.md
|
||||
- [ 0d] current-state.md
|
||||
- [ 8d] decisions/index.md
|
||||
- [ 8d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 8d] decisions/persistent-shell-enabled.md
|
||||
- [ 8d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 8d] experiments/index.md
|
||||
- [ 4d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 8d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 4d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 6d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 5d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 16d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 3d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 8d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 7d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 17d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 2d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 2d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 4d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 19d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 12d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 1d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 9d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 1d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 6d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 3d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 2d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 2d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 2d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 0d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 0d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 6d] infrastructure/backups.md
|
||||
- [ 8d] infrastructure/credentials-map.md
|
||||
- [ 17d] infrastructure/docker-services.md
|
||||
- [ 8d] infrastructure/domains-and-tunnels.md
|
||||
- [ 2d] infrastructure/hosts.md
|
||||
- [ 14d] infrastructure/network-map.md
|
||||
- [ 14d] infrastructure/proxmox.md
|
||||
- [ 0d] infrastructure/tailscale.md
|
||||
- [ 16d] queries/retained-research-answers.md
|
||||
- [ 3d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 7d] reports/uber-eats/2026-07.md
|
||||
- [ 7d] reports/uber-eats/2026-08.md
|
||||
- [ 7d] reports/uber-eats/2026-09.md
|
||||
- [ 7d] reports/uber-eats/index.md
|
||||
- [ 8d] runbooks/backup-wiki.md
|
||||
- [ 8d] runbooks/diagnose-docker.md
|
||||
- [ 8d] runbooks/gateway-resource-resilience.md
|
||||
- [ 0d] runbooks/job-radar.md
|
||||
- [ 8d] runbooks/multiplexer-setup.md
|
||||
- [ 2d] runbooks/provider-health.md
|
||||
- [ 8d] runbooks/recover-docker-service.md
|
||||
- [ 8d] runbooks/restart-browser.md
|
||||
- [ 8d] runbooks/restart-hermes.md
|
||||
- [ 8d] runbooks/restore-openrouter.md
|
||||
- [ 8d] runbooks/test-openrouter.md
|
||||
- [ 8d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 2d] runbooks/update-hermes-safely.md
|
||||
- [ 15d] systems/browser-backend.md
|
||||
- [ 8d] systems/cloudflare-tunnel.md
|
||||
- [ 0d] systems/conduit.md
|
||||
- [ 2d] systems/current-profile.md
|
||||
- [ 8d] systems/guanaco.md
|
||||
- [ 8d] systems/headless-server.md
|
||||
- [ 0d] systems/hermes-agent.md
|
||||
- [ 8d] systems/hermes-ui.md
|
||||
- [ 16d] systems/home-assistant.md
|
||||
- [ 3d] systems/homepage-dashboard.md
|
||||
- [ 17d] systems/ignis.md
|
||||
- [ 2d] systems/memory-backend.md
|
||||
- [ 2d] systems/messaging-integrations.md
|
||||
- [ 4d] systems/mnemosyne.md
|
||||
- [ 2d] systems/model-providers.md
|
||||
- [ 16d] systems/nanobot.md
|
||||
- [ 8d] systems/nextcloud.md
|
||||
- [ 8d] systems/obsidian.md
|
||||
- [ 6d] systems/omniroute.md
|
||||
- [ 8d] systems/opencode-go.md
|
||||
- [ 8d] systems/openrouter.md
|
||||
- [ 17d] systems/paseo.md
|
||||
- [ 9d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 8d] systems/terminal-backend.md
|
||||
- [ 8d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (4)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- index.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 2 stale, 103 fresh, 4 no-date of 109 pages
|
||||
+22
-4
@@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-27
|
||||
updated: 2026-10-28
|
||||
updated: 2026-10-29
|
||||
review_after: 2026-11-10
|
||||
verified_on: 2026-10-28
|
||||
verified_on: 2026-10-29
|
||||
version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2)
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, dashboard, runbook]
|
||||
@@ -326,13 +326,31 @@ now owns a job of its own and the writes own a second one.
|
||||
- Stage 1 keeps the `custom:omni-101` / `smart-route` pin. Both jobs still inherit the profile-level
|
||||
`fallback_providers` chain; Hermes stores no per-job fallback list.
|
||||
- First fires: two test runs of stage 2 on 2026-10-27 (17:35:00 → 17:36:52 and 17:38:38 → 17:43:42), both
|
||||
`completed` with delivery `suppressed`. **The first real 07:00 → 11:30 cycle had not yet run at the
|
||||
2026-10-28 audit.**
|
||||
`completed` with delivery `suppressed`. **The first real 07:00 → 11:30 cycle
|
||||
ran on 2026-10-28 and landed nothing** — stage 1 never wrote its handoff (see the 2026-10-28 slot below).
|
||||
|
||||
**Recovery if stage 2 lands nothing:** check that stage 1 actually wrote its handoff file before running stage 2
|
||||
by hand (`hermes cron run 5fc292ff0f3b`). A stage-2 run with no handoff is a no-op, not a failed search — do not
|
||||
re-run stage 1's discovery just to satisfy it.
|
||||
|
||||
### 2026-10-28 07:00 slot — stage 1 hit the iteration cap, so stage 2 had nothing to land
|
||||
|
||||
Stage 1 (`dca8482e4f76`) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**) and closed **`failed`** on
|
||||
*"hit the agent iteration cap before the handoff could be written"* — its **second** consecutive failure
|
||||
(`failure_streak: 2`). No handoff file was written: `~/.hermes/radar-ref/handoff/` held only `2026-10-27.json`
|
||||
at the 2026-10-29 audit.
|
||||
|
||||
Both stage-2 slots then behaved exactly as designed — **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both
|
||||
`completed`, both `delivery_outcome: suppressed`. **A stage-2 run with no handoff is a silent no-op, not a
|
||||
failure**, which is the recovery rule below holding in production for the first time.
|
||||
|
||||
Read with 10-27, the picture is: *the split stops the writes from being lost, but it does not make discovery
|
||||
finish.* Two runs, two different ceilings — 10-27 stopped at the **tool-call budget** after 4 h 2 m (writes not
|
||||
started), 10-28 stopped at the **agent iteration cap** after 3 h 32 m, earlier still (no handoff at all). The
|
||||
discovery stage is the thing to bound, not the wall clock.
|
||||
|
||||
Full ledger: [[incidents/2026-10-28-job-radar-stage1-iteration-cap]].
|
||||
|
||||
## Browser-layer hazard inside the run (2026-10-12)
|
||||
|
||||
This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Browser Backend
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-13
|
||||
verified_on: 2026-10-13
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
confidence: high
|
||||
tags: [browser, camofox, automation]
|
||||
sources: [live-probe, config, session-history]
|
||||
@@ -20,7 +20,7 @@ Browser automation for Hermes — web search, form filling, screenshot, CDP cont
|
||||
- **Provider config:** `browser.cloud_provider: camofox`, `camera_provider: camofox`
|
||||
- **REST API:** **`:9377`** — live-verified 2026-10-13 (`node server.js`, `CAMOFOX_PORT=9377`; `LISTEN *:9377`). **The `:9093` figure in earlier revisions of this page is stale** — nothing was listening on 9093 on 2026-10-13.
|
||||
- **VNC viewer:** NovNC `websockify` on `127.0.0.1:6080` → `127.0.0.1:5900`, published as `https://ct460.kangaroo-eel.ts.net/vnc-camofox`
|
||||
- **Steel Browser (CT201):** Still running but Hermes not connected — plugin disabled
|
||||
- **Steel Browser (CT201):** **Not reachable as of 2026-10-29 00:3x** — host answers ping (2.5 ms) but CDP `:9223` and REST `:3000` refuse; plugin removed. See the 2026-10-29 correction below.
|
||||
|
||||
## Camofox (local `systemd --user` service)
|
||||
|
||||
@@ -109,6 +109,29 @@ until the next session starts.
|
||||
the Hermes host (CT460). Containment drop-in documented. Same day, the 07:01 Job Radar session was audited
|
||||
end-to-end and its memory-pressure evidence added above.
|
||||
|
||||
## Correction (2026-10-29) — Steel is out of the stack, not just disabled
|
||||
|
||||
Verified against live state at **2026-10-29 00:3x AWST**:
|
||||
|
||||
- **Hermes is back on Camofox.** `config.yaml` (mtime **2026-10-29 00:15:13**) reads
|
||||
`browser.cloud_provider: camofox`, `browser.camera_provider: camofox`, `engine: auto`, `cdp_url: ""`.
|
||||
- **The `browser-steel` plugin is gone** — `~/.hermes/plugins/browser-steel/` no longer exists and
|
||||
`plugins.enabled` carries no steel entry, so the `plugin_steel` toolset cannot be selected. Backup taken
|
||||
immediately before the change: `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` (written
|
||||
**2026-10-29 00:13:24**). A second backup, `config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z`, records
|
||||
the 2026-10-28 22:00 attempt.
|
||||
- **The Steel endpoint is not serving.** `ping 100.96.244.39` (runtipi/CT201) answers **2.5 ms**, but
|
||||
`http://100.96.244.39:9223/json/version` (CDP) and `http://100.96.244.39:3000/v1/health` (REST) both return
|
||||
**000** — as does the LAN address `192.168.178.77:9223`. The tunnel route
|
||||
`https://runtipi.kangaroo-eel.ts.net/steel` returns **502**, and `steel.martinwa.org` no longer resolves.
|
||||
Earlier pages describing Steel as "still running on CT201" are superseded by this probe.
|
||||
- ⚠ **Stale environment variable still exported:** `STEEL_BASE_URL=http://100.96.244.39:3000` is present in
|
||||
**10 live Hermes processes** — the gateway tree (`3371780`, `3372125`, `3372912`, `3844013`, `3880214`,
|
||||
`3880663`), `agent-browser` 0.26.0 (`3843841`), two chromium crashpad handlers (`3843878`, `3843880`) and
|
||||
`agentmail-webhook.py` (`402`). It now points at a dead endpoint and clears only when the gateway is
|
||||
restarted. The config is explicit (`cloud_provider: camofox`), so it does not currently misroute browser
|
||||
calls — but it is a trap for anything that reads the variable.
|
||||
|
||||
## Related
|
||||
- [[comparisons/steel-vs-camoufox]]
|
||||
- [[concepts/browser-automation]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-28
|
||||
verified_on: 2026-10-28
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
review_after: 2026-11-03
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-10-28 00:2x AWST (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed.
|
||||
- **Current as of 2026-10-29 00:3x AWST (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real and the discovery stage still could not finish: stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **removed** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and the endpoint is **not serving** — CDP `:9223`/REST `:3000` refuse, tunnel `/steel` **502**, `steel.martinwa.org` unresolved — while **`STEEL_BASE_URL=http://100.96.244.39:3000` is still exported into 10 live Hermes processes** and clears only on a gateway restart → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.**
|
||||
- **Prior window — as of 2026-10-28 00:2x AWST** (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. *(superseded by the 2026-10-29 entry above)*
|
||||
- **Prior window — as of 2026-10-27 00:1x AWST *(superseded by the 2026-10-28 entry above)* (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`).
|
||||
- **Current as of 2026-10-26 00:1x AWST** *(superseded by the 2026-10-27 entry above)* **(live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`).
|
||||
- **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page. *(Superseded by the 2026-10-26 entry above.)*
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-27
|
||||
verified_on: 2026-10-27
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
confidence: high
|
||||
tags: [skills, hermes]
|
||||
sources: []
|
||||
@@ -531,3 +531,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
## Yuanbao (1)
|
||||
|
||||
- `yuanbao` — # Yuanbao Group Interaction
|
||||
|
||||
## Recent stocktake runs
|
||||
|
||||
- **2026-10-28 03:00** — regenerated this index (**302 skills**, mtime 03:01:40) and wrote `raw/stocktake/freshness-20261028.txt` (**2 stale / 103 fresh / 4 no-date of 109 pages**), then **failed** before committing: `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` (provider `custom:omni-101 / free-stack`). Incident `1a8e90_f57ab4cafab8` (`detected`, no alert). This index and the freshness file were left uncommitted and were committed by the 2026-10-29 wiki audit — see [[systems/scheduled-tasks]] and [[incidents/2026-10-28-job-radar-stage1-iteration-cap]].
|
||||
|
||||
Reference in New Issue
Block a user