diff --git a/incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md b/incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md index 7a83a42..db74a39 100644 --- a/incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md +++ b/incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md @@ -1,17 +1,22 @@ --- -title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open +title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Recovered and Delivered 7 h 26 m Later type: incident -status: active +status: resolved created: 2026-11-02 -updated: 2026-11-02 -verified_on: 2026-11-02 -review_after: 2026-11-09 +updated: 2026-11-03 +verified_on: 2026-11-03 +review_after: 2026-11-17 confidence: high tags: [cron, gateway, handoff, delivery, weekly-review, silent-failure] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-jobs-json, live-logs, live-probe] --- -# Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open +# Weekly Review Lost at 18:00 on 2026-11-01 — Recovered and Delivered 7 h 26 m Later + +> **Status: RESOLVED 2026-11-02T01:27:21+08:00.** The Sunday 18:00 run did *not* lose the review — it +> resumed, completed and delivered at **01:27:21**, and the incident row auto-closed. The body below is +> the record **as it stood at the 2026-11-02 audit**, before the recovery landed. See +> [Resolution](#resolution-2026-11-02) for the corrected outcome and the answers to the open questions. ## Headline @@ -79,6 +84,8 @@ appearance of the "green/red row, no artefact" anti-pattern the wiki already doc ## Open questions +> ⚠ **Questions 1–3 are answered in [Resolution](#resolution-2026-11-02).** Question 4 remains open. + 1. **What did "Gateway connection failed; no local fallback" mean?** Same message for two different jobs 11 ms apart, both in the same handoff batch, with no gateway restart anywhere in the window. 2. **Why did `b63ed5` self-heal in 68 s and `79f9bb` never do so?** Both were in the same batch with the @@ -88,6 +95,47 @@ appearance of the "green/red row, no artefact" anti-pattern the wiki already doc 4. Are the two unreconciled admission files (`141448f9…`, `69b5588f…`) the durable record needed to decide whether either run had side effects? Reconcile before the next 18:00 Sunday slot. +## Resolution (2026-11-02) + +**The run was not lost. It resumed, finished, and delivered — 7 h 26 m 43 s after it started.** + +| Field | Value | +|---|---| +| Execution | `be6ec11a8e68428084af9ab18fb148db` — **the same execution record** the audit saw stuck at `unknown` | +| `scheduled_instant` | `2026-11-01T10:00:00+00:00` (18:00 AWST) | +| `started_at` | `2026-11-01T18:00:39.125776+08:00` | +| `finished_at` | **`2026-11-02T01:27:22.000881+08:00`** | +| `status` | **`completed`** | +| `error` | `NULL` | +| `delivery_outcome` | **`delivered`** | +| Delivery row | `2026-11-02T01:27:21.802246+08:00` — `delivered` | +| Incident row | `79f9bb_9c443b602eec` → `state: resolved`, `closed_at: 2026-11-02T01:27:21.606691+08:00` | + +Anthony received the Weekly Review at 01:27 AWST on 2026-11-02. There is still exactly **one** execution +row for the 18:00:39 slot, so this is a reconciliation of the existing claim rather than a re-run — the +`unknown` terminal state at 18:05:13 was replaced when the run finally wrote its real outcome. + +**Answers to the open questions:** + +1. *(the "no local fallback" message)* — the handoff to the external worker did ultimately succeed; the + 18:05:13 message recorded a **failed reconciliation attempt**, not a failed run. The run kept executing + for a further 7 h 26 m and produced its deliverable. The message is therefore a **false negative from + the reconciler**, not evidence that the work was abandoned. +2. **`b63ed5` closed in 68 s; `79f9bb` closed in 7 h 26 m — both eventually reached `completed`/`delivered`.** + The difference was run duration, not failure mode. The reconciler's window is shorter than the Weekly + Review's runtime, so any long-running job in a handoff batch will be reported `unknown` while it is + still working. **This is the general lesson: an `unknown` execution state on a long job is a timing + artefact, not a verdict.** +3. **No manual re-run was needed** — the run self-healed and delivered 12 minutes before the 2026-11-02 + audit would have recommended one. Had the audit re-triggered the job, it would have duplicated the run. + +### Consequences for the audit method + +The 2026-11-02 audit read `status: unknown` + `closed_at: NULL` at 00:4x and concluded the deliverable was +lost. Both were true at the time of reading and both had inverted by 01:27. **A nightly audit that runs +~2 h after midnight will read every long-running job mid-flight and must not treat a null terminal state as +final until the job's expected runtime has elapsed** — see [[systems/scheduled-tasks]] §9. + ## Related - [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] diff --git a/incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md b/incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md new file mode 100644 index 0000000..6e7774a --- /dev/null +++ b/incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md @@ -0,0 +1,118 @@ +--- +title: Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete Run Recorded `completed`, Output Suppressed +type: incident +status: active +created: 2026-11-03 +updated: 2026-11-03 +verified_on: 2026-11-03 +review_after: 2026-11-10 +confidence: high +tags: [cron, job-radar, iteration-cap, silent-failure, notion, camofox] +sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-state-db, live-jobs-json, session-history] +--- + +# Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete Run Recorded `completed`, Output Suppressed + +## Headline + +**Third occurrence of the Stage-1 budget class, and the first that the cron ledger reports as healthy.** +On **2026-11-02** the 07:00 `dca8482e4f76` (*Daily Marketing Job Radar — Stage 1 (Discovery)*) ran for +**9 h 01 m 19 s** (`started 07:00:33.378239` → `finished 16:01:52.496335 +08:00`), consumed the harness's +entire tool-calling iteration budget, and closed with: + +| Field | Value | +|---|---| +| `status` | **`completed`** | +| `error` | `NULL` | +| `delivery_outcome` | **`suppressed`** | + +No handoff artefact was written. **Nothing reached Notion, no cover letters were generated, and Stage 2 was +never triggered.** Because the run's own status is `completed` with no error, a health check that inspects +`executions.status` alone reports this day as green — the failure is invisible in the ledger. + +## What the run did and did not do + +The session (`Daily Marketing Job Radar - Discovery · Nov 02 16:01`, `local-368c8fe9…`) shows the discovery +half genuinely worked: + +- Camofox was restarted at run start after its HTTP server wedged (see [[systems/browser-backend]]). +- **SEEK 51 unique roles, LinkedIn 60 unique job IDs, Indeed live, WA Government 35 results** — 197 raw + candidates, **164 new after three-level dedup** against the 479-row Notion board. +- A **URL-canonicalisation bug was found and fixed mid-run**: `au.seek.com` vs `au.seek.com.au` had been + dropping **SEEK and Indeed entries** from the consolidated list; the fix was a job-ID-based + canonicalisation. +- Adverts were individually verified fresh and active; salaries, closing dates and JD bodies were pulled. + +Then the ceiling hit. The runtime injected *"You've reached the maximum number of tool-calling iterations +allowed. Please provide a final response…"* while the agent was still reading JD text for the borderline +finalists, and the run terminated in the **scoring** phase. + +The agent's own closing report was explicit — and it was **not delivered**: + +> *"**Status: INCOMPLETE — handoff file NOT written, Stage 2 NOT triggered.** … Because the mandatory +> deliverable (the handoff JSON) was not produced, today's roles have **not** been handed to Stage 2."* + +That text is `delivery_outcome: suppressed`, so Anthony saw nothing on 11-02 — not the roles and not the +warning that the roles were missing. + +## Downstream: both Stage 2 slots behaved correctly + +| Slot | Session | Ran | Result | +|---|---|---|---| +| 11:30 | `local-92665337…` (*Write and Letters*) | 11:30 → 12:25 | Exited `[SILENT]` after verifying Stage 1 was alive but had no handoff | +| 14:30 (safety net) | `local-05a3f00b…` (*Write and Letters*) | 14:30 → 14:30 | Exited `[SILENT]` — 6 messages, 2 tool calls | + +Both declines were correct: no handoff existed to consume. The safety-net design worked as intended — the +gap is that Stage 1 cannot finish inside its budget. + +## Recurrence pattern + +| Date | Incident page | Failure shape | +|---|---|---| +| 2026-10-27 | [[incidents/2026-10-27-job-radar-budget-exhausted]] | Tool-call budget exhausted before the Notion write stage | +| 2026-10-28 | [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] | Stage 1 hit the agent iteration cap before the handoff | +| **2026-11-02** | **this page** | Same cap; **run recorded `completed` and its report suppressed** | + +Three hits in seven days, each ending in the same place: discovery complete, handoff absent, Notion write +stage never reached. + +## The "green row, no artefact" anti-pattern + +This is the **fourth appearance** of the class [[systems/scheduled-tasks]] §9 of the maintenance playbook +already documents (a job whose durable ledger entry says success while the thing it was supposed to produce +does not exist). The new variant here is that the ledger flipped from red to green: + +| Date | `cron_incidents` row for `dca8482e4f76` | `executions.status` | +|---|---|---| +| 2026-10-27 | `dca848_e93a538ade6b` (`failure_type: auth`) | non-clean | +| 2026-10-28 | `dca848_0a2f03b5952b` (`failure_type: timeout`) | non-clean | +| **2026-11-02** | **none raised** | **`completed`, `error: NULL`** | + +The two earlier caps each raised an incident whose `error` text is literally the run's INCOMPLETE report. +The 2026-11-02 cap raised **nothing at all** (`cron_incidents` holds no row for `dca8482e4f76` with a +`first_seen_at` on 2026-11-02; fleet-wide unclosed rows were **0** at the 2026-11-03 audit). The run's +`delivery_outcome` is `suppressed`, so no message reached Anthony either. + +**Detection rule:** for any job whose deliverable is a *file* (a handoff JSON, a report, a generated page), +`executions.status` is not sufficient evidence. Verify the artefact exists at its expected path, or that +`cron_incidents` stayed quiet for a reason, before treating the slot as served. Silence from +`cron_incidents` is not the same as success — here it is the *symptom*. + +## Open questions + +1. **Is the iteration budget the right control, or is the task too large for one agent turn?** Three caps in + a row suggest Stage 1's work (four job boards → dedup → individual advert verification → JD reading → + scoring) does not fit a single turn at any budget. +2. **Should the harness record a cap-hit as something other than `completed`?** A distinct terminal status + (or a non-null `error`) would make this visible to every existing health check. +3. **Should the run's final response be delivered even when the job "succeeded"?** Suppressing an INCOMPLETE + report converts a loud failure into silence. +4. **Did the 11-02 roles get picked up on the next day's run**, or is 2026-11-02's candidate set lost? + The 164 deduped candidates were never persisted. + +## Related + +- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] +- [[incidents/2026-10-27-job-radar-budget-exhausted]] +- [[systems/scheduled-tasks]] +- [[systems/browser-backend]] diff --git a/index.md b/index.md index dfffee7..0511adc 100644 --- a/index.md +++ b/index.md @@ -3,15 +3,15 @@ title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 -updated: 2026-11-02 -verified_on: 2026-11-02 +updated: 2026-11-03 +verified_on: 2026-11-03 confidence: high tags: [index, infrastructure, systems] --- # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-11-02 (nightly audit — the 11-01 fleet settled **167 / 1 / 2**, its first non-clean day in four: the Sunday **18:00 Weekly Review never ran and never delivered** (`79f9bb41ecf5`, *"Gateway connection failed; no local fallback"*, incident still open) and **three cron workers were `SIGTERM`ed (`-15`) mid-adoption** without writing a durable terminal state (`85edf6772bca` 07:30:46; `96eb95ce12f2` + `0e753e9720fd` 09:02:18) — for the first time with **no gateway restart anywhere in the window**, so the orphan class is now known to fire without a reload; a **fourth variant** is documented where **`ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning no execution row and no incident row**; delivery went **7 of 8**, the miss being `066bb5191280`'s `thread_id 34795` home-DM thread fallback (content delivered, run marked `failed`), while **Telegram flapped all day** (dual-stack `api.telegram.org` timeouts, disconnect 20:51:53, restarts 22:33:30 / 23:12:23) and the **relay adapter has now been absent ~29 h** (`Reconnecting relay` attempt 336); the longest-standing incident on the board — the 2026-10-01 Facebook-repin row — finally **closed at 10:44:12**) +> model routing and automation. Last updated: 2026-11-03 (nightly audit — the 11-02 fleet settled **177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`**, the **first fully clean ledger day in five**, and **both 11-01 `unknown` rows resolved in-window**: the Sunday 18:00 **Weekly Review reconciled at 01:27:22** after **7 h 26 m 43 s** and **delivered**, so *"the lost review" was never lost* — incident `79f9bb_9c443b602eec` auto-closed **01:27:21.606691** and its page is now **`resolved`** — while `ecc1fc13655a`'s 11-01 ghost `error` cleared on its 12:00:33 slot; **unclosed incident rows 1 → 0** (the first zero in the period), non-clean `last_status` **2 → 1**, deliveries **9 rows / 9 delivered**; ⚠ **but two ~9-hour runs failed invisibly.** The **Job Radar Stage 1** (`dca8482e4f76`, 07:00:33 → 16:01:52, **9 h 01 m 19 s**) hit the **agent iteration cap before writing its handoff**, so **both Stage 2 slots exited `[SILENT]` and nothing reached Notion** — and it **raised no `cron_incidents` row at all**, where the same class raised incidents on 10-27 and 10-28, because its terminal state flipped to `completed` with `error: NULL`; the **03:00 stocktake** (`1a8e90ea2271`) likewise ran **9 h 43 m 29 s**, **delivered no report** and left its output uncommitted in the working tree (committed by this audit).) > > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. @@ -65,7 +65,8 @@ tags: [index, infrastructure, systems] - [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since *(superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → [[incidents/2026-10-29-conduit-voice-silence]])*. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]] - [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]] - [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery. -- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] — **The Sunday 18:00 Weekly Review did not run on 2026-11-01 and nothing was delivered:** its execution closed `unknown` at 18:05:13 with *"Cron execution unverified … Gateway connection failed; no local fallback"*, its incident row is **still `alerted`**, the job sits at `error` / `failure_streak: 1`, and `deliveries.db` shows **no row after 11:01:12** for the rest of the day. Its batch-mate `b63ed5817212` took the identical error and self-healed in 68 s. Same page records the **inverse silent failure**: `ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning **no execution row and no incident row at all**. +- [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] — **Third iteration-cap in seven days, and the first that raised no incident at all.** Job Radar Stage 1 (`dca8482e4f76`) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** on 2026-11-02, hit the **agent iteration cap before the handoff was written**, and closed **`status: completed` / `error: NULL` / `delivery_outcome: suppressed`**. Both Stage 2 slots declined correctly (`92665337…` 11:30 and the `05a3f00b…` 14:30 safety net both exited `[SILENT]`), so **nothing reached Notion, no cover letters were generated, and the day's 164 deduped candidates were never persisted** — a state a `status`-only health check reports as green. Contrast 10-27 (`dca848_e93a538ade6b`) and 10-28 (`dca848_0a2f03b5952b`), where the identical cap **did** open rows whose `error` text is the run's own INCOMPLETE report: on 11-02 the ledger **lost the red row it used to produce**. Discovery itself worked — SEEK **51**, LinkedIn **60 IDs**, Indeed live, WA Government **35** (197 raw / **164 new** after three-level dedup) — and a **URL-canonicalisation bug was found and fixed mid-run** (`au.seek.com` vs `au.seek.com.au` had been dropping SEEK *and* Indeed entries from the consolidated list). → [[systems/scheduled-tasks]] +- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] — **The Sunday 18:00 Weekly Review did not run on 2026-11-01 and nothing was delivered:** its execution closed `unknown` at 18:05:13 with *"Cron execution unverified … Gateway connection failed; no local fallback"*, its incident row is **still `alerted`**, the job sits at `error` / `failure_streak: 1`, and `deliveries.db` shows **no row after 11:01:12** for the rest of the day. Its batch-mate `b63ed5817212` took the identical error and self-healed in 68 s. Same page records the **inverse silent failure**: `ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning **no execution row and no incident row at all**. ✅ **RESOLVED 2026-11-02T01:27:21 — the review was never lost.** The *same* execution row (`be6ec11a8e68428084af9ab18fb148db`) closed **`completed` / `delivered` at 01:27:22** after **7 h 26 m 43 s**, the delivery lands at **01:27:21.802246** and incident `79f9bb_9c443b602eec` auto-closed at **01:27:21.606691** — so the 18:05:13 `unknown` was a **reconciler timing artefact on a still-running job**, not a verdict (page now `resolved`). Batch-mate `b63ed5` closed in 68 s and `79f9bb` in 7 h 26 m, both `completed`; **an `unknown` row on a long job is a timing artefact, not a failure**. `ecc1fc13655a`'s ghost `error` cleared too (12:00:33 → `completed` / `delivered` 12:29:26), taking **non-clean `last_status` 2 → 1**. - [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] — **Three restart-safe cron workers killed by `SIGTERM` (`-15`) after adopting their claim**, so the scheduler recorded *"whether side effects ran is unknown"*: `85edf6772bca` @07:30:46, `96eb95ce12f2` + `0e753e9720fd` @09:02:18. **A fourth variant of the orphan class** — and the first with **no gateway restart, reload or drain anywhere in the window** (PID `175269` up since 10-31 19:24:40; relay counter monotonic through both signals), so it cannot be attributed to reload behaviour. Likely trigger: a **30 s worker-ack window** smaller than this host's cold start (**83 plugins / 69 enabled** discovered in 70 s). All three rows now `resolved`; one execution left `unknown`. - [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]] diff --git a/log.md b/log.md index f9b4141..64cc16a 100644 --- a/log.md +++ b/log.md @@ -1854,3 +1854,56 @@ recorded wrong.** (`origin/master..HEAD` empty) before this audit; touched pages were mirrored into the vault's existing conventions and hash-verified. **No secrets, passwords, tokens or keys written or quoted** — every job id, PID, thread id and timestamp above is a non-sensitive operational identifier. + +## [2026-11-03] audit | Nightly audit — 2026-11-02 window (first clean ledger day in five, and two ~9-hour failures it cannot see) + +- **Scope:** `state.db` sessions spanning 2026-11-02 00:00 → 2026-11-03 00:2x AWST; `cron/executions.db`, + `cron/deliveries.db`, `cron_incidents`, `cron/jobs.json` (all read-only). Authoring tree + `/home/hermes/workspace/agent-estate-wiki` was **clean and in sync with Gitea** (`origin/master..HEAD` + empty) before this audit; `systems/skills-index.md` and `raw/stocktake/freshness-20261102.txt` were + **committed on the stocktake's behalf** (see below). +- **NEW — `incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md`:** the **third iteration-cap + in seven days** (10-27 tool-call budget, 10-28 cap, 11-02 cap) on `dca8482e4f76` + (**07:00:33 → 16:01:52, 9 h 01 m 19 s**), and **the first that raised no `cron_incidents` row at all** — + it closed `completed` / `error: NULL` / `delivery_outcome: suppressed`. Both Stage 2 slots exited + `[SILENT]`, so **nothing reached Notion, no cover letters were written and the 164 deduped candidates + were never persisted**. Page carries the **"green row, no artefact" detection table** contrasting the + three occurrences, and records the mid-run **URL-canonicalisation fix** (`au.seek.com` vs + `au.seek.com.au`) that had been silently dropping SEEK and Indeed entries from dedup. +- **UPDATED — `incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md`:** status + **`active` → `resolved`**. Live evidence: the **same execution row** + (`be6ec11a8e68428084af9ab18fb148db`) closed `completed` / `delivered` at **01:27:22** after + **7 h 26 m 43 s**, delivery at 01:27:21.802246, incident `79f9bb_9c443b602eec` auto-closed at + **01:27:21.606691**. The 18:05:13 `unknown` was a **reconciler timing artefact on a still-running job**, + not a verdict. Resolution section added; open questions 1–3 answered, **question 4 retained** (the two + unreconciled admission files `141448f9…json` / `69b5588f…json`). +- **UPDATED — `systems/scheduled-tasks.md`:** `## Count` advanced to the 2026-11-02 window — + **177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows) vs 167/1/2 on 11-01, 160/4 on + 10-31, 138/0 on 10-30; fleet unchanged at **29 jobs / 26 enabled / 3 disabled**; **unclosed incident + rows 1 → 0** (the first zero in the period); both 11-01 `unknown` rows resolved and `ecc1fc13655a`'s + ghost `error` cleared; deliveries **9 rows / 9 `delivered`**. New **`## Re-verified 2026-11-03`** + section documents the two invisible failures, the stocktake artefacts committed, the + `substr(started_at,1,10)` grouping rule and the **"do not read `running`/`unknown` as a verdict before + the expected runtime has elapsed"** correction. Frontmatter `updated`/`verified_on` → 2026-11-03, + `review_after` → 2026-11-10. +- **UPDATED — `systems/browser-backend.md`:** `updated`/`verified_on` → 2026-11-03 plus a `## History` + entry for the **2026-11-02 REST-server wedge** (requests timing out with nothing reaching the service + log, cleared by `systemctl --user restart camofox-browser.service`; the four-board Job Radar run then + proceeded). **Live re-probe 2026-11-03 00:16:14 AWST:** unit `active`, `MainPID 953299`, + `ActiveEnterTimestamp Mon 2026-11-02 18:11:04`, `MemoryCurrent 37.5 MB`, `/health` → **200** + `{"ok":true,"engine":"camoufox","browserConnected":false,…}`, `/tabs` → `{"running":true,"tabs":[]}`. + Page now states explicitly that this wedge is a **different fault from the 2026-10-12 memory ceiling** + (hang at 31 MB RSS vs exhaustion at 2.5 G), so the containment limits are not the applicable control. +- **UPDATED — `index.md`:** `updated`/`verified_on` → 2026-11-03, the audit summary line rewritten for + this window, the new 11-02 incident link added, and the 11-01 Weekly Review entry annotated + **✅ RESOLVED 2026-11-02T01:27:21**. +- **COMMITTED on the stocktake's behalf:** `1a8e90ea2271` finished with `systems/skills-index.md` + modified and `raw/stocktake/freshness-20261102.txt` untracked. Both are valid artefacts — freshness + **4 stale / 113 fresh** at a >21-day threshold, skills-index regenerated with **Devops 108 → 109** + (`pulse-turn-handling`) — so this audit committed them rather than let the next slot overwrite them. +- **Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned; no + decision reversed. A full vault mirror pass was deliberately skipped again (drift counts instead of a + bulk copy). +- **DRIFT (context, not findings):** touched pages follow the vault's existing conventions and + SCHEMA.md frontmatter rules. **No secrets, passwords, tokens or keys written or quoted** — every job + id, PID, thread id and timestamp above is a non-sensitive operational identifier. diff --git a/raw/stocktake/freshness-20261102.txt b/raw/stocktake/freshness-20261102.txt new file mode 100644 index 0000000..f58810f --- /dev/null +++ b/raw/stocktake/freshness-20261102.txt @@ -0,0 +1,133 @@ +# Wiki freshness audit — 2026-11-02 +Threshold: >21 days since `updated` + +## STALE (4) + +- [ 22d] agents/qwenpaw.md (updated 2026-10-11) +- [ 22d] incidents/2026-09-28-hermes-managed-runtime-deps.md (updated 2026-10-11) +- [ 22d] systems/ignis.md (updated 2026-10-11) +- [ 22d] systems/paseo.md (updated 2026-10-11) + +## FRESH (113) + +- [ 13d] CONTRIBUTING.md +- [ 7d] agents/hermes-chaos.md +- [ 7d] agents/hermes-production.md +- [ 13d] agents/index.md +- [ 4d] agents/nanobot.md +- [ 13d] agents/open-claw.md +- [ 13d] agents/zeroclaw.md +- [ 4d] comparisons/camoufox-vs-chrome.md +- [ 13d] comparisons/hermes-vs-nanobot.md +- [ 13d] comparisons/memory-backends.md +- [ 13d] comparisons/model-routers.md +- [ 4d] comparisons/steel-vs-camoufox.md +- [ 13d] concepts/agent-memory.md +- [ 4d] concepts/browser-automation.md +- [ 13d] concepts/model-routing.md +- [ 13d] concepts/persistent-shells.md +- [ 13d] concepts/tool-calling.md +- [ 2d] current-state.md +- [ 4d] decisions/index.md +- [ 13d] decisions/keep-openrouter-as-fallback.md +- [ 13d] decisions/persistent-shell-enabled.md +- [ 4d] decisions/rollback-steel-to-camofox.md +- [ 13d] experiments/index.md +- [ 9d] incidents/2026-07-29-mnemosyne-relay.md +- [ 13d] incidents/2026-08-01-mnemosyne-capture-failure.md +- [ 9d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- [ 11d] incidents/2026-08-08-gitea-config-backup-push-dead.md +- [ 10d] incidents/2026-08-15-skills-index-drift.md +- [ 21d] incidents/2026-08-18-mnemosyne-capture-pipeline.md +- [ 8d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 13d] incidents/2026-08-29-gmail-read-state.md +- [ 12d] incidents/2026-09-02-sftpgo-outage.md +- [ 4d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 7d] incidents/2026-10-03-ct450-hermes-update.md +- [ 7d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 4d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 9d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 1d] incidents/2026-10-09-memory-recall-relevance.md +- [ 17d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 6d] incidents/2026-10-16-omniroute-503-resource-pressure.md +- [ 14d] incidents/2026-10-18-degenerate-stream-canned-substitute.md +- [ 6d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md +- [ 11d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md +- [ 8d] incidents/2026-10-22-conduit-memory-start-timeout.md +- [ 7d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md +- [ 7d] incidents/2026-10-24-conduit-call-request-refused.md +- [ 7d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md +- [ 5d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md +- [ 3d] incidents/2026-10-27-job-radar-budget-exhausted.md +- [ 3d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md +- [ 3d] incidents/2026-10-29-conduit-voice-silence.md +- [ 2d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md +- [ 3d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md +- [ 2d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md +- [ 1d] incidents/2026-10-31-asana-access-failures.md +- [ 1d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md +- [ 1d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md +- [ 0d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md +- [ 0d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md +- [ 0d] index.md +- [ 11d] infrastructure/backups.md +- [ 13d] infrastructure/credentials-map.md +- [ 1d] infrastructure/docker-services.md +- [ 13d] infrastructure/domains-and-tunnels.md +- [ 2d] infrastructure/hosts.md +- [ 19d] infrastructure/network-map.md +- [ 1d] infrastructure/proxmox.md +- [ 1d] infrastructure/tailscale.md +- [ 21d] queries/retained-research-answers.md +- [ 8d] reports/communities-wa-social-media-audit-2026.md +- [ 1d] reports/uber-eats/2026-07.md +- [ 1d] reports/uber-eats/2026-08.md +- [ 1d] reports/uber-eats/2026-09.md +- [ 1d] reports/uber-eats/2026-10.md +- [ 1d] reports/uber-eats/index.md +- [ 13d] runbooks/backup-wiki.md +- [ 13d] runbooks/diagnose-docker.md +- [ 13d] runbooks/gateway-resource-resilience.md +- [ 3d] runbooks/job-radar.md +- [ 13d] runbooks/multiplexer-setup.md +- [ 7d] runbooks/provider-health.md +- [ 4d] runbooks/recover-docker-service.md +- [ 13d] runbooks/restart-browser.md +- [ 13d] runbooks/restart-hermes.md +- [ 13d] runbooks/restore-openrouter.md +- [ 13d] runbooks/test-openrouter.md +- [ 13d] runbooks/troubleshoot-tool-loop.md +- [ 7d] runbooks/update-hermes-safely.md +- [ 4d] systems/browser-backend.md +- [ 13d] systems/cloudflare-tunnel.md +- [ 1d] systems/conduit.md +- [ 7d] systems/current-profile.md +- [ 13d] systems/guanaco.md +- [ 13d] systems/headless-server.md +- [ 3d] systems/hermes-agent.md +- [ 13d] systems/hermes-ui.md +- [ 21d] systems/home-assistant.md +- [ 3d] systems/homepage-dashboard.md +- [ 7d] systems/memory-backend.md +- [ 0d] systems/messaging-integrations.md +- [ 1d] systems/mnemosyne.md +- [ 7d] systems/model-providers.md +- [ 21d] systems/nanobot.md +- [ 13d] systems/nextcloud.md +- [ 13d] systems/obsidian.md +- [ 1d] systems/omniroute.md +- [ 13d] systems/opencode-go.md +- [ 13d] systems/openrouter.md +- [ 14d] systems/push-notifications.md +- [ 0d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md +- [ 13d] systems/terminal-backend.md +- [ 13d] systems/tool-search.md + +## NO updated DATE (3) + +- SCHEMA.md +- experiments/active/example.md +- log.md + +TOTAL: 4 stale, 113 fresh, 3 no-date of 120 pages \ No newline at end of file diff --git a/systems/browser-backend.md b/systems/browser-backend.md index 538ecac..422aa17 100644 --- a/systems/browser-backend.md +++ b/systems/browser-backend.md @@ -3,8 +3,8 @@ title: Browser Backend type: system status: active created: 2026-07-22 -updated: 2026-10-29 -verified_on: 2026-10-29 +updated: 2026-11-03 +verified_on: 2026-11-03 confidence: high tags: [browser, camofox, automation] sources: [live-probe, config, session-history] @@ -112,6 +112,19 @@ until the next session starts. on **CT201** (containers, images, network, project dir, Serve routes) and **CT333** (container + image) torn down. Camofox is the sole backend. +- **2026-11-02 — the HTTP server wedged at the start of a browser-heavy cron run; a restart cleared it.** + The 07:00 Job Radar Stage 1 (`dca8482e4f76`) found the REST server accepting nothing — requests timing out + with **no lines reaching the service log** — before its first tab was opened. `systemctl --user restart + camofox-browser.service` cleared it and the run went on to work four boards. **Live re-probe + `2026-11-03 00:16:14 AWST`:** unit `active`, `MainPID 953299`, `ActiveEnterTimestamp Mon 2026-11-02 + 18:11:04`, `MemoryCurrent 37.5 MB`; `/health` → **200** + `{"ok":true,"engine":"camoufox","browserConnected":false,"browserRunning":false,"activeTabs":0,"activeSessions":0,"consecutiveFailures":0}`; + `/tabs` → `{"running":true,"tabs":[]}`. Idle-clean and far inside the containment drop-in. + ⚠ **This is a different fault from the 2026-10-12 ceiling:** that one was memory exhaustion under many + open tabs (`2.5G`, swap fully consumed); this one was a **hung HTTP server at 31 MB RSS**, so the + containment limits are not the control that applies. A health probe inside browser-heavy scripts should + therefore test **responsiveness**, not just `/health` status code. + ## Correction (2026-10-29) — Steel is out of the stack, not just disabled Verified against live state at **2026-10-29**. The earlier 00:3x probe said only "not reachable"; the truth is diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 8dd46dd..a40b1a0 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,9 +3,9 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-11-02 -verified_on: 2026-11-02 -review_after: 2026-11-09 +updated: 2026-11-03 +verified_on: 2026-11-03 +review_after: 2026-11-10 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db] @@ -14,7 +14,33 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count -- **Current as of 2026-11-02 00:4x AWST (live `jobs.json` + `executions.db` + `cron_incidents` + +- **Current as of 2026-11-03 00:1x AWST (live `jobs.json` + `executions.db` + `cron_incidents` + + `deliveries.db`) — the first fully clean ledger day in five, but two long runs ended in a silent cap and + one of them raised no incident at all.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged). + **Window 2026-11-02 settled 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows; + **96** of the completions are `96eb95ce12f2`'s 15-minute watchdog slots) against **167 / 1 / 2** on + 11-01, **160 / 4** on 10-31 and **138 / 0** on 10-30. **Both 11-01 `unknown` rows resolved on 11-02:** + the Sunday 18:00 Weekly Review reconciled at **01:27:22** (`be6ec11a8e68428084af9ab18fb148db`, + `completed` / `delivered`, **7 h 26 m 43 s** after it started) and its incident + `79f9bb_9c443b602eec` auto-closed at **01:27:21.606691** → + [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] (**now `resolved`; the "lost + review" was never lost**). `ecc1fc13655a`'s 11-01 ghost `error` also cleared — its 11-02 12:00:33 slot + ran `completed` / `delivered` at 12:29:26. **Unclosed incident rows 1 → 0 — the first zero in the + period.** **Non-clean `last_status` 2 → 1:** only `066bb5191280` (Uber Eats Monthly Spend Report, + `delivery_failed`, `failure_streak: 0`, `last_run 2026-11-01T07:18:57`) — the 11-01 DM-thread miss, which + **did not recur**; it has **no `executions` row on 11-02 at all**. **Delivery health: 9 rows on 11-02, + 9/9 `delivered`** (01:27:21, 07:30:35, 07:42:21, 08:10:18, 08:41:14, 09:07:20, 10:23:15, 12:29:25, + 18:45:37). ⚠ **Two long runs finished `completed` with `error: NULL` and `delivery_outcome: + suppressed` after ~9 h — and the first raised no incident:** (a) **`dca8482e4f76`** (Job Radar Stage 1) + **07:00:33 → 16:01:52 = 9 h 01 m 19 s**, hit the agent iteration cap before the handoff, so **both Stage 2 + slots (`92665337…` 11:30, `05a3f00b…` 14:30 safety net) exited `[SILENT]` and nothing reached Notion** — + the same class that raised incidents on 10-27 and 10-28 **raised none this time** because the terminal + state flipped to `completed` → [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]; + (b) **`1a8e90ea2271`** (daily stocktake) **03:00:31 → 12:44:00 = 9 h 43 m 29 s**, `suppressed` — it + **delivered no report** (unlike 11-01's 03:00:37 delivery) and **left its output uncommitted** + (`systems/skills-index.md` modified, `raw/stocktake/freshness-20261102.txt` untracked; the 2026-11-03 + audit committed both on its behalf). See **Known Issues → Open** below. +- **Prior window (2026-11-02 00:4x AWST, live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + gateway log) — the fleet ran clean all day, then failed in three different ways and missed a delivery.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged). **Window 2026-11-01 settled 167 `completed` / 1 `running` / 2 `unknown`** (+ this audit `running`, started 23:59:28) @@ -190,6 +216,23 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] ## Known Issues +### Resolved 2026-11-02 + +- **The "lost" Weekly Review was never lost — it delivered at 01:27:21, 7 h 26 m 43 s after its 18:00 slot.** + Execution `be6ec11a8e68428084af9ab18fb148db` (job `79f9bb41ecf5`, `scheduled_instant` + `2026-11-01T10:00:00+00:00`) started **2026-11-01T18:00:39.125776+08:00** and finally closed + **2026-11-02T01:27:22.000881+08:00** with `status: completed`, `error: NULL`, + `delivery_outcome: delivered`; the delivery row is timestamped **01:27:21.802246** and incident + `79f9bb_9c443b602eec` auto-closed at **01:27:21.606691**. **Same execution row, not a re-run** — the + `unknown` terminal state recorded at 18:05:13 was replaced when the run wrote its real outcome. + → [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] is now **`resolved`**. + **Method lesson for this audit:** the 2026-11-02 00:4x reading of `status: unknown` + `closed_at: NULL` + was accurate *at the time of reading* and had inverted 43 minutes later. **A nightly audit that runs + ~2 h after midnight must not treat a null terminal state on a long-running job as final** — the + reconciler's window is shorter than these runs, so `unknown` on a long job is a timing artefact, not a + verdict. The same false-negative logic applies to `ecc1fc13655a`, whose 11-01 ghost `error` cleared when + its 11-02 12:00:33 slot closed `completed` / `delivered` at 12:29:26. + ### Resolved 2026-10-30 - **The Mnemosyne Health Monitor was never really down — and it is enabled again (`ea79a2107e36`).** Its 00:01:08 @@ -236,6 +279,23 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] - **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`. ### Open +- **Two ~9-hour runs ended in a silent iteration cap on 2026-11-02, and the more damaging one raised no incident (NEW 2026-11-03):** + - **`dca8482e4f76`** (Job Radar Stage 1) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** and closed + `status: completed` / `error: NULL` / `delivery_outcome: suppressed` **after hitting the agent + iteration cap before the handoff was written**. Both Stage 2 slots declined (`92665337…` 11:30 and the + `05a3f00b…` 14:30 safety net both exited `[SILENT]`), so **nothing reached Notion**. This is the same + class that raised `dca848_e93a538ade6b` (10-27) and `dca848_0a2f03b5952b` (10-28) — whose `error` + columns are literally the runs' INCOMPLETE reports — yet **2026-11-02 produced no `cron_incidents` row + at all**, because the terminal state flipped from non-clean to `completed`. Three caps in seven days + (10-27, 10-28, 11-02) is a pattern, not bad luck. + → [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]. **Detection rule: for a job whose + deliverable is a file, `executions.status` is not evidence — check the artefact.** + - **`1a8e90ea2271`** (wiki stocktake) ran **03:00:31 → 12:44:00 = 9 h 43 m 29 s**, `completed` / + `suppressed`, and **delivered no report** (11-01 delivered at 03:00:37; 11-02 has no delivery row near + 03:00). It left its output **uncommitted** — `systems/skills-index.md` modified and + `raw/stocktake/freshness-20261102.txt` untracked — which the 2026-11-03 audit committed on its behalf. + The artefacts themselves are sound (freshness report: **4 stale / 113 fresh** at a >21-day threshold; + skills-index regenerated). - **GitHub Trending Radar's delivery token is revoked — needs a new @BotFather token (NEW 2026-10-13):** live `getMe` on the token in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401**, so `digest.py`'s Telegram send fails every Monday while the cron run itself stays green (`last_status: ok`). A *completed* run therefore does not mean the digest was sent by the script — the 2026-10-12 run only reached the DM because the agent recovered the content with `--dry-run` and its own response was delivered. Fix: issue a fresh token from @BotFather, replace it in `config.json`, and re-run `getMe` to confirm; do not re-read a green cron status as proof of delivery. **This corrects the 2026-10-05 "claim refuted" entry, which was wrong.** See [[systems/scheduled-tasks]] job `c0a9fa314542`. - **Camofox memory pressure during long browser runs (NEW 2026-10-13):** the 2026-10-12 07:01 Daily Marketing Job Radar run drove Camofox into the ceiling of its own systemd containment drop-in (`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`: `MemoryHigh=2G`, `MemoryMax=3G`, swap max 1G). Live probe mid-run read `Memory: 2.5G (high: 2G, max: 3G, swap: 1023.9M, available: 0B, peak: 2.5G)` with 383 tasks — swap fully consumed and no reclaimable headroom, which is why the `/tabs` endpoint and browser evals started hanging. The run still delivered: the agent stopped piling on memory, restarted the service, and finished one tab at a time with immediate close (post-restart: 2.1 GB used of 14 GB on CT460). Any job that opens many tabs in one pass is exposed to the same wall. Keep multi-tab extraction bounded, close each tab before opening the next, and consider raising the containment limits only if the host headroom justifies it. See [[systems/browser-backend]]. - **Stocktake's Quartz rebuild step is dead code (NEW 2026-10-07):** `wiki-maintain.sh` documents a third step — rebuild the Quartz site — but the 2026-10-07 03:00 run logged it as skipped: no `quartz.config.ts` is present and `npx` is unavailable, so `public/` has not been refreshed. The skills-index and freshness artifacts are regenerated and committed correctly; only the published-HTML step is inert. Either restore the Quartz toolchain or drop the step from the script and the job description. @@ -754,3 +814,72 @@ GitHub Trending Radar token; Camofox containment under multi-tab load; the Condu retention; **plus new:** the unreconciled admission files `~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json` (both 11-01) and the unidentified `-15` signal source. + +## Re-verified 2026-11-03 (audit window 2026-11-02 00:00 → 2026-11-03 00:2x AWST; live probes 00:0x–00:2x) + +**The first fully clean ledger day in five — and it contains two failures the ledger does not show.** + +**Settled: 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows) against 167 / 1 / 2 on +11-01, 160 / 4 on 10-31 and 138 / 0 on 10-30. Fleet unchanged at **29 jobs, 26 enabled, 3 disabled**. +Nothing was added, removed, disabled or repinned in the window. **96** of the completions are +`96eb95ce12f2`'s 15-minute watchdog slots. + +**Both 11-01 `unknown` rows resolved in-window** (see §Resolved 2026-11-02). **Unclosed incident rows +1 → 0** — the first zero in the period. **Non-clean `last_status` 2 → 1.** Deliveries **9 rows, 9/9 +`delivered`.** + +### Both faults of the day are invisible in the ledger + +| # | Fault | Job | What the ledger says | Page | +|---|---|---|---|---| +| 1 | **Iteration cap reached before the handoff** — 9 h 01 m 19 s, Stage 2 never triggered, **nothing reached Notion**, and **no `cron_incidents` row was raised** | `dca8482e4f76` 07:00:33 → 16:01:52 | `completed` / `error: NULL` / `delivery_outcome: suppressed` | [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] | +| 2 | **9 h 43 m run, no report delivered, output left uncommitted** | `1a8e90ea2271` 03:00:31 → 12:44:00 | `completed` / `error: NULL` / `delivery_outcome: suppressed` | this page → Known Issues → Open | + +Fault 1 is the **third cap in seven days** (10-27 tool-call budget, 10-28 iteration cap, 11-02 iteration +cap) and the **first that raised no incident at all** — the two earlier rows carry their runs' INCOMPLETE +reports verbatim in `cron_incidents.error`. That is the sharpest form of the §9 anti-pattern yet: the +ledger did not merely stay green over a dead job, it **lost the red row it used to produce**. Silence from +`cron_incidents` is not success. + +### Delivery health: 9 rows, 9 delivered + +`deliveries.db` for 11-02: **01:27:21 ✔** (the Weekly Review recovery), 07:30:35 ✔, 07:42:21 ✔, +08:10:18 ✔, 08:41:14 ✔, 09:07:20 ✔, 10:23:15 ✔, 12:29:25 ✔, 18:45:37 ✔. **The 11-01 `066bb5191280` +thread-miss did not recur.** That job has **no `executions` row on 11-02 at all** (`0 2 1 * *`, next slot +2026-12-01), so its 11-01 `delivery_failed` flag will sit on the fleet for the rest of November — the +fleet's **only** non-clean `last_status`. + +### Stocktake artefacts committed on the stocktake's behalf + +`1a8e90ea2271` finished with `systems/skills-index.md` modified and +`raw/stocktake/freshness-20261102.txt` untracked in the working tree. Both are valid artefacts — +freshness **4 stale / 113 fresh** at a >21-day threshold; skills-index regenerated with **Devops 108 → 109** +(`pulse-turn-handling` added) — so this audit **committed both** rather than leave them to be overwritten +by the next slot. + +### Method note (correction carried forward) + +Two rules now bind this audit's tallies: group by `substr(started_at,1,10)`, never `date(...)` (which +normalises to UTC and shifts 00:00–07:59 AWST rows onto the previous day); and **do not read a +`running`/`unknown` row as a verdict until the job's expected runtime has elapsed** — the Weekly Review +was still working at the 00:4x reading and delivered at 01:27, 43 minutes later. An audit that runs two +hours after midnight will read every long job mid-flight. + +### Pages touched + +`incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` (new), +`incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md` (status → **`resolved`**, +Resolution section + answers to its open questions 1–3 added), `systems/scheduled-tasks.md`, +`systems/browser-backend.md`, `systems/skills-index.md` (stocktake output, committed), +`raw/stocktake/freshness-20261102.txt` (stocktake output, committed), `index.md`, `log.md`. +**Nothing removed or archived:** no job retired, no system obsolete, no decision reversed in the window. + +### Still open (carried + new) + +The single-node no-failover topology behind every model-calling job; the GitHub Trending Radar delivery +token; Camofox containment under multi-tab load; the Conduit health store with no retention; the +unreconciled admission files `~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json`; the +unidentified `-15` signal source; and the **30 s worker-ack window vs this host's cold-start time**. +**Plus new:** whether 2026-11-02's **164 deduped Job Radar candidates are recoverable**, whether a +cap-hit should keep being recorded as `completed`, and why the 03:00 stocktake now takes ~9 h 45 m. + diff --git a/systems/skills-index.md b/systems/skills-index.md index 1e18fbc..30302ff 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-11-01 -verified_on: 2026-11-01 +updated: 2026-11-02 +verified_on: 2026-11-02 confidence: high tags: [skills, hermes] sources: [] @@ -12,7 +12,7 @@ sources: [] # Installed Skills -**Total installed: 301 skills** in `~/.hermes/skills/`. +**Total installed: 302 skills** in `~/.hermes/skills/`. - **46 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them. - The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory. @@ -102,7 +102,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit. - `ddia-systems` — # Designing Data-Intensive Applications Framework -## Devops (108) +## Devops (109) - `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS) - `agent-extension-audit` — # Agent Extension Audit @@ -191,6 +191,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit. - `omniroute-skill-caller` — # Call an OmniRoute Skill by Name (auto-resolving wrapper) - `omniroute-skill-execution` — # Execute Skills on OmniRoute MCP - `openclaw-ops` — # Open-Claw Operations & Configuration +- `pulse-turn-handling` — # Pulse Turn Handling - `pve-ops` — # PVE Operations - `radar-jobs-dashboard` — # RADAR Jobs Dashboard - `readlater-ops` — # ReadLater Operations