Nightly audit 2026-11-03 (window 2026-11-02): new incident — Job Radar Stage 1 iteration-cap recurrence (9h 01m, suppressed, no incident raised); Weekly Review incident resolved (reconciled 01:27 AWST); scheduled-tasks + browser-backend re-verified; index/log updated

This commit is contained in:
Tony0410
2026-11-03 00:22:13 +08:00
parent fda56e53fe
commit ca014efd63
8 changed files with 516 additions and 20 deletions
@@ -1,17 +1,22 @@
---
title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open
title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Recovered and Delivered 7 h 26 m Later
type: incident
status: active
status: resolved
created: 2026-11-02
updated: 2026-11-02
verified_on: 2026-11-02
review_after: 2026-11-09
updated: 2026-11-03
verified_on: 2026-11-03
review_after: 2026-11-17
confidence: high
tags: [cron, gateway, handoff, delivery, weekly-review, silent-failure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-jobs-json, live-logs, live-probe]
---
# Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open
# Weekly Review Lost at 18:00 on 2026-11-01 — Recovered and Delivered 7 h 26 m Later
> **Status: RESOLVED 2026-11-02T01:27:21+08:00.** The Sunday 18:00 run did *not* lose the review — it
> resumed, completed and delivered at **01:27:21**, and the incident row auto-closed. The body below is
> the record **as it stood at the 2026-11-02 audit**, before the recovery landed. See
> [Resolution](#resolution-2026-11-02) for the corrected outcome and the answers to the open questions.
## Headline
@@ -79,6 +84,8 @@ appearance of the "green/red row, no artefact" anti-pattern the wiki already doc
## Open questions
> ⚠ **Questions 1–3 are answered in [Resolution](#resolution-2026-11-02).** Question 4 remains open.
1. **What did "Gateway connection failed; no local fallback" mean?** Same message for two different
jobs 11 ms apart, both in the same handoff batch, with no gateway restart anywhere in the window.
2. **Why did `b63ed5` self-heal in 68 s and `79f9bb` never do so?** Both were in the same batch with the
@@ -88,6 +95,47 @@ appearance of the "green/red row, no artefact" anti-pattern the wiki already doc
4. Are the two unreconciled admission files (`141448f9…`, `69b5588f…`) the durable record needed to
decide whether either run had side effects? Reconcile before the next 18:00 Sunday slot.
## Resolution (2026-11-02)
**The run was not lost. It resumed, finished, and delivered — 7 h 26 m 43 s after it started.**
| Field | Value |
|---|---|
| Execution | `be6ec11a8e68428084af9ab18fb148db` — **the same execution record** the audit saw stuck at `unknown` |
| `scheduled_instant` | `2026-11-01T10:00:00+00:00` (18:00 AWST) |
| `started_at` | `2026-11-01T18:00:39.125776+08:00` |
| `finished_at` | **`2026-11-02T01:27:22.000881+08:00`** |
| `status` | **`completed`** |
| `error` | `NULL` |
| `delivery_outcome` | **`delivered`** |
| Delivery row | `2026-11-02T01:27:21.802246+08:00` — `delivered` |
| Incident row | `79f9bb_9c443b602eec` → `state: resolved`, `closed_at: 2026-11-02T01:27:21.606691+08:00` |
Anthony received the Weekly Review at 01:27 AWST on 2026-11-02. There is still exactly **one** execution
row for the 18:00:39 slot, so this is a reconciliation of the existing claim rather than a re-run — the
`unknown` terminal state at 18:05:13 was replaced when the run finally wrote its real outcome.
**Answers to the open questions:**
1. *(the "no local fallback" message)* — the handoff to the external worker did ultimately succeed; the
18:05:13 message recorded a **failed reconciliation attempt**, not a failed run. The run kept executing
for a further 7 h 26 m and produced its deliverable. The message is therefore a **false negative from
the reconciler**, not evidence that the work was abandoned.
2. **`b63ed5` closed in 68 s; `79f9bb` closed in 7 h 26 m — both eventually reached `completed`/`delivered`.**
The difference was run duration, not failure mode. The reconciler's window is shorter than the Weekly
Review's runtime, so any long-running job in a handoff batch will be reported `unknown` while it is
still working. **This is the general lesson: an `unknown` execution state on a long job is a timing
artefact, not a verdict.**
3. **No manual re-run was needed** — the run self-healed and delivered 12 minutes before the 2026-11-02
audit would have recommended one. Had the audit re-triggered the job, it would have duplicated the run.
### Consequences for the audit method
The 2026-11-02 audit read `status: unknown` + `closed_at: NULL` at 00:4x and concluded the deliverable was
lost. Both were true at the time of reading and both had inverted by 01:27. **A nightly audit that runs
~2 h after midnight will read every long-running job mid-flight and must not treat a null terminal state as
final until the job's expected runtime has elapsed** — see [[systems/scheduled-tasks]] §9.
## Related
- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]]
@@ -0,0 +1,118 @@
---
title: Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete Run Recorded `completed`, Output Suppressed
type: incident
status: active
created: 2026-11-03
updated: 2026-11-03
verified_on: 2026-11-03
review_after: 2026-11-10
confidence: high
tags: [cron, job-radar, iteration-cap, silent-failure, notion, camofox]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-state-db, live-jobs-json, session-history]
---
# Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete Run Recorded `completed`, Output Suppressed
## Headline
**Third occurrence of the Stage-1 budget class, and the first that the cron ledger reports as healthy.**
On **2026-11-02** the 07:00 `dca8482e4f76` (*Daily Marketing Job Radar — Stage 1 (Discovery)*) ran for
**9 h 01 m 19 s** (`started 07:00:33.378239` → `finished 16:01:52.496335 +08:00`), consumed the harness's
entire tool-calling iteration budget, and closed with:
| Field | Value |
|---|---|
| `status` | **`completed`** |
| `error` | `NULL` |
| `delivery_outcome` | **`suppressed`** |
No handoff artefact was written. **Nothing reached Notion, no cover letters were generated, and Stage 2 was
never triggered.** Because the run's own status is `completed` with no error, a health check that inspects
`executions.status` alone reports this day as green — the failure is invisible in the ledger.
## What the run did and did not do
The session (`Daily Marketing Job Radar - Discovery · Nov 02 16:01`, `local-368c8fe9…`) shows the discovery
half genuinely worked:
- Camofox was restarted at run start after its HTTP server wedged (see [[systems/browser-backend]]).
- **SEEK 51 unique roles, LinkedIn 60 unique job IDs, Indeed live, WA Government 35 results** — 197 raw
candidates, **164 new after three-level dedup** against the 479-row Notion board.
- A **URL-canonicalisation bug was found and fixed mid-run**: `au.seek.com` vs `au.seek.com.au` had been
dropping **SEEK and Indeed entries** from the consolidated list; the fix was a job-ID-based
canonicalisation.
- Adverts were individually verified fresh and active; salaries, closing dates and JD bodies were pulled.
Then the ceiling hit. The runtime injected *"You've reached the maximum number of tool-calling iterations
allowed. Please provide a final response…"* while the agent was still reading JD text for the borderline
finalists, and the run terminated in the **scoring** phase.
The agent's own closing report was explicit — and it was **not delivered**:
> *"**Status: INCOMPLETE — handoff file NOT written, Stage 2 NOT triggered.** … Because the mandatory
> deliverable (the handoff JSON) was not produced, today's roles have **not** been handed to Stage 2."*
That text is `delivery_outcome: suppressed`, so Anthony saw nothing on 11-02 — not the roles and not the
warning that the roles were missing.
## Downstream: both Stage 2 slots behaved correctly
| Slot | Session | Ran | Result |
|---|---|---|---|
| 11:30 | `local-92665337…` (*Write and Letters*) | 11:30 → 12:25 | Exited `[SILENT]` after verifying Stage 1 was alive but had no handoff |
| 14:30 (safety net) | `local-05a3f00b…` (*Write and Letters*) | 14:30 → 14:30 | Exited `[SILENT]` — 6 messages, 2 tool calls |
Both declines were correct: no handoff existed to consume. The safety-net design worked as intended — the
gap is that Stage 1 cannot finish inside its budget.
## Recurrence pattern
| Date | Incident page | Failure shape |
|---|---|---|
| 2026-10-27 | [[incidents/2026-10-27-job-radar-budget-exhausted]] | Tool-call budget exhausted before the Notion write stage |
| 2026-10-28 | [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] | Stage 1 hit the agent iteration cap before the handoff |
| **2026-11-02** | **this page** | Same cap; **run recorded `completed` and its report suppressed** |
Three hits in seven days, each ending in the same place: discovery complete, handoff absent, Notion write
stage never reached.
## The "green row, no artefact" anti-pattern
This is the **fourth appearance** of the class [[systems/scheduled-tasks]] §9 of the maintenance playbook
already documents (a job whose durable ledger entry says success while the thing it was supposed to produce
does not exist). The new variant here is that the ledger flipped from red to green:
| Date | `cron_incidents` row for `dca8482e4f76` | `executions.status` |
|---|---|---|
| 2026-10-27 | `dca848_e93a538ade6b` (`failure_type: auth`) | non-clean |
| 2026-10-28 | `dca848_0a2f03b5952b` (`failure_type: timeout`) | non-clean |
| **2026-11-02** | **none raised** | **`completed`, `error: NULL`** |
The two earlier caps each raised an incident whose `error` text is literally the run's INCOMPLETE report.
The 2026-11-02 cap raised **nothing at all** (`cron_incidents` holds no row for `dca8482e4f76` with a
`first_seen_at` on 2026-11-02; fleet-wide unclosed rows were **0** at the 2026-11-03 audit). The run's
`delivery_outcome` is `suppressed`, so no message reached Anthony either.
**Detection rule:** for any job whose deliverable is a *file* (a handoff JSON, a report, a generated page),
`executions.status` is not sufficient evidence. Verify the artefact exists at its expected path, or that
`cron_incidents` stayed quiet for a reason, before treating the slot as served. Silence from
`cron_incidents` is not the same as success — here it is the *symptom*.
## Open questions
1. **Is the iteration budget the right control, or is the task too large for one agent turn?** Three caps in
a row suggest Stage 1's work (four job boards → dedup → individual advert verification → JD reading →
scoring) does not fit a single turn at any budget.
2. **Should the harness record a cap-hit as something other than `completed`?** A distinct terminal status
(or a non-null `error`) would make this visible to every existing health check.
3. **Should the run's final response be delivered even when the job "succeeded"?** Suppressing an INCOMPLETE
report converts a loud failure into silence.
4. **Did the 11-02 roles get picked up on the next day's run**, or is 2026-11-02's candidate set lost?
The 164 deduped candidates were never persisted.
## Related
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]
- [[incidents/2026-10-27-job-radar-budget-exhausted]]
- [[systems/scheduled-tasks]]
- [[systems/browser-backend]]
+5 -4
View File
@@ -3,15 +3,15 @@ title: Anthony's Systems Wiki
type: collection
status: active
created: 2026-07-22
updated: 2026-11-02
verified_on: 2026-11-02
updated: 2026-11-03
verified_on: 2026-11-03
confidence: high
tags: [index, infrastructure, systems]
---
# Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-11-02 (nightly audit — the 11-01 fleet settled **167 / 1 / 2**, its first non-clean day in four: the Sunday **18:00 Weekly Review never ran and never delivered** (`79f9bb41ecf5`, *"Gateway connection failed; no local fallback"*, incident still open) and **three cron workers were `SIGTERM`ed (`-15`) mid-adoption** without writing a durable terminal state (`85edf6772bca` 07:30:46; `96eb95ce12f2` + `0e753e9720fd` 09:02:18) — for the first time with **no gateway restart anywhere in the window**, so the orphan class is now known to fire without a reload; a **fourth variant** is documented where **`ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning no execution row and no incident row**; delivery went **7 of 8**, the miss being `066bb5191280`'s `thread_id 34795` home-DM thread fallback (content delivered, run marked `failed`), while **Telegram flapped all day** (dual-stack `api.telegram.org` timeouts, disconnect 20:51:53, restarts 22:33:30 / 23:12:23) and the **relay adapter has now been absent ~29 h** (`Reconnecting relay` attempt 336); the longest-standing incident on the board — the 2026-10-01 Facebook-repin row — finally **closed at 10:44:12**)
> model routing and automation. Last updated: 2026-11-03 (nightly audit — the 11-02 fleet settled **177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`**, the **first fully clean ledger day in five**, and **both 11-01 `unknown` rows resolved in-window**: the Sunday 18:00 **Weekly Review reconciled at 01:27:22** after **7 h 26 m 43 s** and **delivered**, so *"the lost review" was never lost* — incident `79f9bb_9c443b602eec` auto-closed **01:27:21.606691** and its page is now **`resolved`** — while `ecc1fc13655a`'s 11-01 ghost `error` cleared on its 12:00:33 slot; **unclosed incident rows 1 → 0** (the first zero in the period), non-clean `last_status` **2 → 1**, deliveries **9 rows / 9 delivered**; ⚠ **but two ~9-hour runs failed invisibly.** The **Job Radar Stage 1** (`dca8482e4f76`, 07:00:33 → 16:01:52, **9 h 01 m 19 s**) hit the **agent iteration cap before writing its handoff**, so **both Stage 2 slots exited `[SILENT]` and nothing reached Notion** — and it **raised no `cron_incidents` row at all**, where the same class raised incidents on 10-27 and 10-28, because its terminal state flipped to `completed` with `error: NULL`; the **03:00 stocktake** (`1a8e90ea2271`) likewise ran **9 h 43 m 29 s**, **delivered no report** and left its output uncommitted in the working tree (committed by this audit).)
>
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
@@ -65,7 +65,8 @@ tags: [index, infrastructure, systems]
- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since *(superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → [[incidents/2026-10-29-conduit-voice-silence]])*. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]]
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]]
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery.
- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] — **The Sunday 18:00 Weekly Review did not run on 2026-11-01 and nothing was delivered:** its execution closed `unknown` at 18:05:13 with *"Cron execution unverified … Gateway connection failed; no local fallback"*, its incident row is **still `alerted`**, the job sits at `error` / `failure_streak: 1`, and `deliveries.db` shows **no row after 11:01:12** for the rest of the day. Its batch-mate `b63ed5817212` took the identical error and self-healed in 68 s. Same page records the **inverse silent failure**: `ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning **no execution row and no incident row at all**.
- [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] — **Third iteration-cap in seven days, and the first that raised no incident at all.** Job Radar Stage 1 (`dca8482e4f76`) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** on 2026-11-02, hit the **agent iteration cap before the handoff was written**, and closed **`status: completed` / `error: NULL` / `delivery_outcome: suppressed`**. Both Stage 2 slots declined correctly (`92665337…` 11:30 and the `05a3f00b…` 14:30 safety net both exited `[SILENT]`), so **nothing reached Notion, no cover letters were generated, and the day's 164 deduped candidates were never persisted** — a state a `status`-only health check reports as green. Contrast 10-27 (`dca848_e93a538ade6b`) and 10-28 (`dca848_0a2f03b5952b`), where the identical cap **did** open rows whose `error` text is the run's own INCOMPLETE report: on 11-02 the ledger **lost the red row it used to produce**. Discovery itself worked — SEEK **51**, LinkedIn **60 IDs**, Indeed live, WA Government **35** (197 raw / **164 new** after three-level dedup) — and a **URL-canonicalisation bug was found and fixed mid-run** (`au.seek.com` vs `au.seek.com.au` had been dropping SEEK *and* Indeed entries from the consolidated list). → [[systems/scheduled-tasks]]
- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] — **The Sunday 18:00 Weekly Review did not run on 2026-11-01 and nothing was delivered:** its execution closed `unknown` at 18:05:13 with *"Cron execution unverified … Gateway connection failed; no local fallback"*, its incident row is **still `alerted`**, the job sits at `error` / `failure_streak: 1`, and `deliveries.db` shows **no row after 11:01:12** for the rest of the day. Its batch-mate `b63ed5817212` took the identical error and self-healed in 68 s. Same page records the **inverse silent failure**: `ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning **no execution row and no incident row at all**. ✅ **RESOLVED 2026-11-02T01:27:21 — the review was never lost.** The *same* execution row (`be6ec11a8e68428084af9ab18fb148db`) closed **`completed` / `delivered` at 01:27:22** after **7 h 26 m 43 s**, the delivery lands at **01:27:21.802246** and incident `79f9bb_9c443b602eec` auto-closed at **01:27:21.606691** — so the 18:05:13 `unknown` was a **reconciler timing artefact on a still-running job**, not a verdict (page now `resolved`). Batch-mate `b63ed5` closed in 68 s and `79f9bb` in 7 h 26 m, both `completed`; **an `unknown` row on a long job is a timing artefact, not a failure**. `ecc1fc13655a`'s ghost `error` cleared too (12:00:33 → `completed` / `delivered` 12:29:26), taking **non-clean `last_status` 2 → 1**.
- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] — **Three restart-safe cron workers killed by `SIGTERM` (`-15`) after adopting their claim**, so the scheduler recorded *"whether side effects ran is unknown"*: `85edf6772bca` @07:30:46, `96eb95ce12f2` + `0e753e9720fd` @09:02:18. **A fourth variant of the orphan class** — and the first with **no gateway restart, reload or drain anywhere in the window** (PID `175269` up since 10-31 19:24:40; relay counter monotonic through both signals), so it cannot be attributed to reload behaviour. Likely trigger: a **30 s worker-ack window** smaller than this host's cold start (**83 plugins / 69 enabled** discovered in 70 s). All three rows now `resolved`; one execution left `unknown`.
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]]
+53
View File
@@ -1854,3 +1854,56 @@ recorded wrong.**
(`origin/master..HEAD` empty) before this audit; touched pages were mirrored into the vault's
existing conventions and hash-verified. **No secrets, passwords, tokens or keys written or quoted** —
every job id, PID, thread id and timestamp above is a non-sensitive operational identifier.
## [2026-11-03] audit | Nightly audit — 2026-11-02 window (first clean ledger day in five, and two ~9-hour failures it cannot see)
- **Scope:** `state.db` sessions spanning 2026-11-02 00:00 → 2026-11-03 00:2x AWST; `cron/executions.db`,
`cron/deliveries.db`, `cron_incidents`, `cron/jobs.json` (all read-only). Authoring tree
`/home/hermes/workspace/agent-estate-wiki` was **clean and in sync with Gitea** (`origin/master..HEAD`
empty) before this audit; `systems/skills-index.md` and `raw/stocktake/freshness-20261102.txt` were
**committed on the stocktake's behalf** (see below).
- **NEW — `incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md`:** the **third iteration-cap
in seven days** (10-27 tool-call budget, 10-28 cap, 11-02 cap) on `dca8482e4f76`
(**07:00:33 → 16:01:52, 9 h 01 m 19 s**), and **the first that raised no `cron_incidents` row at all** —
it closed `completed` / `error: NULL` / `delivery_outcome: suppressed`. Both Stage 2 slots exited
`[SILENT]`, so **nothing reached Notion, no cover letters were written and the 164 deduped candidates
were never persisted**. Page carries the **"green row, no artefact" detection table** contrasting the
three occurrences, and records the mid-run **URL-canonicalisation fix** (`au.seek.com` vs
`au.seek.com.au`) that had been silently dropping SEEK and Indeed entries from dedup.
- **UPDATED — `incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md`:** status
**`active` → `resolved`**. Live evidence: the **same execution row**
(`be6ec11a8e68428084af9ab18fb148db`) closed `completed` / `delivered` at **01:27:22** after
**7 h 26 m 43 s**, delivery at 01:27:21.802246, incident `79f9bb_9c443b602eec` auto-closed at
**01:27:21.606691**. The 18:05:13 `unknown` was a **reconciler timing artefact on a still-running job**,
not a verdict. Resolution section added; open questions 1–3 answered, **question 4 retained** (the two
unreconciled admission files `141448f9…json` / `69b5588f…json`).
- **UPDATED — `systems/scheduled-tasks.md`:** `## Count` advanced to the 2026-11-02 window —
**177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows) vs 167/1/2 on 11-01, 160/4 on
10-31, 138/0 on 10-30; fleet unchanged at **29 jobs / 26 enabled / 3 disabled**; **unclosed incident
rows 1 → 0** (the first zero in the period); both 11-01 `unknown` rows resolved and `ecc1fc13655a`'s
ghost `error` cleared; deliveries **9 rows / 9 `delivered`**. New **`## Re-verified 2026-11-03`**
section documents the two invisible failures, the stocktake artefacts committed, the
`substr(started_at,1,10)` grouping rule and the **"do not read `running`/`unknown` as a verdict before
the expected runtime has elapsed"** correction. Frontmatter `updated`/`verified_on` → 2026-11-03,
`review_after` → 2026-11-10.
- **UPDATED — `systems/browser-backend.md`:** `updated`/`verified_on` → 2026-11-03 plus a `## History`
entry for the **2026-11-02 REST-server wedge** (requests timing out with nothing reaching the service
log, cleared by `systemctl --user restart camofox-browser.service`; the four-board Job Radar run then
proceeded). **Live re-probe 2026-11-03 00:16:14 AWST:** unit `active`, `MainPID 953299`,
`ActiveEnterTimestamp Mon 2026-11-02 18:11:04`, `MemoryCurrent 37.5 MB`, `/health` → **200**
`{"ok":true,"engine":"camoufox","browserConnected":false,…}`, `/tabs` → `{"running":true,"tabs":[]}`.
Page now states explicitly that this wedge is a **different fault from the 2026-10-12 memory ceiling**
(hang at 31 MB RSS vs exhaustion at 2.5 G), so the containment limits are not the applicable control.
- **UPDATED — `index.md`:** `updated`/`verified_on` → 2026-11-03, the audit summary line rewritten for
this window, the new 11-02 incident link added, and the 11-01 Weekly Review entry annotated
**✅ RESOLVED 2026-11-02T01:27:21**.
- **COMMITTED on the stocktake's behalf:** `1a8e90ea2271` finished with `systems/skills-index.md`
modified and `raw/stocktake/freshness-20261102.txt` untracked. Both are valid artefacts — freshness
**4 stale / 113 fresh** at a >21-day threshold, skills-index regenerated with **Devops 108 → 109**
(`pulse-turn-handling`) — so this audit committed them rather than let the next slot overwrite them.
- **Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned; no
decision reversed. A full vault mirror pass was deliberately skipped again (drift counts instead of a
bulk copy).
- **DRIFT (context, not findings):** touched pages follow the vault's existing conventions and
SCHEMA.md frontmatter rules. **No secrets, passwords, tokens or keys written or quoted** — every job
id, PID, thread id and timestamp above is a non-sensitive operational identifier.
+133
View File
@@ -0,0 +1,133 @@
# Wiki freshness audit — 2026-11-02
Threshold: >21 days since `updated`
## STALE (4)
- [ 22d] agents/qwenpaw.md (updated 2026-10-11)
- [ 22d] incidents/2026-09-28-hermes-managed-runtime-deps.md (updated 2026-10-11)
- [ 22d] systems/ignis.md (updated 2026-10-11)
- [ 22d] systems/paseo.md (updated 2026-10-11)
## FRESH (113)
- [ 13d] CONTRIBUTING.md
- [ 7d] agents/hermes-chaos.md
- [ 7d] agents/hermes-production.md
- [ 13d] agents/index.md
- [ 4d] agents/nanobot.md
- [ 13d] agents/open-claw.md
- [ 13d] agents/zeroclaw.md
- [ 4d] comparisons/camoufox-vs-chrome.md
- [ 13d] comparisons/hermes-vs-nanobot.md
- [ 13d] comparisons/memory-backends.md
- [ 13d] comparisons/model-routers.md
- [ 4d] comparisons/steel-vs-camoufox.md
- [ 13d] concepts/agent-memory.md
- [ 4d] concepts/browser-automation.md
- [ 13d] concepts/model-routing.md
- [ 13d] concepts/persistent-shells.md
- [ 13d] concepts/tool-calling.md
- [ 2d] current-state.md
- [ 4d] decisions/index.md
- [ 13d] decisions/keep-openrouter-as-fallback.md
- [ 13d] decisions/persistent-shell-enabled.md
- [ 4d] decisions/rollback-steel-to-camofox.md
- [ 13d] experiments/index.md
- [ 9d] incidents/2026-07-29-mnemosyne-relay.md
- [ 13d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 9d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 11d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 10d] incidents/2026-08-15-skills-index-drift.md
- [ 21d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 8d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 13d] incidents/2026-08-29-gmail-read-state.md
- [ 12d] incidents/2026-09-02-sftpgo-outage.md
- [ 4d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 7d] incidents/2026-10-03-ct450-hermes-update.md
- [ 7d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 4d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 9d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 1d] incidents/2026-10-09-memory-recall-relevance.md
- [ 17d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 6d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 14d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 6d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 11d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 8d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 7d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 7d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 7d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 5d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 3d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 3d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 3d] incidents/2026-10-29-conduit-voice-silence.md
- [ 2d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 3d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 2d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 1d] incidents/2026-10-31-asana-access-failures.md
- [ 1d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 1d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 0d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 0d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 0d] index.md
- [ 11d] infrastructure/backups.md
- [ 13d] infrastructure/credentials-map.md
- [ 1d] infrastructure/docker-services.md
- [ 13d] infrastructure/domains-and-tunnels.md
- [ 2d] infrastructure/hosts.md
- [ 19d] infrastructure/network-map.md
- [ 1d] infrastructure/proxmox.md
- [ 1d] infrastructure/tailscale.md
- [ 21d] queries/retained-research-answers.md
- [ 8d] reports/communities-wa-social-media-audit-2026.md
- [ 1d] reports/uber-eats/2026-07.md
- [ 1d] reports/uber-eats/2026-08.md
- [ 1d] reports/uber-eats/2026-09.md
- [ 1d] reports/uber-eats/2026-10.md
- [ 1d] reports/uber-eats/index.md
- [ 13d] runbooks/backup-wiki.md
- [ 13d] runbooks/diagnose-docker.md
- [ 13d] runbooks/gateway-resource-resilience.md
- [ 3d] runbooks/job-radar.md
- [ 13d] runbooks/multiplexer-setup.md
- [ 7d] runbooks/provider-health.md
- [ 4d] runbooks/recover-docker-service.md
- [ 13d] runbooks/restart-browser.md
- [ 13d] runbooks/restart-hermes.md
- [ 13d] runbooks/restore-openrouter.md
- [ 13d] runbooks/test-openrouter.md
- [ 13d] runbooks/troubleshoot-tool-loop.md
- [ 7d] runbooks/update-hermes-safely.md
- [ 4d] systems/browser-backend.md
- [ 13d] systems/cloudflare-tunnel.md
- [ 1d] systems/conduit.md
- [ 7d] systems/current-profile.md
- [ 13d] systems/guanaco.md
- [ 13d] systems/headless-server.md
- [ 3d] systems/hermes-agent.md
- [ 13d] systems/hermes-ui.md
- [ 21d] systems/home-assistant.md
- [ 3d] systems/homepage-dashboard.md
- [ 7d] systems/memory-backend.md
- [ 0d] systems/messaging-integrations.md
- [ 1d] systems/mnemosyne.md
- [ 7d] systems/model-providers.md
- [ 21d] systems/nanobot.md
- [ 13d] systems/nextcloud.md
- [ 13d] systems/obsidian.md
- [ 1d] systems/omniroute.md
- [ 13d] systems/opencode-go.md
- [ 13d] systems/openrouter.md
- [ 14d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 13d] systems/terminal-backend.md
- [ 13d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 4 stale, 113 fresh, 3 no-date of 120 pages
+15 -2
View File
@@ -3,8 +3,8 @@ title: Browser Backend
type: system
status: active
created: 2026-07-22
updated: 2026-10-29
verified_on: 2026-10-29
updated: 2026-11-03
verified_on: 2026-11-03
confidence: high
tags: [browser, camofox, automation]
sources: [live-probe, config, session-history]
@@ -112,6 +112,19 @@ until the next session starts.
on **CT201** (containers, images, network, project dir, Serve routes) and **CT333** (container + image)
torn down. Camofox is the sole backend.
- **2026-11-02 — the HTTP server wedged at the start of a browser-heavy cron run; a restart cleared it.**
The 07:00 Job Radar Stage 1 (`dca8482e4f76`) found the REST server accepting nothing — requests timing out
with **no lines reaching the service log** — before its first tab was opened. `systemctl --user restart
camofox-browser.service` cleared it and the run went on to work four boards. **Live re-probe
`2026-11-03 00:16:14 AWST`:** unit `active`, `MainPID 953299`, `ActiveEnterTimestamp Mon 2026-11-02
18:11:04`, `MemoryCurrent 37.5 MB`; `/health` → **200**
`{"ok":true,"engine":"camoufox","browserConnected":false,"browserRunning":false,"activeTabs":0,"activeSessions":0,"consecutiveFailures":0}`;
`/tabs` → `{"running":true,"tabs":[]}`. Idle-clean and far inside the containment drop-in.
⚠ **This is a different fault from the 2026-10-12 ceiling:** that one was memory exhaustion under many
open tabs (`2.5G`, swap fully consumed); this one was a **hung HTTP server at 31 MB RSS**, so the
containment limits are not the control that applies. A health probe inside browser-heavy scripts should
therefore test **responsiveness**, not just `/health` status code.
## Correction (2026-10-29) — Steel is out of the stack, not just disabled
Verified against live state at **2026-10-29**. The earlier 00:3x probe said only "not reachable"; the truth is
+133 -4
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-11-02
verified_on: 2026-11-02
review_after: 2026-11-09
updated: 2026-11-03
verified_on: 2026-11-03
review_after: 2026-11-10
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -14,7 +14,33 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks
## Count
- **Current as of 2026-11-02 00:4x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
- **Current as of 2026-11-03 00:1x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
`deliveries.db`) — the first fully clean ledger day in five, but two long runs ended in a silent cap and
one of them raised no incident at all.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged).
**Window 2026-11-02 settled 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows;
**96** of the completions are `96eb95ce12f2`'s 15-minute watchdog slots) against **167 / 1 / 2** on
11-01, **160 / 4** on 10-31 and **138 / 0** on 10-30. **Both 11-01 `unknown` rows resolved on 11-02:**
the Sunday 18:00 Weekly Review reconciled at **01:27:22** (`be6ec11a8e68428084af9ab18fb148db`,
`completed` / `delivered`, **7 h 26 m 43 s** after it started) and its incident
`79f9bb_9c443b602eec` auto-closed at **01:27:21.606691** →
[[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] (**now `resolved`; the "lost
review" was never lost**). `ecc1fc13655a`'s 11-01 ghost `error` also cleared — its 11-02 12:00:33 slot
ran `completed` / `delivered` at 12:29:26. **Unclosed incident rows 1 → 0 — the first zero in the
period.** **Non-clean `last_status` 2 → 1:** only `066bb5191280` (Uber Eats Monthly Spend Report,
`delivery_failed`, `failure_streak: 0`, `last_run 2026-11-01T07:18:57`) — the 11-01 DM-thread miss, which
**did not recur**; it has **no `executions` row on 11-02 at all**. **Delivery health: 9 rows on 11-02,
9/9 `delivered`** (01:27:21, 07:30:35, 07:42:21, 08:10:18, 08:41:14, 09:07:20, 10:23:15, 12:29:25,
18:45:37). ⚠ **Two long runs finished `completed` with `error: NULL` and `delivery_outcome:
suppressed` after ~9 h — and the first raised no incident:** (a) **`dca8482e4f76`** (Job Radar Stage 1)
**07:00:33 → 16:01:52 = 9 h 01 m 19 s**, hit the agent iteration cap before the handoff, so **both Stage 2
slots (`92665337…` 11:30, `05a3f00b…` 14:30 safety net) exited `[SILENT]` and nothing reached Notion** —
the same class that raised incidents on 10-27 and 10-28 **raised none this time** because the terminal
state flipped to `completed` → [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]];
(b) **`1a8e90ea2271`** (daily stocktake) **03:00:31 → 12:44:00 = 9 h 43 m 29 s**, `suppressed` — it
**delivered no report** (unlike 11-01's 03:00:37 delivery) and **left its output uncommitted**
(`systems/skills-index.md` modified, `raw/stocktake/freshness-20261102.txt` untracked; the 2026-11-03
audit committed both on its behalf). See **Known Issues → Open** below.
- **Prior window (2026-11-02 00:4x AWST, live `jobs.json` + `executions.db` + `cron_incidents` +
`deliveries.db` + gateway log) — the fleet ran clean all day, then failed in three different ways
and missed a delivery.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged). **Window 2026-11-01
settled 167 `completed` / 1 `running` / 2 `unknown`** (+ this audit `running`, started 23:59:28)
@@ -190,6 +216,23 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
## Known Issues
### Resolved 2026-11-02
- **The "lost" Weekly Review was never lost — it delivered at 01:27:21, 7 h 26 m 43 s after its 18:00 slot.**
Execution `be6ec11a8e68428084af9ab18fb148db` (job `79f9bb41ecf5`, `scheduled_instant`
`2026-11-01T10:00:00+00:00`) started **2026-11-01T18:00:39.125776+08:00** and finally closed
**2026-11-02T01:27:22.000881+08:00** with `status: completed`, `error: NULL`,
`delivery_outcome: delivered`; the delivery row is timestamped **01:27:21.802246** and incident
`79f9bb_9c443b602eec` auto-closed at **01:27:21.606691**. **Same execution row, not a re-run** — the
`unknown` terminal state recorded at 18:05:13 was replaced when the run wrote its real outcome.
→ [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] is now **`resolved`**.
**Method lesson for this audit:** the 2026-11-02 00:4x reading of `status: unknown` + `closed_at: NULL`
was accurate *at the time of reading* and had inverted 43 minutes later. **A nightly audit that runs
~2 h after midnight must not treat a null terminal state on a long-running job as final** — the
reconciler's window is shorter than these runs, so `unknown` on a long job is a timing artefact, not a
verdict. The same false-negative logic applies to `ecc1fc13655a`, whose 11-01 ghost `error` cleared when
its 11-02 12:00:33 slot closed `completed` / `delivered` at 12:29:26.
### Resolved 2026-10-30
- **The Mnemosyne Health Monitor was never really down — and it is enabled again (`ea79a2107e36`).** Its 00:01:08
@@ -236,6 +279,23 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
- **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`.
### Open
- **Two ~9-hour runs ended in a silent iteration cap on 2026-11-02, and the more damaging one raised no incident (NEW 2026-11-03):**
- **`dca8482e4f76`** (Job Radar Stage 1) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** and closed
`status: completed` / `error: NULL` / `delivery_outcome: suppressed` **after hitting the agent
iteration cap before the handoff was written**. Both Stage 2 slots declined (`92665337…` 11:30 and the
`05a3f00b…` 14:30 safety net both exited `[SILENT]`), so **nothing reached Notion**. This is the same
class that raised `dca848_e93a538ade6b` (10-27) and `dca848_0a2f03b5952b` (10-28) — whose `error`
columns are literally the runs' INCOMPLETE reports — yet **2026-11-02 produced no `cron_incidents` row
at all**, because the terminal state flipped from non-clean to `completed`. Three caps in seven days
(10-27, 10-28, 11-02) is a pattern, not bad luck.
→ [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]. **Detection rule: for a job whose
deliverable is a file, `executions.status` is not evidence — check the artefact.**
- **`1a8e90ea2271`** (wiki stocktake) ran **03:00:31 → 12:44:00 = 9 h 43 m 29 s**, `completed` /
`suppressed`, and **delivered no report** (11-01 delivered at 03:00:37; 11-02 has no delivery row near
03:00). It left its output **uncommitted** — `systems/skills-index.md` modified and
`raw/stocktake/freshness-20261102.txt` untracked — which the 2026-11-03 audit committed on its behalf.
The artefacts themselves are sound (freshness report: **4 stale / 113 fresh** at a >21-day threshold;
skills-index regenerated).
- **GitHub Trending Radar's delivery token is revoked — needs a new @BotFather token (NEW 2026-10-13):** live `getMe` on the token in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401**, so `digest.py`'s Telegram send fails every Monday while the cron run itself stays green (`last_status: ok`). A *completed* run therefore does not mean the digest was sent by the script — the 2026-10-12 run only reached the DM because the agent recovered the content with `--dry-run` and its own response was delivered. Fix: issue a fresh token from @BotFather, replace it in `config.json`, and re-run `getMe` to confirm; do not re-read a green cron status as proof of delivery. **This corrects the 2026-10-05 "claim refuted" entry, which was wrong.** See [[systems/scheduled-tasks]] job `c0a9fa314542`.
- **Camofox memory pressure during long browser runs (NEW 2026-10-13):** the 2026-10-12 07:01 Daily Marketing Job Radar run drove Camofox into the ceiling of its own systemd containment drop-in (`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`: `MemoryHigh=2G`, `MemoryMax=3G`, swap max 1G). Live probe mid-run read `Memory: 2.5G (high: 2G, max: 3G, swap: 1023.9M, available: 0B, peak: 2.5G)` with 383 tasks — swap fully consumed and no reclaimable headroom, which is why the `/tabs` endpoint and browser evals started hanging. The run still delivered: the agent stopped piling on memory, restarted the service, and finished one tab at a time with immediate close (post-restart: 2.1 GB used of 14 GB on CT460). Any job that opens many tabs in one pass is exposed to the same wall. Keep multi-tab extraction bounded, close each tab before opening the next, and consider raising the containment limits only if the host headroom justifies it. See [[systems/browser-backend]].
- **Stocktake's Quartz rebuild step is dead code (NEW 2026-10-07):** `wiki-maintain.sh` documents a third step — rebuild the Quartz site — but the 2026-10-07 03:00 run logged it as skipped: no `quartz.config.ts` is present and `npx` is unavailable, so `public/` has not been refreshed. The skills-index and freshness artifacts are regenerated and committed correctly; only the published-HTML step is inert. Either restore the Quartz toolchain or drop the step from the script and the job description.
@@ -754,3 +814,72 @@ GitHub Trending Radar token; Camofox containment under multi-tab load; the Condu
retention; **plus new:** the unreconciled admission files
`~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json` (both 11-01) and the unidentified `-15`
signal source.
## Re-verified 2026-11-03 (audit window 2026-11-02 00:00 → 2026-11-03 00:2x AWST; live probes 00:0x–00:2x)
**The first fully clean ledger day in five — and it contains two failures the ledger does not show.**
**Settled: 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows) against 167 / 1 / 2 on
11-01, 160 / 4 on 10-31 and 138 / 0 on 10-30. Fleet unchanged at **29 jobs, 26 enabled, 3 disabled**.
Nothing was added, removed, disabled or repinned in the window. **96** of the completions are
`96eb95ce12f2`'s 15-minute watchdog slots.
**Both 11-01 `unknown` rows resolved in-window** (see §Resolved 2026-11-02). **Unclosed incident rows
1 → 0** — the first zero in the period. **Non-clean `last_status` 2 → 1.** Deliveries **9 rows, 9/9
`delivered`.**
### Both faults of the day are invisible in the ledger
| # | Fault | Job | What the ledger says | Page |
|---|---|---|---|---|
| 1 | **Iteration cap reached before the handoff** — 9 h 01 m 19 s, Stage 2 never triggered, **nothing reached Notion**, and **no `cron_incidents` row was raised** | `dca8482e4f76` 07:00:33 → 16:01:52 | `completed` / `error: NULL` / `delivery_outcome: suppressed` | [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] |
| 2 | **9 h 43 m run, no report delivered, output left uncommitted** | `1a8e90ea2271` 03:00:31 → 12:44:00 | `completed` / `error: NULL` / `delivery_outcome: suppressed` | this page → Known Issues → Open |
Fault 1 is the **third cap in seven days** (10-27 tool-call budget, 10-28 iteration cap, 11-02 iteration
cap) and the **first that raised no incident at all** — the two earlier rows carry their runs' INCOMPLETE
reports verbatim in `cron_incidents.error`. That is the sharpest form of the §9 anti-pattern yet: the
ledger did not merely stay green over a dead job, it **lost the red row it used to produce**. Silence from
`cron_incidents` is not success.
### Delivery health: 9 rows, 9 delivered
`deliveries.db` for 11-02: **01:27:21 ✔** (the Weekly Review recovery), 07:30:35 ✔, 07:42:21 ✔,
08:10:18 ✔, 08:41:14 ✔, 09:07:20 ✔, 10:23:15 ✔, 12:29:25 ✔, 18:45:37 ✔. **The 11-01 `066bb5191280`
thread-miss did not recur.** That job has **no `executions` row on 11-02 at all** (`0 2 1 * *`, next slot
2026-12-01), so its 11-01 `delivery_failed` flag will sit on the fleet for the rest of November — the
fleet's **only** non-clean `last_status`.
### Stocktake artefacts committed on the stocktake's behalf
`1a8e90ea2271` finished with `systems/skills-index.md` modified and
`raw/stocktake/freshness-20261102.txt` untracked in the working tree. Both are valid artefacts —
freshness **4 stale / 113 fresh** at a >21-day threshold; skills-index regenerated with **Devops 108 → 109**
(`pulse-turn-handling` added) — so this audit **committed both** rather than leave them to be overwritten
by the next slot.
### Method note (correction carried forward)
Two rules now bind this audit's tallies: group by `substr(started_at,1,10)`, never `date(...)` (which
normalises to UTC and shifts 00:00–07:59 AWST rows onto the previous day); and **do not read a
`running`/`unknown` row as a verdict until the job's expected runtime has elapsed** — the Weekly Review
was still working at the 00:4x reading and delivered at 01:27, 43 minutes later. An audit that runs two
hours after midnight will read every long job mid-flight.
### Pages touched
`incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` (new),
`incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md` (status → **`resolved`**,
Resolution section + answers to its open questions 1–3 added), `systems/scheduled-tasks.md`,
`systems/browser-backend.md`, `systems/skills-index.md` (stocktake output, committed),
`raw/stocktake/freshness-20261102.txt` (stocktake output, committed), `index.md`, `log.md`.
**Nothing removed or archived:** no job retired, no system obsolete, no decision reversed in the window.
### Still open (carried + new)
The single-node no-failover topology behind every model-calling job; the GitHub Trending Radar delivery
token; Camofox containment under multi-tab load; the Conduit health store with no retention; the
unreconciled admission files `~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json`; the
unidentified `-15` signal source; and the **30 s worker-ack window vs this host's cold-start time**.
**Plus new:** whether 2026-11-02's **164 deduped Job Radar candidates are recoverable**, whether a
cap-hit should keep being recorded as `completed`, and why the 03:00 stocktake now takes ~9 h 45 m.
+5 -4
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system
status: active
created: 2026-07-22
updated: 2026-11-01
verified_on: 2026-11-01
updated: 2026-11-02
verified_on: 2026-11-02
confidence: high
tags: [skills, hermes]
sources: []
@@ -12,7 +12,7 @@ sources: []
# Installed Skills
**Total installed: 301 skills** in `~/.hermes/skills/`.
**Total installed: 302 skills** in `~/.hermes/skills/`.
- **46 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
- The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory.
@@ -102,7 +102,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `ddia-systems` — # Designing Data-Intensive Applications Framework
## Devops (108)
## Devops (109)
- `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS)
- `agent-extension-audit` — # Agent Extension Audit
@@ -191,6 +191,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `omniroute-skill-caller` — # Call an OmniRoute Skill by Name (auto-resolving wrapper)
- `omniroute-skill-execution` — # Execute Skills on OmniRoute MCP
- `openclaw-ops` — # Open-Claw Operations & Configuration
- `pulse-turn-handling` — # Pulse Turn Handling
- `pve-ops` — # PVE Operations
- `radar-jobs-dashboard` — # RADAR Jobs Dashboard
- `readlater-ops` — # ReadLater Operations