Nightly audit 2026-11-03 (window 2026-11-02): new incident — Job Radar Stage 1 iteration-cap recurrence (9h 01m, suppressed, no incident raised); Weekly Review incident resolved (reconciled 01:27 AWST); scheduled-tasks + browser-backend re-verified; index/log updated
This commit is contained in:
@@ -1,17 +1,22 @@
|
||||
---
|
||||
title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open
|
||||
title: Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Recovered and Delivered 7 h 26 m Later
|
||||
type: incident
|
||||
status: active
|
||||
status: resolved
|
||||
created: 2026-11-02
|
||||
updated: 2026-11-02
|
||||
verified_on: 2026-11-02
|
||||
review_after: 2026-11-09
|
||||
updated: 2026-11-03
|
||||
verified_on: 2026-11-03
|
||||
review_after: 2026-11-17
|
||||
confidence: high
|
||||
tags: [cron, gateway, handoff, delivery, weekly-review, silent-failure]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-jobs-json, live-logs, live-probe]
|
||||
---
|
||||
|
||||
# Weekly Review Lost at 18:00 on 2026-11-01 — Gateway Handoff Failed, Incident Still Open
|
||||
# Weekly Review Lost at 18:00 on 2026-11-01 — Recovered and Delivered 7 h 26 m Later
|
||||
|
||||
> **Status: RESOLVED 2026-11-02T01:27:21+08:00.** The Sunday 18:00 run did *not* lose the review — it
|
||||
> resumed, completed and delivered at **01:27:21**, and the incident row auto-closed. The body below is
|
||||
> the record **as it stood at the 2026-11-02 audit**, before the recovery landed. See
|
||||
> [Resolution](#resolution-2026-11-02) for the corrected outcome and the answers to the open questions.
|
||||
|
||||
## Headline
|
||||
|
||||
@@ -79,6 +84,8 @@ appearance of the "green/red row, no artefact" anti-pattern the wiki already doc
|
||||
|
||||
## Open questions
|
||||
|
||||
> ⚠ **Questions 1–3 are answered in [Resolution](#resolution-2026-11-02).** Question 4 remains open.
|
||||
|
||||
1. **What did "Gateway connection failed; no local fallback" mean?** Same message for two different
|
||||
jobs 11 ms apart, both in the same handoff batch, with no gateway restart anywhere in the window.
|
||||
2. **Why did `b63ed5` self-heal in 68 s and `79f9bb` never do so?** Both were in the same batch with the
|
||||
@@ -88,6 +95,47 @@ appearance of the "green/red row, no artefact" anti-pattern the wiki already doc
|
||||
4. Are the two unreconciled admission files (`141448f9…`, `69b5588f…`) the durable record needed to
|
||||
decide whether either run had side effects? Reconcile before the next 18:00 Sunday slot.
|
||||
|
||||
## Resolution (2026-11-02)
|
||||
|
||||
**The run was not lost. It resumed, finished, and delivered — 7 h 26 m 43 s after it started.**
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Execution | `be6ec11a8e68428084af9ab18fb148db` — **the same execution record** the audit saw stuck at `unknown` |
|
||||
| `scheduled_instant` | `2026-11-01T10:00:00+00:00` (18:00 AWST) |
|
||||
| `started_at` | `2026-11-01T18:00:39.125776+08:00` |
|
||||
| `finished_at` | **`2026-11-02T01:27:22.000881+08:00`** |
|
||||
| `status` | **`completed`** |
|
||||
| `error` | `NULL` |
|
||||
| `delivery_outcome` | **`delivered`** |
|
||||
| Delivery row | `2026-11-02T01:27:21.802246+08:00` — `delivered` |
|
||||
| Incident row | `79f9bb_9c443b602eec` → `state: resolved`, `closed_at: 2026-11-02T01:27:21.606691+08:00` |
|
||||
|
||||
Anthony received the Weekly Review at 01:27 AWST on 2026-11-02. There is still exactly **one** execution
|
||||
row for the 18:00:39 slot, so this is a reconciliation of the existing claim rather than a re-run — the
|
||||
`unknown` terminal state at 18:05:13 was replaced when the run finally wrote its real outcome.
|
||||
|
||||
**Answers to the open questions:**
|
||||
|
||||
1. *(the "no local fallback" message)* — the handoff to the external worker did ultimately succeed; the
|
||||
18:05:13 message recorded a **failed reconciliation attempt**, not a failed run. The run kept executing
|
||||
for a further 7 h 26 m and produced its deliverable. The message is therefore a **false negative from
|
||||
the reconciler**, not evidence that the work was abandoned.
|
||||
2. **`b63ed5` closed in 68 s; `79f9bb` closed in 7 h 26 m — both eventually reached `completed`/`delivered`.**
|
||||
The difference was run duration, not failure mode. The reconciler's window is shorter than the Weekly
|
||||
Review's runtime, so any long-running job in a handoff batch will be reported `unknown` while it is
|
||||
still working. **This is the general lesson: an `unknown` execution state on a long job is a timing
|
||||
artefact, not a verdict.**
|
||||
3. **No manual re-run was needed** — the run self-healed and delivered 12 minutes before the 2026-11-02
|
||||
audit would have recommended one. Had the audit re-triggered the job, it would have duplicated the run.
|
||||
|
||||
### Consequences for the audit method
|
||||
|
||||
The 2026-11-02 audit read `status: unknown` + `closed_at: NULL` at 00:4x and concluded the deliverable was
|
||||
lost. Both were true at the time of reading and both had inverted by 01:27. **A nightly audit that runs
|
||||
~2 h after midnight will read every long-running job mid-flight and must not treat a null terminal state as
|
||||
final until the job's expected runtime has elapsed** — see [[systems/scheduled-tasks]] §9.
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]]
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
---
|
||||
title: Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete Run Recorded `completed`, Output Suppressed
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-03
|
||||
updated: 2026-11-03
|
||||
verified_on: 2026-11-03
|
||||
review_after: 2026-11-10
|
||||
confidence: high
|
||||
tags: [cron, job-radar, iteration-cap, silent-failure, notion, camofox]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-state-db, live-jobs-json, session-history]
|
||||
---
|
||||
|
||||
# Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete Run Recorded `completed`, Output Suppressed
|
||||
|
||||
## Headline
|
||||
|
||||
**Third occurrence of the Stage-1 budget class, and the first that the cron ledger reports as healthy.**
|
||||
On **2026-11-02** the 07:00 `dca8482e4f76` (*Daily Marketing Job Radar — Stage 1 (Discovery)*) ran for
|
||||
**9 h 01 m 19 s** (`started 07:00:33.378239` → `finished 16:01:52.496335 +08:00`), consumed the harness's
|
||||
entire tool-calling iteration budget, and closed with:
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| `status` | **`completed`** |
|
||||
| `error` | `NULL` |
|
||||
| `delivery_outcome` | **`suppressed`** |
|
||||
|
||||
No handoff artefact was written. **Nothing reached Notion, no cover letters were generated, and Stage 2 was
|
||||
never triggered.** Because the run's own status is `completed` with no error, a health check that inspects
|
||||
`executions.status` alone reports this day as green — the failure is invisible in the ledger.
|
||||
|
||||
## What the run did and did not do
|
||||
|
||||
The session (`Daily Marketing Job Radar - Discovery · Nov 02 16:01`, `local-368c8fe9…`) shows the discovery
|
||||
half genuinely worked:
|
||||
|
||||
- Camofox was restarted at run start after its HTTP server wedged (see [[systems/browser-backend]]).
|
||||
- **SEEK 51 unique roles, LinkedIn 60 unique job IDs, Indeed live, WA Government 35 results** — 197 raw
|
||||
candidates, **164 new after three-level dedup** against the 479-row Notion board.
|
||||
- A **URL-canonicalisation bug was found and fixed mid-run**: `au.seek.com` vs `au.seek.com.au` had been
|
||||
dropping **SEEK and Indeed entries** from the consolidated list; the fix was a job-ID-based
|
||||
canonicalisation.
|
||||
- Adverts were individually verified fresh and active; salaries, closing dates and JD bodies were pulled.
|
||||
|
||||
Then the ceiling hit. The runtime injected *"You've reached the maximum number of tool-calling iterations
|
||||
allowed. Please provide a final response…"* while the agent was still reading JD text for the borderline
|
||||
finalists, and the run terminated in the **scoring** phase.
|
||||
|
||||
The agent's own closing report was explicit — and it was **not delivered**:
|
||||
|
||||
> *"**Status: INCOMPLETE — handoff file NOT written, Stage 2 NOT triggered.** … Because the mandatory
|
||||
> deliverable (the handoff JSON) was not produced, today's roles have **not** been handed to Stage 2."*
|
||||
|
||||
That text is `delivery_outcome: suppressed`, so Anthony saw nothing on 11-02 — not the roles and not the
|
||||
warning that the roles were missing.
|
||||
|
||||
## Downstream: both Stage 2 slots behaved correctly
|
||||
|
||||
| Slot | Session | Ran | Result |
|
||||
|---|---|---|---|
|
||||
| 11:30 | `local-92665337…` (*Write and Letters*) | 11:30 → 12:25 | Exited `[SILENT]` after verifying Stage 1 was alive but had no handoff |
|
||||
| 14:30 (safety net) | `local-05a3f00b…` (*Write and Letters*) | 14:30 → 14:30 | Exited `[SILENT]` — 6 messages, 2 tool calls |
|
||||
|
||||
Both declines were correct: no handoff existed to consume. The safety-net design worked as intended — the
|
||||
gap is that Stage 1 cannot finish inside its budget.
|
||||
|
||||
## Recurrence pattern
|
||||
|
||||
| Date | Incident page | Failure shape |
|
||||
|---|---|---|
|
||||
| 2026-10-27 | [[incidents/2026-10-27-job-radar-budget-exhausted]] | Tool-call budget exhausted before the Notion write stage |
|
||||
| 2026-10-28 | [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] | Stage 1 hit the agent iteration cap before the handoff |
|
||||
| **2026-11-02** | **this page** | Same cap; **run recorded `completed` and its report suppressed** |
|
||||
|
||||
Three hits in seven days, each ending in the same place: discovery complete, handoff absent, Notion write
|
||||
stage never reached.
|
||||
|
||||
## The "green row, no artefact" anti-pattern
|
||||
|
||||
This is the **fourth appearance** of the class [[systems/scheduled-tasks]] §9 of the maintenance playbook
|
||||
already documents (a job whose durable ledger entry says success while the thing it was supposed to produce
|
||||
does not exist). The new variant here is that the ledger flipped from red to green:
|
||||
|
||||
| Date | `cron_incidents` row for `dca8482e4f76` | `executions.status` |
|
||||
|---|---|---|
|
||||
| 2026-10-27 | `dca848_e93a538ade6b` (`failure_type: auth`) | non-clean |
|
||||
| 2026-10-28 | `dca848_0a2f03b5952b` (`failure_type: timeout`) | non-clean |
|
||||
| **2026-11-02** | **none raised** | **`completed`, `error: NULL`** |
|
||||
|
||||
The two earlier caps each raised an incident whose `error` text is literally the run's INCOMPLETE report.
|
||||
The 2026-11-02 cap raised **nothing at all** (`cron_incidents` holds no row for `dca8482e4f76` with a
|
||||
`first_seen_at` on 2026-11-02; fleet-wide unclosed rows were **0** at the 2026-11-03 audit). The run's
|
||||
`delivery_outcome` is `suppressed`, so no message reached Anthony either.
|
||||
|
||||
**Detection rule:** for any job whose deliverable is a *file* (a handoff JSON, a report, a generated page),
|
||||
`executions.status` is not sufficient evidence. Verify the artefact exists at its expected path, or that
|
||||
`cron_incidents` stayed quiet for a reason, before treating the slot as served. Silence from
|
||||
`cron_incidents` is not the same as success — here it is the *symptom*.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Is the iteration budget the right control, or is the task too large for one agent turn?** Three caps in
|
||||
a row suggest Stage 1's work (four job boards → dedup → individual advert verification → JD reading →
|
||||
scoring) does not fit a single turn at any budget.
|
||||
2. **Should the harness record a cap-hit as something other than `completed`?** A distinct terminal status
|
||||
(or a non-null `error`) would make this visible to every existing health check.
|
||||
3. **Should the run's final response be delivered even when the job "succeeded"?** Suppressing an INCOMPLETE
|
||||
report converts a loud failure into silence.
|
||||
4. **Did the 11-02 roles get picked up on the next day's run**, or is 2026-11-02's candidate set lost?
|
||||
The 164 deduped candidates were never persisted.
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]
|
||||
- [[incidents/2026-10-27-job-radar-budget-exhausted]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/browser-backend]]
|
||||
@@ -3,15 +3,15 @@ title: Anthony's Systems Wiki
|
||||
type: collection
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-02
|
||||
verified_on: 2026-11-02
|
||||
updated: 2026-11-03
|
||||
verified_on: 2026-11-03
|
||||
confidence: high
|
||||
tags: [index, infrastructure, systems]
|
||||
---
|
||||
|
||||
# Anthony's Systems Wiki
|
||||
> Operational knowledge for Anthony's agents, infrastructure,
|
||||
> model routing and automation. Last updated: 2026-11-02 (nightly audit — the 11-01 fleet settled **167 / 1 / 2**, its first non-clean day in four: the Sunday **18:00 Weekly Review never ran and never delivered** (`79f9bb41ecf5`, *"Gateway connection failed; no local fallback"*, incident still open) and **three cron workers were `SIGTERM`ed (`-15`) mid-adoption** without writing a durable terminal state (`85edf6772bca` 07:30:46; `96eb95ce12f2` + `0e753e9720fd` 09:02:18) — for the first time with **no gateway restart anywhere in the window**, so the orphan class is now known to fire without a reload; a **fourth variant** is documented where **`ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning no execution row and no incident row**; delivery went **7 of 8**, the miss being `066bb5191280`'s `thread_id 34795` home-DM thread fallback (content delivered, run marked `failed`), while **Telegram flapped all day** (dual-stack `api.telegram.org` timeouts, disconnect 20:51:53, restarts 22:33:30 / 23:12:23) and the **relay adapter has now been absent ~29 h** (`Reconnecting relay` attempt 336); the longest-standing incident on the board — the 2026-10-01 Facebook-repin row — finally **closed at 10:44:12**)
|
||||
> model routing and automation. Last updated: 2026-11-03 (nightly audit — the 11-02 fleet settled **177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`**, the **first fully clean ledger day in five**, and **both 11-01 `unknown` rows resolved in-window**: the Sunday 18:00 **Weekly Review reconciled at 01:27:22** after **7 h 26 m 43 s** and **delivered**, so *"the lost review" was never lost* — incident `79f9bb_9c443b602eec` auto-closed **01:27:21.606691** and its page is now **`resolved`** — while `ecc1fc13655a`'s 11-01 ghost `error` cleared on its 12:00:33 slot; **unclosed incident rows 1 → 0** (the first zero in the period), non-clean `last_status` **2 → 1**, deliveries **9 rows / 9 delivered**; ⚠ **but two ~9-hour runs failed invisibly.** The **Job Radar Stage 1** (`dca8482e4f76`, 07:00:33 → 16:01:52, **9 h 01 m 19 s**) hit the **agent iteration cap before writing its handoff**, so **both Stage 2 slots exited `[SILENT]` and nothing reached Notion** — and it **raised no `cron_incidents` row at all**, where the same class raised incidents on 10-27 and 10-28, because its terminal state flipped to `completed` with `error: NULL`; the **03:00 stocktake** (`1a8e90ea2271`) likewise ran **9 h 43 m 29 s**, **delivered no report** and left its output uncommitted in the working tree (committed by this audit).)
|
||||
>
|
||||
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
|
||||
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
|
||||
@@ -65,7 +65,8 @@ tags: [index, infrastructure, systems]
|
||||
- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since *(superseded 2026-10-29 — two real calls are now on record: 08:38:04 / 08:49:36 → [[incidents/2026-10-29-conduit-voice-silence]])*. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]]
|
||||
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]]
|
||||
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery.
|
||||
- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] — **The Sunday 18:00 Weekly Review did not run on 2026-11-01 and nothing was delivered:** its execution closed `unknown` at 18:05:13 with *"Cron execution unverified … Gateway connection failed; no local fallback"*, its incident row is **still `alerted`**, the job sits at `error` / `failure_streak: 1`, and `deliveries.db` shows **no row after 11:01:12** for the rest of the day. Its batch-mate `b63ed5817212` took the identical error and self-healed in 68 s. Same page records the **inverse silent failure**: `ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning **no execution row and no incident row at all**.
|
||||
- [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] — **Third iteration-cap in seven days, and the first that raised no incident at all.** Job Radar Stage 1 (`dca8482e4f76`) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** on 2026-11-02, hit the **agent iteration cap before the handoff was written**, and closed **`status: completed` / `error: NULL` / `delivery_outcome: suppressed`**. Both Stage 2 slots declined correctly (`92665337…` 11:30 and the `05a3f00b…` 14:30 safety net both exited `[SILENT]`), so **nothing reached Notion, no cover letters were generated, and the day's 164 deduped candidates were never persisted** — a state a `status`-only health check reports as green. Contrast 10-27 (`dca848_e93a538ade6b`) and 10-28 (`dca848_0a2f03b5952b`), where the identical cap **did** open rows whose `error` text is the run's own INCOMPLETE report: on 11-02 the ledger **lost the red row it used to produce**. Discovery itself worked — SEEK **51**, LinkedIn **60 IDs**, Indeed live, WA Government **35** (197 raw / **164 new** after three-level dedup) — and a **URL-canonicalisation bug was found and fixed mid-run** (`au.seek.com` vs `au.seek.com.au` had been dropping SEEK *and* Indeed entries from the consolidated list). → [[systems/scheduled-tasks]]
|
||||
- [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] — **The Sunday 18:00 Weekly Review did not run on 2026-11-01 and nothing was delivered:** its execution closed `unknown` at 18:05:13 with *"Cron execution unverified … Gateway connection failed; no local fallback"*, its incident row is **still `alerted`**, the job sits at `error` / `failure_streak: 1`, and `deliveries.db` shows **no row after 11:01:12** for the rest of the day. Its batch-mate `b63ed5817212` took the identical error and self-healed in 68 s. Same page records the **inverse silent failure**: `ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning **no execution row and no incident row at all**. ✅ **RESOLVED 2026-11-02T01:27:21 — the review was never lost.** The *same* execution row (`be6ec11a8e68428084af9ab18fb148db`) closed **`completed` / `delivered` at 01:27:22** after **7 h 26 m 43 s**, the delivery lands at **01:27:21.802246** and incident `79f9bb_9c443b602eec` auto-closed at **01:27:21.606691** — so the 18:05:13 `unknown` was a **reconciler timing artefact on a still-running job**, not a verdict (page now `resolved`). Batch-mate `b63ed5` closed in 68 s and `79f9bb` in 7 h 26 m, both `completed`; **an `unknown` row on a long job is a timing artefact, not a failure**. `ecc1fc13655a`'s ghost `error` cleared too (12:00:33 → `completed` / `delivered` 12:29:26), taking **non-clean `last_status` 2 → 1**.
|
||||
- [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] — **Three restart-safe cron workers killed by `SIGTERM` (`-15`) after adopting their claim**, so the scheduler recorded *"whether side effects ran is unknown"*: `85edf6772bca` @07:30:46, `96eb95ce12f2` + `0e753e9720fd` @09:02:18. **A fourth variant of the orphan class** — and the first with **no gateway restart, reload or drain anywhere in the window** (PID `175269` up since 10-31 19:24:40; relay counter monotonic through both signals), so it cannot be attributed to reload behaviour. Likely trigger: a **30 s worker-ack window** smaller than this host's cold start (**83 plugins / 69 enabled** discovered in 70 s). All three rows now `resolved`; one execution left `unknown`.
|
||||
|
||||
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]]
|
||||
|
||||
@@ -1854,3 +1854,56 @@ recorded wrong.**
|
||||
(`origin/master..HEAD` empty) before this audit; touched pages were mirrored into the vault's
|
||||
existing conventions and hash-verified. **No secrets, passwords, tokens or keys written or quoted** —
|
||||
every job id, PID, thread id and timestamp above is a non-sensitive operational identifier.
|
||||
|
||||
## [2026-11-03] audit | Nightly audit — 2026-11-02 window (first clean ledger day in five, and two ~9-hour failures it cannot see)
|
||||
|
||||
- **Scope:** `state.db` sessions spanning 2026-11-02 00:00 → 2026-11-03 00:2x AWST; `cron/executions.db`,
|
||||
`cron/deliveries.db`, `cron_incidents`, `cron/jobs.json` (all read-only). Authoring tree
|
||||
`/home/hermes/workspace/agent-estate-wiki` was **clean and in sync with Gitea** (`origin/master..HEAD`
|
||||
empty) before this audit; `systems/skills-index.md` and `raw/stocktake/freshness-20261102.txt` were
|
||||
**committed on the stocktake's behalf** (see below).
|
||||
- **NEW — `incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md`:** the **third iteration-cap
|
||||
in seven days** (10-27 tool-call budget, 10-28 cap, 11-02 cap) on `dca8482e4f76`
|
||||
(**07:00:33 → 16:01:52, 9 h 01 m 19 s**), and **the first that raised no `cron_incidents` row at all** —
|
||||
it closed `completed` / `error: NULL` / `delivery_outcome: suppressed`. Both Stage 2 slots exited
|
||||
`[SILENT]`, so **nothing reached Notion, no cover letters were written and the 164 deduped candidates
|
||||
were never persisted**. Page carries the **"green row, no artefact" detection table** contrasting the
|
||||
three occurrences, and records the mid-run **URL-canonicalisation fix** (`au.seek.com` vs
|
||||
`au.seek.com.au`) that had been silently dropping SEEK and Indeed entries from dedup.
|
||||
- **UPDATED — `incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md`:** status
|
||||
**`active` → `resolved`**. Live evidence: the **same execution row**
|
||||
(`be6ec11a8e68428084af9ab18fb148db`) closed `completed` / `delivered` at **01:27:22** after
|
||||
**7 h 26 m 43 s**, delivery at 01:27:21.802246, incident `79f9bb_9c443b602eec` auto-closed at
|
||||
**01:27:21.606691**. The 18:05:13 `unknown` was a **reconciler timing artefact on a still-running job**,
|
||||
not a verdict. Resolution section added; open questions 1–3 answered, **question 4 retained** (the two
|
||||
unreconciled admission files `141448f9…json` / `69b5588f…json`).
|
||||
- **UPDATED — `systems/scheduled-tasks.md`:** `## Count` advanced to the 2026-11-02 window —
|
||||
**177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows) vs 167/1/2 on 11-01, 160/4 on
|
||||
10-31, 138/0 on 10-30; fleet unchanged at **29 jobs / 26 enabled / 3 disabled**; **unclosed incident
|
||||
rows 1 → 0** (the first zero in the period); both 11-01 `unknown` rows resolved and `ecc1fc13655a`'s
|
||||
ghost `error` cleared; deliveries **9 rows / 9 `delivered`**. New **`## Re-verified 2026-11-03`**
|
||||
section documents the two invisible failures, the stocktake artefacts committed, the
|
||||
`substr(started_at,1,10)` grouping rule and the **"do not read `running`/`unknown` as a verdict before
|
||||
the expected runtime has elapsed"** correction. Frontmatter `updated`/`verified_on` → 2026-11-03,
|
||||
`review_after` → 2026-11-10.
|
||||
- **UPDATED — `systems/browser-backend.md`:** `updated`/`verified_on` → 2026-11-03 plus a `## History`
|
||||
entry for the **2026-11-02 REST-server wedge** (requests timing out with nothing reaching the service
|
||||
log, cleared by `systemctl --user restart camofox-browser.service`; the four-board Job Radar run then
|
||||
proceeded). **Live re-probe 2026-11-03 00:16:14 AWST:** unit `active`, `MainPID 953299`,
|
||||
`ActiveEnterTimestamp Mon 2026-11-02 18:11:04`, `MemoryCurrent 37.5 MB`, `/health` → **200**
|
||||
`{"ok":true,"engine":"camoufox","browserConnected":false,…}`, `/tabs` → `{"running":true,"tabs":[]}`.
|
||||
Page now states explicitly that this wedge is a **different fault from the 2026-10-12 memory ceiling**
|
||||
(hang at 31 MB RSS vs exhaustion at 2.5 G), so the containment limits are not the applicable control.
|
||||
- **UPDATED — `index.md`:** `updated`/`verified_on` → 2026-11-03, the audit summary line rewritten for
|
||||
this window, the new 11-02 incident link added, and the 11-01 Weekly Review entry annotated
|
||||
**✅ RESOLVED 2026-11-02T01:27:21**.
|
||||
- **COMMITTED on the stocktake's behalf:** `1a8e90ea2271` finished with `systems/skills-index.md`
|
||||
modified and `raw/stocktake/freshness-20261102.txt` untracked. Both are valid artefacts — freshness
|
||||
**4 stale / 113 fresh** at a >21-day threshold, skills-index regenerated with **Devops 108 → 109**
|
||||
(`pulse-turn-handling`) — so this audit committed them rather than let the next slot overwrite them.
|
||||
- **Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned; no
|
||||
decision reversed. A full vault mirror pass was deliberately skipped again (drift counts instead of a
|
||||
bulk copy).
|
||||
- **DRIFT (context, not findings):** touched pages follow the vault's existing conventions and
|
||||
SCHEMA.md frontmatter rules. **No secrets, passwords, tokens or keys written or quoted** — every job
|
||||
id, PID, thread id and timestamp above is a non-sensitive operational identifier.
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
# Wiki freshness audit — 2026-11-02
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (4)
|
||||
|
||||
- [ 22d] agents/qwenpaw.md (updated 2026-10-11)
|
||||
- [ 22d] incidents/2026-09-28-hermes-managed-runtime-deps.md (updated 2026-10-11)
|
||||
- [ 22d] systems/ignis.md (updated 2026-10-11)
|
||||
- [ 22d] systems/paseo.md (updated 2026-10-11)
|
||||
|
||||
## FRESH (113)
|
||||
|
||||
- [ 13d] CONTRIBUTING.md
|
||||
- [ 7d] agents/hermes-chaos.md
|
||||
- [ 7d] agents/hermes-production.md
|
||||
- [ 13d] agents/index.md
|
||||
- [ 4d] agents/nanobot.md
|
||||
- [ 13d] agents/open-claw.md
|
||||
- [ 13d] agents/zeroclaw.md
|
||||
- [ 4d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 13d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 13d] comparisons/memory-backends.md
|
||||
- [ 13d] comparisons/model-routers.md
|
||||
- [ 4d] comparisons/steel-vs-camoufox.md
|
||||
- [ 13d] concepts/agent-memory.md
|
||||
- [ 4d] concepts/browser-automation.md
|
||||
- [ 13d] concepts/model-routing.md
|
||||
- [ 13d] concepts/persistent-shells.md
|
||||
- [ 13d] concepts/tool-calling.md
|
||||
- [ 2d] current-state.md
|
||||
- [ 4d] decisions/index.md
|
||||
- [ 13d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 13d] decisions/persistent-shell-enabled.md
|
||||
- [ 4d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 13d] experiments/index.md
|
||||
- [ 9d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 13d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 9d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 11d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 10d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 21d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 8d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 13d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 12d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 4d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 7d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 7d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 4d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 9d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 1d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 17d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 6d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 14d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 6d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 11d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 8d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 7d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 7d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 7d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 5d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 3d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 3d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 3d] incidents/2026-10-29-conduit-voice-silence.md
|
||||
- [ 2d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 3d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||
- [ 2d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 1d] incidents/2026-10-31-asana-access-failures.md
|
||||
- [ 1d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||
- [ 1d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||
- [ 0d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||
- [ 0d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||
- [ 0d] index.md
|
||||
- [ 11d] infrastructure/backups.md
|
||||
- [ 13d] infrastructure/credentials-map.md
|
||||
- [ 1d] infrastructure/docker-services.md
|
||||
- [ 13d] infrastructure/domains-and-tunnels.md
|
||||
- [ 2d] infrastructure/hosts.md
|
||||
- [ 19d] infrastructure/network-map.md
|
||||
- [ 1d] infrastructure/proxmox.md
|
||||
- [ 1d] infrastructure/tailscale.md
|
||||
- [ 21d] queries/retained-research-answers.md
|
||||
- [ 8d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 1d] reports/uber-eats/2026-07.md
|
||||
- [ 1d] reports/uber-eats/2026-08.md
|
||||
- [ 1d] reports/uber-eats/2026-09.md
|
||||
- [ 1d] reports/uber-eats/2026-10.md
|
||||
- [ 1d] reports/uber-eats/index.md
|
||||
- [ 13d] runbooks/backup-wiki.md
|
||||
- [ 13d] runbooks/diagnose-docker.md
|
||||
- [ 13d] runbooks/gateway-resource-resilience.md
|
||||
- [ 3d] runbooks/job-radar.md
|
||||
- [ 13d] runbooks/multiplexer-setup.md
|
||||
- [ 7d] runbooks/provider-health.md
|
||||
- [ 4d] runbooks/recover-docker-service.md
|
||||
- [ 13d] runbooks/restart-browser.md
|
||||
- [ 13d] runbooks/restart-hermes.md
|
||||
- [ 13d] runbooks/restore-openrouter.md
|
||||
- [ 13d] runbooks/test-openrouter.md
|
||||
- [ 13d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 7d] runbooks/update-hermes-safely.md
|
||||
- [ 4d] systems/browser-backend.md
|
||||
- [ 13d] systems/cloudflare-tunnel.md
|
||||
- [ 1d] systems/conduit.md
|
||||
- [ 7d] systems/current-profile.md
|
||||
- [ 13d] systems/guanaco.md
|
||||
- [ 13d] systems/headless-server.md
|
||||
- [ 3d] systems/hermes-agent.md
|
||||
- [ 13d] systems/hermes-ui.md
|
||||
- [ 21d] systems/home-assistant.md
|
||||
- [ 3d] systems/homepage-dashboard.md
|
||||
- [ 7d] systems/memory-backend.md
|
||||
- [ 0d] systems/messaging-integrations.md
|
||||
- [ 1d] systems/mnemosyne.md
|
||||
- [ 7d] systems/model-providers.md
|
||||
- [ 21d] systems/nanobot.md
|
||||
- [ 13d] systems/nextcloud.md
|
||||
- [ 13d] systems/obsidian.md
|
||||
- [ 1d] systems/omniroute.md
|
||||
- [ 13d] systems/opencode-go.md
|
||||
- [ 13d] systems/openrouter.md
|
||||
- [ 14d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 13d] systems/terminal-backend.md
|
||||
- [ 13d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 4 stale, 113 fresh, 3 no-date of 120 pages
|
||||
@@ -3,8 +3,8 @@ title: Browser Backend
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
updated: 2026-11-03
|
||||
verified_on: 2026-11-03
|
||||
confidence: high
|
||||
tags: [browser, camofox, automation]
|
||||
sources: [live-probe, config, session-history]
|
||||
@@ -112,6 +112,19 @@ until the next session starts.
|
||||
on **CT201** (containers, images, network, project dir, Serve routes) and **CT333** (container + image)
|
||||
torn down. Camofox is the sole backend.
|
||||
|
||||
- **2026-11-02 — the HTTP server wedged at the start of a browser-heavy cron run; a restart cleared it.**
|
||||
The 07:00 Job Radar Stage 1 (`dca8482e4f76`) found the REST server accepting nothing — requests timing out
|
||||
with **no lines reaching the service log** — before its first tab was opened. `systemctl --user restart
|
||||
camofox-browser.service` cleared it and the run went on to work four boards. **Live re-probe
|
||||
`2026-11-03 00:16:14 AWST`:** unit `active`, `MainPID 953299`, `ActiveEnterTimestamp Mon 2026-11-02
|
||||
18:11:04`, `MemoryCurrent 37.5 MB`; `/health` → **200**
|
||||
`{"ok":true,"engine":"camoufox","browserConnected":false,"browserRunning":false,"activeTabs":0,"activeSessions":0,"consecutiveFailures":0}`;
|
||||
`/tabs` → `{"running":true,"tabs":[]}`. Idle-clean and far inside the containment drop-in.
|
||||
⚠ **This is a different fault from the 2026-10-12 ceiling:** that one was memory exhaustion under many
|
||||
open tabs (`2.5G`, swap fully consumed); this one was a **hung HTTP server at 31 MB RSS**, so the
|
||||
containment limits are not the control that applies. A health probe inside browser-heavy scripts should
|
||||
therefore test **responsiveness**, not just `/health` status code.
|
||||
|
||||
## Correction (2026-10-29) — Steel is out of the stack, not just disabled
|
||||
|
||||
Verified against live state at **2026-10-29**. The earlier 00:3x probe said only "not reachable"; the truth is
|
||||
|
||||
+133
-4
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-02
|
||||
verified_on: 2026-11-02
|
||||
review_after: 2026-11-09
|
||||
updated: 2026-11-03
|
||||
verified_on: 2026-11-03
|
||||
review_after: 2026-11-10
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
@@ -14,7 +14,33 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-11-02 00:4x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
|
||||
- **Current as of 2026-11-03 00:1x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
|
||||
`deliveries.db`) — the first fully clean ledger day in five, but two long runs ended in a silent cap and
|
||||
one of them raised no incident at all.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged).
|
||||
**Window 2026-11-02 settled 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows;
|
||||
**96** of the completions are `96eb95ce12f2`'s 15-minute watchdog slots) against **167 / 1 / 2** on
|
||||
11-01, **160 / 4** on 10-31 and **138 / 0** on 10-30. **Both 11-01 `unknown` rows resolved on 11-02:**
|
||||
the Sunday 18:00 Weekly Review reconciled at **01:27:22** (`be6ec11a8e68428084af9ab18fb148db`,
|
||||
`completed` / `delivered`, **7 h 26 m 43 s** after it started) and its incident
|
||||
`79f9bb_9c443b602eec` auto-closed at **01:27:21.606691** →
|
||||
[[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] (**now `resolved`; the "lost
|
||||
review" was never lost**). `ecc1fc13655a`'s 11-01 ghost `error` also cleared — its 11-02 12:00:33 slot
|
||||
ran `completed` / `delivered` at 12:29:26. **Unclosed incident rows 1 → 0 — the first zero in the
|
||||
period.** **Non-clean `last_status` 2 → 1:** only `066bb5191280` (Uber Eats Monthly Spend Report,
|
||||
`delivery_failed`, `failure_streak: 0`, `last_run 2026-11-01T07:18:57`) — the 11-01 DM-thread miss, which
|
||||
**did not recur**; it has **no `executions` row on 11-02 at all**. **Delivery health: 9 rows on 11-02,
|
||||
9/9 `delivered`** (01:27:21, 07:30:35, 07:42:21, 08:10:18, 08:41:14, 09:07:20, 10:23:15, 12:29:25,
|
||||
18:45:37). ⚠ **Two long runs finished `completed` with `error: NULL` and `delivery_outcome:
|
||||
suppressed` after ~9 h — and the first raised no incident:** (a) **`dca8482e4f76`** (Job Radar Stage 1)
|
||||
**07:00:33 → 16:01:52 = 9 h 01 m 19 s**, hit the agent iteration cap before the handoff, so **both Stage 2
|
||||
slots (`92665337…` 11:30, `05a3f00b…` 14:30 safety net) exited `[SILENT]` and nothing reached Notion** —
|
||||
the same class that raised incidents on 10-27 and 10-28 **raised none this time** because the terminal
|
||||
state flipped to `completed` → [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]];
|
||||
(b) **`1a8e90ea2271`** (daily stocktake) **03:00:31 → 12:44:00 = 9 h 43 m 29 s**, `suppressed` — it
|
||||
**delivered no report** (unlike 11-01's 03:00:37 delivery) and **left its output uncommitted**
|
||||
(`systems/skills-index.md` modified, `raw/stocktake/freshness-20261102.txt` untracked; the 2026-11-03
|
||||
audit committed both on its behalf). See **Known Issues → Open** below.
|
||||
- **Prior window (2026-11-02 00:4x AWST, live `jobs.json` + `executions.db` + `cron_incidents` +
|
||||
`deliveries.db` + gateway log) — the fleet ran clean all day, then failed in three different ways
|
||||
and missed a delivery.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged). **Window 2026-11-01
|
||||
settled 167 `completed` / 1 `running` / 2 `unknown`** (+ this audit `running`, started 23:59:28)
|
||||
@@ -190,6 +216,23 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
|
||||
## Known Issues
|
||||
|
||||
### Resolved 2026-11-02
|
||||
|
||||
- **The "lost" Weekly Review was never lost — it delivered at 01:27:21, 7 h 26 m 43 s after its 18:00 slot.**
|
||||
Execution `be6ec11a8e68428084af9ab18fb148db` (job `79f9bb41ecf5`, `scheduled_instant`
|
||||
`2026-11-01T10:00:00+00:00`) started **2026-11-01T18:00:39.125776+08:00** and finally closed
|
||||
**2026-11-02T01:27:22.000881+08:00** with `status: completed`, `error: NULL`,
|
||||
`delivery_outcome: delivered`; the delivery row is timestamped **01:27:21.802246** and incident
|
||||
`79f9bb_9c443b602eec` auto-closed at **01:27:21.606691**. **Same execution row, not a re-run** — the
|
||||
`unknown` terminal state recorded at 18:05:13 was replaced when the run wrote its real outcome.
|
||||
→ [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] is now **`resolved`**.
|
||||
**Method lesson for this audit:** the 2026-11-02 00:4x reading of `status: unknown` + `closed_at: NULL`
|
||||
was accurate *at the time of reading* and had inverted 43 minutes later. **A nightly audit that runs
|
||||
~2 h after midnight must not treat a null terminal state on a long-running job as final** — the
|
||||
reconciler's window is shorter than these runs, so `unknown` on a long job is a timing artefact, not a
|
||||
verdict. The same false-negative logic applies to `ecc1fc13655a`, whose 11-01 ghost `error` cleared when
|
||||
its 11-02 12:00:33 slot closed `completed` / `delivered` at 12:29:26.
|
||||
|
||||
### Resolved 2026-10-30
|
||||
|
||||
- **The Mnemosyne Health Monitor was never really down — and it is enabled again (`ea79a2107e36`).** Its 00:01:08
|
||||
@@ -236,6 +279,23 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
- **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`.
|
||||
|
||||
### Open
|
||||
- **Two ~9-hour runs ended in a silent iteration cap on 2026-11-02, and the more damaging one raised no incident (NEW 2026-11-03):**
|
||||
- **`dca8482e4f76`** (Job Radar Stage 1) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** and closed
|
||||
`status: completed` / `error: NULL` / `delivery_outcome: suppressed` **after hitting the agent
|
||||
iteration cap before the handoff was written**. Both Stage 2 slots declined (`92665337…` 11:30 and the
|
||||
`05a3f00b…` 14:30 safety net both exited `[SILENT]`), so **nothing reached Notion**. This is the same
|
||||
class that raised `dca848_e93a538ade6b` (10-27) and `dca848_0a2f03b5952b` (10-28) — whose `error`
|
||||
columns are literally the runs' INCOMPLETE reports — yet **2026-11-02 produced no `cron_incidents` row
|
||||
at all**, because the terminal state flipped from non-clean to `completed`. Three caps in seven days
|
||||
(10-27, 10-28, 11-02) is a pattern, not bad luck.
|
||||
→ [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]. **Detection rule: for a job whose
|
||||
deliverable is a file, `executions.status` is not evidence — check the artefact.**
|
||||
- **`1a8e90ea2271`** (wiki stocktake) ran **03:00:31 → 12:44:00 = 9 h 43 m 29 s**, `completed` /
|
||||
`suppressed`, and **delivered no report** (11-01 delivered at 03:00:37; 11-02 has no delivery row near
|
||||
03:00). It left its output **uncommitted** — `systems/skills-index.md` modified and
|
||||
`raw/stocktake/freshness-20261102.txt` untracked — which the 2026-11-03 audit committed on its behalf.
|
||||
The artefacts themselves are sound (freshness report: **4 stale / 113 fresh** at a >21-day threshold;
|
||||
skills-index regenerated).
|
||||
- **GitHub Trending Radar's delivery token is revoked — needs a new @BotFather token (NEW 2026-10-13):** live `getMe` on the token in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401**, so `digest.py`'s Telegram send fails every Monday while the cron run itself stays green (`last_status: ok`). A *completed* run therefore does not mean the digest was sent by the script — the 2026-10-12 run only reached the DM because the agent recovered the content with `--dry-run` and its own response was delivered. Fix: issue a fresh token from @BotFather, replace it in `config.json`, and re-run `getMe` to confirm; do not re-read a green cron status as proof of delivery. **This corrects the 2026-10-05 "claim refuted" entry, which was wrong.** See [[systems/scheduled-tasks]] job `c0a9fa314542`.
|
||||
- **Camofox memory pressure during long browser runs (NEW 2026-10-13):** the 2026-10-12 07:01 Daily Marketing Job Radar run drove Camofox into the ceiling of its own systemd containment drop-in (`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`: `MemoryHigh=2G`, `MemoryMax=3G`, swap max 1G). Live probe mid-run read `Memory: 2.5G (high: 2G, max: 3G, swap: 1023.9M, available: 0B, peak: 2.5G)` with 383 tasks — swap fully consumed and no reclaimable headroom, which is why the `/tabs` endpoint and browser evals started hanging. The run still delivered: the agent stopped piling on memory, restarted the service, and finished one tab at a time with immediate close (post-restart: 2.1 GB used of 14 GB on CT460). Any job that opens many tabs in one pass is exposed to the same wall. Keep multi-tab extraction bounded, close each tab before opening the next, and consider raising the containment limits only if the host headroom justifies it. See [[systems/browser-backend]].
|
||||
- **Stocktake's Quartz rebuild step is dead code (NEW 2026-10-07):** `wiki-maintain.sh` documents a third step — rebuild the Quartz site — but the 2026-10-07 03:00 run logged it as skipped: no `quartz.config.ts` is present and `npx` is unavailable, so `public/` has not been refreshed. The skills-index and freshness artifacts are regenerated and committed correctly; only the published-HTML step is inert. Either restore the Quartz toolchain or drop the step from the script and the job description.
|
||||
@@ -754,3 +814,72 @@ GitHub Trending Radar token; Camofox containment under multi-tab load; the Condu
|
||||
retention; **plus new:** the unreconciled admission files
|
||||
`~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json` (both 11-01) and the unidentified `-15`
|
||||
signal source.
|
||||
|
||||
## Re-verified 2026-11-03 (audit window 2026-11-02 00:00 → 2026-11-03 00:2x AWST; live probes 00:0x–00:2x)
|
||||
|
||||
**The first fully clean ledger day in five — and it contains two failures the ledger does not show.**
|
||||
|
||||
**Settled: 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows) against 167 / 1 / 2 on
|
||||
11-01, 160 / 4 on 10-31 and 138 / 0 on 10-30. Fleet unchanged at **29 jobs, 26 enabled, 3 disabled**.
|
||||
Nothing was added, removed, disabled or repinned in the window. **96** of the completions are
|
||||
`96eb95ce12f2`'s 15-minute watchdog slots.
|
||||
|
||||
**Both 11-01 `unknown` rows resolved in-window** (see §Resolved 2026-11-02). **Unclosed incident rows
|
||||
1 → 0** — the first zero in the period. **Non-clean `last_status` 2 → 1.** Deliveries **9 rows, 9/9
|
||||
`delivered`.**
|
||||
|
||||
### Both faults of the day are invisible in the ledger
|
||||
|
||||
| # | Fault | Job | What the ledger says | Page |
|
||||
|---|---|---|---|---|
|
||||
| 1 | **Iteration cap reached before the handoff** — 9 h 01 m 19 s, Stage 2 never triggered, **nothing reached Notion**, and **no `cron_incidents` row was raised** | `dca8482e4f76` 07:00:33 → 16:01:52 | `completed` / `error: NULL` / `delivery_outcome: suppressed` | [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] |
|
||||
| 2 | **9 h 43 m run, no report delivered, output left uncommitted** | `1a8e90ea2271` 03:00:31 → 12:44:00 | `completed` / `error: NULL` / `delivery_outcome: suppressed` | this page → Known Issues → Open |
|
||||
|
||||
Fault 1 is the **third cap in seven days** (10-27 tool-call budget, 10-28 iteration cap, 11-02 iteration
|
||||
cap) and the **first that raised no incident at all** — the two earlier rows carry their runs' INCOMPLETE
|
||||
reports verbatim in `cron_incidents.error`. That is the sharpest form of the §9 anti-pattern yet: the
|
||||
ledger did not merely stay green over a dead job, it **lost the red row it used to produce**. Silence from
|
||||
`cron_incidents` is not success.
|
||||
|
||||
### Delivery health: 9 rows, 9 delivered
|
||||
|
||||
`deliveries.db` for 11-02: **01:27:21 ✔** (the Weekly Review recovery), 07:30:35 ✔, 07:42:21 ✔,
|
||||
08:10:18 ✔, 08:41:14 ✔, 09:07:20 ✔, 10:23:15 ✔, 12:29:25 ✔, 18:45:37 ✔. **The 11-01 `066bb5191280`
|
||||
thread-miss did not recur.** That job has **no `executions` row on 11-02 at all** (`0 2 1 * *`, next slot
|
||||
2026-12-01), so its 11-01 `delivery_failed` flag will sit on the fleet for the rest of November — the
|
||||
fleet's **only** non-clean `last_status`.
|
||||
|
||||
### Stocktake artefacts committed on the stocktake's behalf
|
||||
|
||||
`1a8e90ea2271` finished with `systems/skills-index.md` modified and
|
||||
`raw/stocktake/freshness-20261102.txt` untracked in the working tree. Both are valid artefacts —
|
||||
freshness **4 stale / 113 fresh** at a >21-day threshold; skills-index regenerated with **Devops 108 → 109**
|
||||
(`pulse-turn-handling` added) — so this audit **committed both** rather than leave them to be overwritten
|
||||
by the next slot.
|
||||
|
||||
### Method note (correction carried forward)
|
||||
|
||||
Two rules now bind this audit's tallies: group by `substr(started_at,1,10)`, never `date(...)` (which
|
||||
normalises to UTC and shifts 00:00–07:59 AWST rows onto the previous day); and **do not read a
|
||||
`running`/`unknown` row as a verdict until the job's expected runtime has elapsed** — the Weekly Review
|
||||
was still working at the 00:4x reading and delivered at 01:27, 43 minutes later. An audit that runs two
|
||||
hours after midnight will read every long job mid-flight.
|
||||
|
||||
### Pages touched
|
||||
|
||||
`incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` (new),
|
||||
`incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md` (status → **`resolved`**,
|
||||
Resolution section + answers to its open questions 1–3 added), `systems/scheduled-tasks.md`,
|
||||
`systems/browser-backend.md`, `systems/skills-index.md` (stocktake output, committed),
|
||||
`raw/stocktake/freshness-20261102.txt` (stocktake output, committed), `index.md`, `log.md`.
|
||||
**Nothing removed or archived:** no job retired, no system obsolete, no decision reversed in the window.
|
||||
|
||||
### Still open (carried + new)
|
||||
|
||||
The single-node no-failover topology behind every model-calling job; the GitHub Trending Radar delivery
|
||||
token; Camofox containment under multi-tab load; the Conduit health store with no retention; the
|
||||
unreconciled admission files `~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json`; the
|
||||
unidentified `-15` signal source; and the **30 s worker-ack window vs this host's cold-start time**.
|
||||
**Plus new:** whether 2026-11-02's **164 deduped Job Radar candidates are recoverable**, whether a
|
||||
cap-hit should keep being recorded as `completed`, and why the 03:00 stocktake now takes ~9 h 45 m.
|
||||
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
updated: 2026-11-02
|
||||
verified_on: 2026-11-02
|
||||
confidence: high
|
||||
tags: [skills, hermes]
|
||||
sources: []
|
||||
@@ -12,7 +12,7 @@ sources: []
|
||||
|
||||
# Installed Skills
|
||||
|
||||
**Total installed: 301 skills** in `~/.hermes/skills/`.
|
||||
**Total installed: 302 skills** in `~/.hermes/skills/`.
|
||||
|
||||
- **46 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
|
||||
- The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory.
|
||||
@@ -102,7 +102,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
|
||||
- `ddia-systems` — # Designing Data-Intensive Applications Framework
|
||||
|
||||
## Devops (108)
|
||||
## Devops (109)
|
||||
|
||||
- `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS)
|
||||
- `agent-extension-audit` — # Agent Extension Audit
|
||||
@@ -191,6 +191,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
- `omniroute-skill-caller` — # Call an OmniRoute Skill by Name (auto-resolving wrapper)
|
||||
- `omniroute-skill-execution` — # Execute Skills on OmniRoute MCP
|
||||
- `openclaw-ops` — # Open-Claw Operations & Configuration
|
||||
- `pulse-turn-handling` — # Pulse Turn Handling
|
||||
- `pve-ops` — # PVE Operations
|
||||
- `radar-jobs-dashboard` — # RADAR Jobs Dashboard
|
||||
- `readlater-ops` — # ReadLater Operations
|
||||
|
||||
Reference in New Issue
Block a user