Nightly audit 2026-11-03 (window 2026-11-02): new incident — Job Radar Stage 1 iteration-cap recurrence (9h 01m, suppressed, no incident raised); Weekly Review incident resolved (reconciled 01:27 AWST); scheduled-tasks + browser-backend re-verified; index/log updated

This commit is contained in:
Tony0410
2026-11-03 00:22:13 +08:00
parent fda56e53fe
commit ca014efd63
8 changed files with 516 additions and 20 deletions
+15 -2
View File
@@ -3,8 +3,8 @@ title: Browser Backend
type: system
status: active
created: 2026-07-22
updated: 2026-10-29
verified_on: 2026-10-29
updated: 2026-11-03
verified_on: 2026-11-03
confidence: high
tags: [browser, camofox, automation]
sources: [live-probe, config, session-history]
@@ -112,6 +112,19 @@ until the next session starts.
on **CT201** (containers, images, network, project dir, Serve routes) and **CT333** (container + image)
torn down. Camofox is the sole backend.
- **2026-11-02 — the HTTP server wedged at the start of a browser-heavy cron run; a restart cleared it.**
The 07:00 Job Radar Stage 1 (`dca8482e4f76`) found the REST server accepting nothing — requests timing out
with **no lines reaching the service log** — before its first tab was opened. `systemctl --user restart
camofox-browser.service` cleared it and the run went on to work four boards. **Live re-probe
`2026-11-03 00:16:14 AWST`:** unit `active`, `MainPID 953299`, `ActiveEnterTimestamp Mon 2026-11-02
18:11:04`, `MemoryCurrent 37.5 MB`; `/health` → **200**
`{"ok":true,"engine":"camoufox","browserConnected":false,"browserRunning":false,"activeTabs":0,"activeSessions":0,"consecutiveFailures":0}`;
`/tabs` → `{"running":true,"tabs":[]}`. Idle-clean and far inside the containment drop-in.
⚠ **This is a different fault from the 2026-10-12 ceiling:** that one was memory exhaustion under many
open tabs (`2.5G`, swap fully consumed); this one was a **hung HTTP server at 31 MB RSS**, so the
containment limits are not the control that applies. A health probe inside browser-heavy scripts should
therefore test **responsiveness**, not just `/health` status code.
## Correction (2026-10-29) — Steel is out of the stack, not just disabled
Verified against live state at **2026-10-29**. The earlier 00:3x probe said only "not reachable"; the truth is
+133 -4
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-11-02
verified_on: 2026-11-02
review_after: 2026-11-09
updated: 2026-11-03
verified_on: 2026-11-03
review_after: 2026-11-10
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -14,7 +14,33 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks
## Count
- **Current as of 2026-11-02 00:4x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
- **Current as of 2026-11-03 00:1x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
`deliveries.db`) — the first fully clean ledger day in five, but two long runs ended in a silent cap and
one of them raised no incident at all.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged).
**Window 2026-11-02 settled 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows;
**96** of the completions are `96eb95ce12f2`'s 15-minute watchdog slots) against **167 / 1 / 2** on
11-01, **160 / 4** on 10-31 and **138 / 0** on 10-30. **Both 11-01 `unknown` rows resolved on 11-02:**
the Sunday 18:00 Weekly Review reconciled at **01:27:22** (`be6ec11a8e68428084af9ab18fb148db`,
`completed` / `delivered`, **7 h 26 m 43 s** after it started) and its incident
`79f9bb_9c443b602eec` auto-closed at **01:27:21.606691** →
[[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] (**now `resolved`; the "lost
review" was never lost**). `ecc1fc13655a`'s 11-01 ghost `error` also cleared — its 11-02 12:00:33 slot
ran `completed` / `delivered` at 12:29:26. **Unclosed incident rows 1 → 0 — the first zero in the
period.** **Non-clean `last_status` 2 → 1:** only `066bb5191280` (Uber Eats Monthly Spend Report,
`delivery_failed`, `failure_streak: 0`, `last_run 2026-11-01T07:18:57`) — the 11-01 DM-thread miss, which
**did not recur**; it has **no `executions` row on 11-02 at all**. **Delivery health: 9 rows on 11-02,
9/9 `delivered`** (01:27:21, 07:30:35, 07:42:21, 08:10:18, 08:41:14, 09:07:20, 10:23:15, 12:29:25,
18:45:37). ⚠ **Two long runs finished `completed` with `error: NULL` and `delivery_outcome:
suppressed` after ~9 h — and the first raised no incident:** (a) **`dca8482e4f76`** (Job Radar Stage 1)
**07:00:33 → 16:01:52 = 9 h 01 m 19 s**, hit the agent iteration cap before the handoff, so **both Stage 2
slots (`92665337…` 11:30, `05a3f00b…` 14:30 safety net) exited `[SILENT]` and nothing reached Notion** —
the same class that raised incidents on 10-27 and 10-28 **raised none this time** because the terminal
state flipped to `completed` → [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]];
(b) **`1a8e90ea2271`** (daily stocktake) **03:00:31 → 12:44:00 = 9 h 43 m 29 s**, `suppressed` — it
**delivered no report** (unlike 11-01's 03:00:37 delivery) and **left its output uncommitted**
(`systems/skills-index.md` modified, `raw/stocktake/freshness-20261102.txt` untracked; the 2026-11-03
audit committed both on its behalf). See **Known Issues → Open** below.
- **Prior window (2026-11-02 00:4x AWST, live `jobs.json` + `executions.db` + `cron_incidents` +
`deliveries.db` + gateway log) — the fleet ran clean all day, then failed in three different ways
and missed a delivery.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged). **Window 2026-11-01
settled 167 `completed` / 1 `running` / 2 `unknown`** (+ this audit `running`, started 23:59:28)
@@ -190,6 +216,23 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
## Known Issues
### Resolved 2026-11-02
- **The "lost" Weekly Review was never lost — it delivered at 01:27:21, 7 h 26 m 43 s after its 18:00 slot.**
Execution `be6ec11a8e68428084af9ab18fb148db` (job `79f9bb41ecf5`, `scheduled_instant`
`2026-11-01T10:00:00+00:00`) started **2026-11-01T18:00:39.125776+08:00** and finally closed
**2026-11-02T01:27:22.000881+08:00** with `status: completed`, `error: NULL`,
`delivery_outcome: delivered`; the delivery row is timestamped **01:27:21.802246** and incident
`79f9bb_9c443b602eec` auto-closed at **01:27:21.606691**. **Same execution row, not a re-run** — the
`unknown` terminal state recorded at 18:05:13 was replaced when the run wrote its real outcome.
→ [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] is now **`resolved`**.
**Method lesson for this audit:** the 2026-11-02 00:4x reading of `status: unknown` + `closed_at: NULL`
was accurate *at the time of reading* and had inverted 43 minutes later. **A nightly audit that runs
~2 h after midnight must not treat a null terminal state on a long-running job as final** — the
reconciler's window is shorter than these runs, so `unknown` on a long job is a timing artefact, not a
verdict. The same false-negative logic applies to `ecc1fc13655a`, whose 11-01 ghost `error` cleared when
its 11-02 12:00:33 slot closed `completed` / `delivered` at 12:29:26.
### Resolved 2026-10-30
- **The Mnemosyne Health Monitor was never really down — and it is enabled again (`ea79a2107e36`).** Its 00:01:08
@@ -236,6 +279,23 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
- **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`.
### Open
- **Two ~9-hour runs ended in a silent iteration cap on 2026-11-02, and the more damaging one raised no incident (NEW 2026-11-03):**
- **`dca8482e4f76`** (Job Radar Stage 1) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** and closed
`status: completed` / `error: NULL` / `delivery_outcome: suppressed` **after hitting the agent
iteration cap before the handoff was written**. Both Stage 2 slots declined (`92665337…` 11:30 and the
`05a3f00b…` 14:30 safety net both exited `[SILENT]`), so **nothing reached Notion**. This is the same
class that raised `dca848_e93a538ade6b` (10-27) and `dca848_0a2f03b5952b` (10-28) — whose `error`
columns are literally the runs' INCOMPLETE reports — yet **2026-11-02 produced no `cron_incidents` row
at all**, because the terminal state flipped from non-clean to `completed`. Three caps in seven days
(10-27, 10-28, 11-02) is a pattern, not bad luck.
→ [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]. **Detection rule: for a job whose
deliverable is a file, `executions.status` is not evidence — check the artefact.**
- **`1a8e90ea2271`** (wiki stocktake) ran **03:00:31 → 12:44:00 = 9 h 43 m 29 s**, `completed` /
`suppressed`, and **delivered no report** (11-01 delivered at 03:00:37; 11-02 has no delivery row near
03:00). It left its output **uncommitted** — `systems/skills-index.md` modified and
`raw/stocktake/freshness-20261102.txt` untracked — which the 2026-11-03 audit committed on its behalf.
The artefacts themselves are sound (freshness report: **4 stale / 113 fresh** at a >21-day threshold;
skills-index regenerated).
- **GitHub Trending Radar's delivery token is revoked — needs a new @BotFather token (NEW 2026-10-13):** live `getMe` on the token in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401**, so `digest.py`'s Telegram send fails every Monday while the cron run itself stays green (`last_status: ok`). A *completed* run therefore does not mean the digest was sent by the script — the 2026-10-12 run only reached the DM because the agent recovered the content with `--dry-run` and its own response was delivered. Fix: issue a fresh token from @BotFather, replace it in `config.json`, and re-run `getMe` to confirm; do not re-read a green cron status as proof of delivery. **This corrects the 2026-10-05 "claim refuted" entry, which was wrong.** See [[systems/scheduled-tasks]] job `c0a9fa314542`.
- **Camofox memory pressure during long browser runs (NEW 2026-10-13):** the 2026-10-12 07:01 Daily Marketing Job Radar run drove Camofox into the ceiling of its own systemd containment drop-in (`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`: `MemoryHigh=2G`, `MemoryMax=3G`, swap max 1G). Live probe mid-run read `Memory: 2.5G (high: 2G, max: 3G, swap: 1023.9M, available: 0B, peak: 2.5G)` with 383 tasks — swap fully consumed and no reclaimable headroom, which is why the `/tabs` endpoint and browser evals started hanging. The run still delivered: the agent stopped piling on memory, restarted the service, and finished one tab at a time with immediate close (post-restart: 2.1 GB used of 14 GB on CT460). Any job that opens many tabs in one pass is exposed to the same wall. Keep multi-tab extraction bounded, close each tab before opening the next, and consider raising the containment limits only if the host headroom justifies it. See [[systems/browser-backend]].
- **Stocktake's Quartz rebuild step is dead code (NEW 2026-10-07):** `wiki-maintain.sh` documents a third step — rebuild the Quartz site — but the 2026-10-07 03:00 run logged it as skipped: no `quartz.config.ts` is present and `npx` is unavailable, so `public/` has not been refreshed. The skills-index and freshness artifacts are regenerated and committed correctly; only the published-HTML step is inert. Either restore the Quartz toolchain or drop the step from the script and the job description.
@@ -754,3 +814,72 @@ GitHub Trending Radar token; Camofox containment under multi-tab load; the Condu
retention; **plus new:** the unreconciled admission files
`~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json` (both 11-01) and the unidentified `-15`
signal source.
## Re-verified 2026-11-03 (audit window 2026-11-02 00:00 → 2026-11-03 00:2x AWST; live probes 00:0x–00:2x)
**The first fully clean ledger day in five — and it contains two failures the ledger does not show.**
**Settled: 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows) against 167 / 1 / 2 on
11-01, 160 / 4 on 10-31 and 138 / 0 on 10-30. Fleet unchanged at **29 jobs, 26 enabled, 3 disabled**.
Nothing was added, removed, disabled or repinned in the window. **96** of the completions are
`96eb95ce12f2`'s 15-minute watchdog slots.
**Both 11-01 `unknown` rows resolved in-window** (see §Resolved 2026-11-02). **Unclosed incident rows
1 → 0** — the first zero in the period. **Non-clean `last_status` 2 → 1.** Deliveries **9 rows, 9/9
`delivered`.**
### Both faults of the day are invisible in the ledger
| # | Fault | Job | What the ledger says | Page |
|---|---|---|---|---|
| 1 | **Iteration cap reached before the handoff** — 9 h 01 m 19 s, Stage 2 never triggered, **nothing reached Notion**, and **no `cron_incidents` row was raised** | `dca8482e4f76` 07:00:33 → 16:01:52 | `completed` / `error: NULL` / `delivery_outcome: suppressed` | [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] |
| 2 | **9 h 43 m run, no report delivered, output left uncommitted** | `1a8e90ea2271` 03:00:31 → 12:44:00 | `completed` / `error: NULL` / `delivery_outcome: suppressed` | this page → Known Issues → Open |
Fault 1 is the **third cap in seven days** (10-27 tool-call budget, 10-28 iteration cap, 11-02 iteration
cap) and the **first that raised no incident at all** — the two earlier rows carry their runs' INCOMPLETE
reports verbatim in `cron_incidents.error`. That is the sharpest form of the §9 anti-pattern yet: the
ledger did not merely stay green over a dead job, it **lost the red row it used to produce**. Silence from
`cron_incidents` is not success.
### Delivery health: 9 rows, 9 delivered
`deliveries.db` for 11-02: **01:27:21 ✔** (the Weekly Review recovery), 07:30:35 ✔, 07:42:21 ✔,
08:10:18 ✔, 08:41:14 ✔, 09:07:20 ✔, 10:23:15 ✔, 12:29:25 ✔, 18:45:37 ✔. **The 11-01 `066bb5191280`
thread-miss did not recur.** That job has **no `executions` row on 11-02 at all** (`0 2 1 * *`, next slot
2026-12-01), so its 11-01 `delivery_failed` flag will sit on the fleet for the rest of November — the
fleet's **only** non-clean `last_status`.
### Stocktake artefacts committed on the stocktake's behalf
`1a8e90ea2271` finished with `systems/skills-index.md` modified and
`raw/stocktake/freshness-20261102.txt` untracked in the working tree. Both are valid artefacts —
freshness **4 stale / 113 fresh** at a >21-day threshold; skills-index regenerated with **Devops 108 → 109**
(`pulse-turn-handling` added) — so this audit **committed both** rather than leave them to be overwritten
by the next slot.
### Method note (correction carried forward)
Two rules now bind this audit's tallies: group by `substr(started_at,1,10)`, never `date(...)` (which
normalises to UTC and shifts 00:00–07:59 AWST rows onto the previous day); and **do not read a
`running`/`unknown` row as a verdict until the job's expected runtime has elapsed** — the Weekly Review
was still working at the 00:4x reading and delivered at 01:27, 43 minutes later. An audit that runs two
hours after midnight will read every long job mid-flight.
### Pages touched
`incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` (new),
`incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md` (status → **`resolved`**,
Resolution section + answers to its open questions 1–3 added), `systems/scheduled-tasks.md`,
`systems/browser-backend.md`, `systems/skills-index.md` (stocktake output, committed),
`raw/stocktake/freshness-20261102.txt` (stocktake output, committed), `index.md`, `log.md`.
**Nothing removed or archived:** no job retired, no system obsolete, no decision reversed in the window.
### Still open (carried + new)
The single-node no-failover topology behind every model-calling job; the GitHub Trending Radar delivery
token; Camofox containment under multi-tab load; the Conduit health store with no retention; the
unreconciled admission files `~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json`; the
unidentified `-15` signal source; and the **30 s worker-ack window vs this host's cold-start time**.
**Plus new:** whether 2026-11-02's **164 deduped Job Radar candidates are recoverable**, whether a
cap-hit should keep being recorded as `completed`, and why the 03:00 stocktake now takes ~9 h 45 m.
+5 -4
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system
status: active
created: 2026-07-22
updated: 2026-11-01
verified_on: 2026-11-01
updated: 2026-11-02
verified_on: 2026-11-02
confidence: high
tags: [skills, hermes]
sources: []
@@ -12,7 +12,7 @@ sources: []
# Installed Skills
**Total installed: 301 skills** in `~/.hermes/skills/`.
**Total installed: 302 skills** in `~/.hermes/skills/`.
- **46 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
- The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory.
@@ -102,7 +102,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `ddia-systems` — # Designing Data-Intensive Applications Framework
## Devops (108)
## Devops (109)
- `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS)
- `agent-extension-audit` — # Agent Extension Audit
@@ -191,6 +191,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `omniroute-skill-caller` — # Call an OmniRoute Skill by Name (auto-resolving wrapper)
- `omniroute-skill-execution` — # Execute Skills on OmniRoute MCP
- `openclaw-ops` — # Open-Claw Operations & Configuration
- `pulse-turn-handling` — # Pulse Turn Handling
- `pve-ops` — # PVE Operations
- `radar-jobs-dashboard` — # RADAR Jobs Dashboard
- `readlater-ops` — # ReadLater Operations