Nightly audit 2026-11-03 (window 2026-11-02): new incident — Job Radar Stage 1 iteration-cap recurrence (9h 01m, suppressed, no incident raised); Weekly Review incident resolved (reconciled 01:27 AWST); scheduled-tasks + browser-backend re-verified; index/log updated
This commit is contained in:
+133
-4
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-02
|
||||
verified_on: 2026-11-02
|
||||
review_after: 2026-11-09
|
||||
updated: 2026-11-03
|
||||
verified_on: 2026-11-03
|
||||
review_after: 2026-11-10
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
@@ -14,7 +14,33 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-11-02 00:4x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
|
||||
- **Current as of 2026-11-03 00:1x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
|
||||
`deliveries.db`) — the first fully clean ledger day in five, but two long runs ended in a silent cap and
|
||||
one of them raised no incident at all.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged).
|
||||
**Window 2026-11-02 settled 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows;
|
||||
**96** of the completions are `96eb95ce12f2`'s 15-minute watchdog slots) against **167 / 1 / 2** on
|
||||
11-01, **160 / 4** on 10-31 and **138 / 0** on 10-30. **Both 11-01 `unknown` rows resolved on 11-02:**
|
||||
the Sunday 18:00 Weekly Review reconciled at **01:27:22** (`be6ec11a8e68428084af9ab18fb148db`,
|
||||
`completed` / `delivered`, **7 h 26 m 43 s** after it started) and its incident
|
||||
`79f9bb_9c443b602eec` auto-closed at **01:27:21.606691** →
|
||||
[[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] (**now `resolved`; the "lost
|
||||
review" was never lost**). `ecc1fc13655a`'s 11-01 ghost `error` also cleared — its 11-02 12:00:33 slot
|
||||
ran `completed` / `delivered` at 12:29:26. **Unclosed incident rows 1 → 0 — the first zero in the
|
||||
period.** **Non-clean `last_status` 2 → 1:** only `066bb5191280` (Uber Eats Monthly Spend Report,
|
||||
`delivery_failed`, `failure_streak: 0`, `last_run 2026-11-01T07:18:57`) — the 11-01 DM-thread miss, which
|
||||
**did not recur**; it has **no `executions` row on 11-02 at all**. **Delivery health: 9 rows on 11-02,
|
||||
9/9 `delivered`** (01:27:21, 07:30:35, 07:42:21, 08:10:18, 08:41:14, 09:07:20, 10:23:15, 12:29:25,
|
||||
18:45:37). ⚠ **Two long runs finished `completed` with `error: NULL` and `delivery_outcome:
|
||||
suppressed` after ~9 h — and the first raised no incident:** (a) **`dca8482e4f76`** (Job Radar Stage 1)
|
||||
**07:00:33 → 16:01:52 = 9 h 01 m 19 s**, hit the agent iteration cap before the handoff, so **both Stage 2
|
||||
slots (`92665337…` 11:30, `05a3f00b…` 14:30 safety net) exited `[SILENT]` and nothing reached Notion** —
|
||||
the same class that raised incidents on 10-27 and 10-28 **raised none this time** because the terminal
|
||||
state flipped to `completed` → [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]];
|
||||
(b) **`1a8e90ea2271`** (daily stocktake) **03:00:31 → 12:44:00 = 9 h 43 m 29 s**, `suppressed` — it
|
||||
**delivered no report** (unlike 11-01's 03:00:37 delivery) and **left its output uncommitted**
|
||||
(`systems/skills-index.md` modified, `raw/stocktake/freshness-20261102.txt` untracked; the 2026-11-03
|
||||
audit committed both on its behalf). See **Known Issues → Open** below.
|
||||
- **Prior window (2026-11-02 00:4x AWST, live `jobs.json` + `executions.db` + `cron_incidents` +
|
||||
`deliveries.db` + gateway log) — the fleet ran clean all day, then failed in three different ways
|
||||
and missed a delivery.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged). **Window 2026-11-01
|
||||
settled 167 `completed` / 1 `running` / 2 `unknown`** (+ this audit `running`, started 23:59:28)
|
||||
@@ -190,6 +216,23 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
|
||||
## Known Issues
|
||||
|
||||
### Resolved 2026-11-02
|
||||
|
||||
- **The "lost" Weekly Review was never lost — it delivered at 01:27:21, 7 h 26 m 43 s after its 18:00 slot.**
|
||||
Execution `be6ec11a8e68428084af9ab18fb148db` (job `79f9bb41ecf5`, `scheduled_instant`
|
||||
`2026-11-01T10:00:00+00:00`) started **2026-11-01T18:00:39.125776+08:00** and finally closed
|
||||
**2026-11-02T01:27:22.000881+08:00** with `status: completed`, `error: NULL`,
|
||||
`delivery_outcome: delivered`; the delivery row is timestamped **01:27:21.802246** and incident
|
||||
`79f9bb_9c443b602eec` auto-closed at **01:27:21.606691**. **Same execution row, not a re-run** — the
|
||||
`unknown` terminal state recorded at 18:05:13 was replaced when the run wrote its real outcome.
|
||||
→ [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] is now **`resolved`**.
|
||||
**Method lesson for this audit:** the 2026-11-02 00:4x reading of `status: unknown` + `closed_at: NULL`
|
||||
was accurate *at the time of reading* and had inverted 43 minutes later. **A nightly audit that runs
|
||||
~2 h after midnight must not treat a null terminal state on a long-running job as final** — the
|
||||
reconciler's window is shorter than these runs, so `unknown` on a long job is a timing artefact, not a
|
||||
verdict. The same false-negative logic applies to `ecc1fc13655a`, whose 11-01 ghost `error` cleared when
|
||||
its 11-02 12:00:33 slot closed `completed` / `delivered` at 12:29:26.
|
||||
|
||||
### Resolved 2026-10-30
|
||||
|
||||
- **The Mnemosyne Health Monitor was never really down — and it is enabled again (`ea79a2107e36`).** Its 00:01:08
|
||||
@@ -236,6 +279,23 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
- **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`.
|
||||
|
||||
### Open
|
||||
- **Two ~9-hour runs ended in a silent iteration cap on 2026-11-02, and the more damaging one raised no incident (NEW 2026-11-03):**
|
||||
- **`dca8482e4f76`** (Job Radar Stage 1) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** and closed
|
||||
`status: completed` / `error: NULL` / `delivery_outcome: suppressed` **after hitting the agent
|
||||
iteration cap before the handoff was written**. Both Stage 2 slots declined (`92665337…` 11:30 and the
|
||||
`05a3f00b…` 14:30 safety net both exited `[SILENT]`), so **nothing reached Notion**. This is the same
|
||||
class that raised `dca848_e93a538ade6b` (10-27) and `dca848_0a2f03b5952b` (10-28) — whose `error`
|
||||
columns are literally the runs' INCOMPLETE reports — yet **2026-11-02 produced no `cron_incidents` row
|
||||
at all**, because the terminal state flipped from non-clean to `completed`. Three caps in seven days
|
||||
(10-27, 10-28, 11-02) is a pattern, not bad luck.
|
||||
→ [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]. **Detection rule: for a job whose
|
||||
deliverable is a file, `executions.status` is not evidence — check the artefact.**
|
||||
- **`1a8e90ea2271`** (wiki stocktake) ran **03:00:31 → 12:44:00 = 9 h 43 m 29 s**, `completed` /
|
||||
`suppressed`, and **delivered no report** (11-01 delivered at 03:00:37; 11-02 has no delivery row near
|
||||
03:00). It left its output **uncommitted** — `systems/skills-index.md` modified and
|
||||
`raw/stocktake/freshness-20261102.txt` untracked — which the 2026-11-03 audit committed on its behalf.
|
||||
The artefacts themselves are sound (freshness report: **4 stale / 113 fresh** at a >21-day threshold;
|
||||
skills-index regenerated).
|
||||
- **GitHub Trending Radar's delivery token is revoked — needs a new @BotFather token (NEW 2026-10-13):** live `getMe` on the token in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401**, so `digest.py`'s Telegram send fails every Monday while the cron run itself stays green (`last_status: ok`). A *completed* run therefore does not mean the digest was sent by the script — the 2026-10-12 run only reached the DM because the agent recovered the content with `--dry-run` and its own response was delivered. Fix: issue a fresh token from @BotFather, replace it in `config.json`, and re-run `getMe` to confirm; do not re-read a green cron status as proof of delivery. **This corrects the 2026-10-05 "claim refuted" entry, which was wrong.** See [[systems/scheduled-tasks]] job `c0a9fa314542`.
|
||||
- **Camofox memory pressure during long browser runs (NEW 2026-10-13):** the 2026-10-12 07:01 Daily Marketing Job Radar run drove Camofox into the ceiling of its own systemd containment drop-in (`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`: `MemoryHigh=2G`, `MemoryMax=3G`, swap max 1G). Live probe mid-run read `Memory: 2.5G (high: 2G, max: 3G, swap: 1023.9M, available: 0B, peak: 2.5G)` with 383 tasks — swap fully consumed and no reclaimable headroom, which is why the `/tabs` endpoint and browser evals started hanging. The run still delivered: the agent stopped piling on memory, restarted the service, and finished one tab at a time with immediate close (post-restart: 2.1 GB used of 14 GB on CT460). Any job that opens many tabs in one pass is exposed to the same wall. Keep multi-tab extraction bounded, close each tab before opening the next, and consider raising the containment limits only if the host headroom justifies it. See [[systems/browser-backend]].
|
||||
- **Stocktake's Quartz rebuild step is dead code (NEW 2026-10-07):** `wiki-maintain.sh` documents a third step — rebuild the Quartz site — but the 2026-10-07 03:00 run logged it as skipped: no `quartz.config.ts` is present and `npx` is unavailable, so `public/` has not been refreshed. The skills-index and freshness artifacts are regenerated and committed correctly; only the published-HTML step is inert. Either restore the Quartz toolchain or drop the step from the script and the job description.
|
||||
@@ -754,3 +814,72 @@ GitHub Trending Radar token; Camofox containment under multi-tab load; the Condu
|
||||
retention; **plus new:** the unreconciled admission files
|
||||
`~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json` (both 11-01) and the unidentified `-15`
|
||||
signal source.
|
||||
|
||||
## Re-verified 2026-11-03 (audit window 2026-11-02 00:00 → 2026-11-03 00:2x AWST; live probes 00:0x–00:2x)
|
||||
|
||||
**The first fully clean ledger day in five — and it contains two failures the ledger does not show.**
|
||||
|
||||
**Settled: 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows) against 167 / 1 / 2 on
|
||||
11-01, 160 / 4 on 10-31 and 138 / 0 on 10-30. Fleet unchanged at **29 jobs, 26 enabled, 3 disabled**.
|
||||
Nothing was added, removed, disabled or repinned in the window. **96** of the completions are
|
||||
`96eb95ce12f2`'s 15-minute watchdog slots.
|
||||
|
||||
**Both 11-01 `unknown` rows resolved in-window** (see §Resolved 2026-11-02). **Unclosed incident rows
|
||||
1 → 0** — the first zero in the period. **Non-clean `last_status` 2 → 1.** Deliveries **9 rows, 9/9
|
||||
`delivered`.**
|
||||
|
||||
### Both faults of the day are invisible in the ledger
|
||||
|
||||
| # | Fault | Job | What the ledger says | Page |
|
||||
|---|---|---|---|---|
|
||||
| 1 | **Iteration cap reached before the handoff** — 9 h 01 m 19 s, Stage 2 never triggered, **nothing reached Notion**, and **no `cron_incidents` row was raised** | `dca8482e4f76` 07:00:33 → 16:01:52 | `completed` / `error: NULL` / `delivery_outcome: suppressed` | [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] |
|
||||
| 2 | **9 h 43 m run, no report delivered, output left uncommitted** | `1a8e90ea2271` 03:00:31 → 12:44:00 | `completed` / `error: NULL` / `delivery_outcome: suppressed` | this page → Known Issues → Open |
|
||||
|
||||
Fault 1 is the **third cap in seven days** (10-27 tool-call budget, 10-28 iteration cap, 11-02 iteration
|
||||
cap) and the **first that raised no incident at all** — the two earlier rows carry their runs' INCOMPLETE
|
||||
reports verbatim in `cron_incidents.error`. That is the sharpest form of the §9 anti-pattern yet: the
|
||||
ledger did not merely stay green over a dead job, it **lost the red row it used to produce**. Silence from
|
||||
`cron_incidents` is not success.
|
||||
|
||||
### Delivery health: 9 rows, 9 delivered
|
||||
|
||||
`deliveries.db` for 11-02: **01:27:21 ✔** (the Weekly Review recovery), 07:30:35 ✔, 07:42:21 ✔,
|
||||
08:10:18 ✔, 08:41:14 ✔, 09:07:20 ✔, 10:23:15 ✔, 12:29:25 ✔, 18:45:37 ✔. **The 11-01 `066bb5191280`
|
||||
thread-miss did not recur.** That job has **no `executions` row on 11-02 at all** (`0 2 1 * *`, next slot
|
||||
2026-12-01), so its 11-01 `delivery_failed` flag will sit on the fleet for the rest of November — the
|
||||
fleet's **only** non-clean `last_status`.
|
||||
|
||||
### Stocktake artefacts committed on the stocktake's behalf
|
||||
|
||||
`1a8e90ea2271` finished with `systems/skills-index.md` modified and
|
||||
`raw/stocktake/freshness-20261102.txt` untracked in the working tree. Both are valid artefacts —
|
||||
freshness **4 stale / 113 fresh** at a >21-day threshold; skills-index regenerated with **Devops 108 → 109**
|
||||
(`pulse-turn-handling` added) — so this audit **committed both** rather than leave them to be overwritten
|
||||
by the next slot.
|
||||
|
||||
### Method note (correction carried forward)
|
||||
|
||||
Two rules now bind this audit's tallies: group by `substr(started_at,1,10)`, never `date(...)` (which
|
||||
normalises to UTC and shifts 00:00–07:59 AWST rows onto the previous day); and **do not read a
|
||||
`running`/`unknown` row as a verdict until the job's expected runtime has elapsed** — the Weekly Review
|
||||
was still working at the 00:4x reading and delivered at 01:27, 43 minutes later. An audit that runs two
|
||||
hours after midnight will read every long job mid-flight.
|
||||
|
||||
### Pages touched
|
||||
|
||||
`incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` (new),
|
||||
`incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md` (status → **`resolved`**,
|
||||
Resolution section + answers to its open questions 1–3 added), `systems/scheduled-tasks.md`,
|
||||
`systems/browser-backend.md`, `systems/skills-index.md` (stocktake output, committed),
|
||||
`raw/stocktake/freshness-20261102.txt` (stocktake output, committed), `index.md`, `log.md`.
|
||||
**Nothing removed or archived:** no job retired, no system obsolete, no decision reversed in the window.
|
||||
|
||||
### Still open (carried + new)
|
||||
|
||||
The single-node no-failover topology behind every model-calling job; the GitHub Trending Radar delivery
|
||||
token; Camofox containment under multi-tab load; the Conduit health store with no retention; the
|
||||
unreconciled admission files `~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json`; the
|
||||
unidentified `-15` signal source; and the **30 s worker-ack window vs this host's cold-start time**.
|
||||
**Plus new:** whether 2026-11-02's **164 deduped Job Radar candidates are recoverable**, whether a
|
||||
cap-hit should keep being recorded as `completed`, and why the 03:00 stocktake now takes ~9 h 45 m.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user