audit: 2026-10-26 nightly — planned restart orphaned a live cron run (new incident page); 503 burst fully closed; radar cover-letter PDF pipeline moved to Chromium-over-HTML
This commit is contained in:
+32
-5
@@ -3,18 +3,45 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-25
|
||||
verified_on: 2026-10-25
|
||||
review_after: 2026-11-01
|
||||
updated: 2026-10-27
|
||||
verified_on: 2026-10-27
|
||||
review_after: 2026-11-03
|
||||
confidence: high
|
||||
tags: [system]
|
||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-10-25
|
||||
Updated: 2026-10-27
|
||||
|
||||
## Recent Changes (2026-10-25, nightly audit; live probes 2026-10-25 00:1x–00:5x AWST)
|
||||
## Recent Changes (2026-10-27, nightly audit; live probes 2026-10-27 00:1x–00:5x AWST)
|
||||
|
||||
- **A *planned* gateway restart still orphaned a live cron run — and this time it alerted loudly.** A planned
|
||||
`SIGTERM` at **2026-10-26 14:33:35** produced a shutdown drain that reported **`timed_out=True` after
|
||||
just 1.35 s** of its 30 s budget and interrupted `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot)
|
||||
**2 h 34 m** in. It closed **`failed`** on a **new** string — `lost its durable fire claim ownership` —
|
||||
against the 2026-10-20 orphan's silent `unknown`, and raised **two** `cron_incidents` rows (14:33:37
|
||||
*unverified / reconcile admission*, 14:35:59 *fire claim*) that are still `alerted`; the failure **was
|
||||
delivered** at 14:35:59. Two Mnemosyne Watchdog reds followed and **self-resolved at 15:16:12**. Window
|
||||
2026-10-26 settled **169 / 3** and **all three failures are this one fault** — so the fleet's failure
|
||||
count is one interruption, not a systemic turn. Fleet unchanged at **28 jobs (25 enabled / 3 disabled)**;
|
||||
**non-clean `last_status` 5 → 4**; **unclosed incident rows 2 → 3**; **pins unchanged**. →
|
||||
new page [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]
|
||||
- **One incident row closed, and the 2026-10-16 503 burst is fully shut.** `c0a9fa_764558b5ae83`
|
||||
(GitHub Trending Radar) **auto-closed 09:13:00** when its 09:00 slot ran clean — the last open row of the
|
||||
resource-pressure fault → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. No execution row
|
||||
carries the 503 string on 10-25 or 10-26. The job stays **`monitoring`**: single-node exposure unchanged.
|
||||
- **Two quiet degradations worth naming.** `fad5a86979d4`'s 03:08 slot **completed but delivered
|
||||
`delivery_failed`** — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead
|
||||
thread at 14:33:37); and the **radar's 07:00 slot closed `completed` at 09:10 while delivering
|
||||
“RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters, batch finished by
|
||||
hand. That morning the cover-letter PDF pipeline moved to **Chromium over the canonical HTML** with model
|
||||
**`deepseek-paid-smart`** and a **600 s** dashboard timeout → [[runbooks/job-radar]]
|
||||
- **Also updated:** `incidents/2026-10-26-planned-restart-orphaned-cron-run.md` (new),
|
||||
`incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md` (recurrence), `incidents/2026-10-16-omniroute-503-resource-pressure.md`,
|
||||
`runbooks/job-radar.md`, `systems/scheduled-tasks.md`, `index.md`, `log.md`. **Nothing removed or archived.**
|
||||
|
||||
## Recent Changes (2026-10-25, nightly audit; live probes 2026-10-25 00:1x–00:5x AWST) *(superseded by the 2026-10-27 block above)*
|
||||
|
||||
- **A third consecutive clean cron day — and an off-cron fault that is not the fleet's.** Window
|
||||
2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at
|
||||
|
||||
@@ -3,9 +3,9 @@ title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bu
|
||||
type: incident
|
||||
status: monitoring
|
||||
created: 2026-10-18
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
review_after: 2026-10-29
|
||||
updated: 2026-10-27
|
||||
verified_on: 2026-10-27
|
||||
review_after: 2026-11-03
|
||||
confidence: high
|
||||
tags: [cron, router, provider, infrastructure]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
|
||||
@@ -28,6 +28,13 @@ days** (10-16 → 10-17 and 10-19) with an identical signature on an identical n
|
||||
> most of the fleet. Two bursts four days apart, both self-clearing, is the pattern to watch; this is a standing
|
||||
> risk, not a resolved incident.
|
||||
|
||||
> **2026-10-26 update — every incident row from this failure mode is now closed.** The last open row,
|
||||
> `c0a9fa_764558b5ae83` (`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.
|
||||
> Retry shortly.`, job `c0a9fa314542` GitHub Trending Radar, `alerted` since 2026-10-19 09:02), **auto-closed
|
||||
> 2026-10-26 09:13:00** when that job's 09:00 slot ran clean. **No execution row carries the resource-pressure
|
||||
> string on 2026-10-25 or 2026-10-26.** Status stays **`monitoring`** for the same reason as 10-21: the rows are
|
||||
> shut, the single-node exposure is not.
|
||||
|
||||
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
|
||||
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
|
||||
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned (status
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-10-21
|
||||
updated: 2026-10-21
|
||||
verified_on: 2026-10-21
|
||||
review_after: 2026-10-28
|
||||
updated: 2026-10-27
|
||||
verified_on: 2026-10-27
|
||||
review_after: 2026-11-03
|
||||
confidence: high
|
||||
tags: [gateway, cron, scheduler, lifecycle, restart, silent-failure]
|
||||
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
|
||||
@@ -87,6 +87,7 @@ sqlite3 ~/.hermes/.hermes/cron/deliveries.db \
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **recurrence on 2026-10-26, and the loud-orphan gap here is now partly closed.** This one was a **planned `SIGTERM`** (not an unclean exit) and the drain still **timed out after 1.35 s**, interrupting `ea79a2107e36` (Mnemosyne Health Monitor) 2 h 34 m in. Unlike this page's case it closed **`failed`** on a **new** string (`lost its durable fire claim ownership`), **raised two `cron_incidents` rows**, and **delivered** the failure — but the work still did not happen. Any triage keyed on this page's `unknown` / "owner exited before a durable terminal state" wording will miss it.
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — the other 2026-10-20 non-delivery (a canned substitute served to a user).
|
||||
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the other "the row is green, the work did not happen" case found in this audit.
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
title: Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in)
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-10-27
|
||||
updated: 2026-10-27
|
||||
verified_on: 2026-10-27
|
||||
review_after: 2026-11-03
|
||||
confidence: high
|
||||
tags: [gateway, cron, scheduler, lifecycle, restart, memory, silent-failure]
|
||||
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-probe]
|
||||
---
|
||||
|
||||
# Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in)
|
||||
|
||||
## Headline
|
||||
|
||||
On **2026-10-26 at 14:33:35 AWST** the gateway received a **planned `SIGTERM`** (`under_systemd=yes`,
|
||||
`parent_name=systemd`) — and the shutdown drain **still interrupted a running cron execution**, because
|
||||
the drain reported `timed_out=True` after only **1.35 s**. The orphaned run was
|
||||
`ea79a2107e36` **Mnemosyne Health Monitor** (12-hourly `0 */12 * * *`), which had been claimed at
|
||||
**12:01:13** and was **2 h 34 m** into its work.
|
||||
|
||||
This is the **second instance** of the orphaned-cron-run class documented for 2026-10-20
|
||||
([[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]), with three material differences:
|
||||
|
||||
| | 2026-10-20 | 2026-10-26 |
|
||||
|---|---|---|
|
||||
| Stop type | **unclean exit** (no exit path, `suspected_oom=False`) | **planned `SIGTERM`** under systemd |
|
||||
| Execution closed as | `unknown` ("owner exited before a durable terminal state") | **`failed`** with a **new** error string |
|
||||
| Incident row raised | **no** | **yes — two** |
|
||||
| Failure delivered | no | **yes**, 14:35:59 |
|
||||
|
||||
So the loud-orphan recommendation from the 10-20 page is **now partly implemented** — the run no longer
|
||||
disappears silently. What it did not do is *finish the work* or *avoid the interruption*.
|
||||
|
||||
## Timeline (all times AWST, 2026-10-26)
|
||||
|
||||
| Time | Event | Source |
|
||||
|---|---|---|
|
||||
| 12:01:13 | `ea79a2107e36` Mnemosyne Health Monitor execution claimed, `running` | `executions.db` |
|
||||
| 00:01:46 → 00:10:45 | the same job's previous (00:01) slot — **completed in 9 m** | `executions.db` |
|
||||
| 14:33:35 | `Received SIGTERM as a planned gateway stop`, `under_systemd=yes`, `parent_name=systemd`, `loadavg_1m=2.15` | `gateway.log` |
|
||||
| 14:33:37 | `Thread 34795 not found` while sending the shutdown notice (same dead thread as the birthday/Uber jobs) | `gateway.log` |
|
||||
| 14:33:37 | incident row `ea79a2_c3ab2d784ba3` opens — *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/724a69108b…json`"* (`alerted`) | `cron_incidents` |
|
||||
| 14:33:38 | `Shutdown drain: 1 in-flight cron job(s) … drain took 1.35s, timed_out=True` → `interrupting remaining work` | `gateway.log` |
|
||||
| 14:34:25 | `1 authority task(s) did not settle within 40s; left for recovery`; `Profile /home/hermes/.hermes still has live writers at shutdown; its admissions stay for recovery` | `gateway.log` |
|
||||
| 14:34:27 | `ERROR gateway.session_authority: Admitted turn 28ad127ffb2a44c0bcd17283a93aa496 failed`; cron session `local-253a96…` **paused: runtime_draining** | `gateway.log` |
|
||||
| 14:34:27 | that session finalises with `end_reason: cron_complete` (66 msgs, 39 tool calls, 127 API calls) | `state.db` |
|
||||
| 14:34:29 | `Gateway stopped (total teardown 54.06s)`; **`Skipping .clean_shutdown marker — drain timed out with interrupted agents`** | `gateway.log` |
|
||||
| 14:35:18 | next gateway life starts (`Starting Hermes Gateway...`) | `gateway.log` |
|
||||
| 14:35:59 | execution closes **`failed`**: `RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership` | `executions.db` |
|
||||
| 14:35:59 | incident row `ea79a2_a4661554a3c2` opens on that RuntimeError (`alerted`); execution `c5ac27e3…` **delivered** the failure | `cron_incidents`, `deliveries.db` |
|
||||
| 14:46:12 / 15:01:15 | Mnemosyne Watchdog `96eb95ce12f2` fails twice (`Script exited with code 1`) reacting to the log `ERROR` — rows `96eb95_0c89bd373a0a`, `96eb95_e32174134546` | `cron_incidents` |
|
||||
| 15:16:12 | both watchdog rows **`resolved`** (the error aged out of its 15-min log window); watchdog back to `ok` | `cron_incidents` |
|
||||
|
||||
## What is new or unexplained
|
||||
|
||||
1. **A planned stop is enough to orphan a run.** No crash was required. The drain had a **30 s** budget and
|
||||
reported `timed_out=True` after **1.35 s** with `cron_now=0` — i.e. the drain did **not** wait out its
|
||||
budget before taking the interrupt path. Logged verbatim; **not diagnosed**.
|
||||
2. **New terminal error string:** `lost its durable fire claim ownership`. The 10-20 orphan closed as
|
||||
`unknown` with a scheduler-side note; this one closed as `failed` from the worker's own perspective.
|
||||
Any alerting or triage that pattern-matches on the 10-20 wording will miss this one.
|
||||
3. **Two rows for one fault.** `ea79a2_a4661554a3c2` (fire claim) and `ea79a2_c3ab2d784ba3` (unverified /
|
||||
reconcile admission) both remain **`alerted`, unclosed** as of 2026-10-27 00:1x, and the job carries
|
||||
**`failure_streak: 2`** from a **single** interrupted fire.
|
||||
4. **The admission was reconciled.** `~/.hermes/cron/admissions/` no longer holds `724a69108b…json`
|
||||
(only the 00:00 / 00:01 files from the following night), so the "reconcile" step in the row's text has
|
||||
been satisfied — the rows were simply never closed.
|
||||
5. **The watchdog works.** This fault **did** alert, twice, and self-resolved — which is a genuine
|
||||
improvement over 10-20 (one wakes up via the incident ledger, not via a missing digest).
|
||||
|
||||
## Impact
|
||||
|
||||
- **One 12-hourly memory consolidation cycle did not run.** `ea79a2107e36` is the Mnemosyne
|
||||
health/consolidation job → [[systems/mnemosyne]]. The 00:01 slot completed normally; the 12:01 slot
|
||||
produced no completed work.
|
||||
- **Day tally:** 2026-10-26 settled **169 `completed` / 3 `failed`** — and **all three failures are this one
|
||||
fault** (1 Health Monitor + 2 derivative Watchdog reds). 2026-10-25 was **172 / 2**.
|
||||
- **Session cost sunk:** the interrupted run burned 66 messages / 39 tool calls / 127 API calls
|
||||
(~597 k input, 254 k output tokens) before the interrupt.
|
||||
- **Unclosed incident rows 2 → 3** (`0e753e_e7f8194a7d06` carried over from 2026-10-01, plus the two new
|
||||
`ea79a2_*` rows).
|
||||
|
||||
## Recommended follow-ups (not yet done)
|
||||
|
||||
1. Decide the drain policy for **long-running cron jobs**: this job legitimately occupies 9 m → 2 h+.
|
||||
Either exempt a cron execution from the drain deadline, or interrupt it with a durable
|
||||
*"interrupted → re-fire this slot"* marker instead of a terminal `RuntimeError`.
|
||||
2. **Close or ack the two `ea79a2_*` rows** — the admitted turn failed and the admission is gone, so they
|
||||
are historical.
|
||||
3. **Verify the drain-timeout arithmetic** (`timed_out=True` at 1.35 s against a 30 s budget) — if the
|
||||
deadline is being bypassed whenever a writer is live, every future restart can interrupt work.
|
||||
4. Confirm `failure_streak` should count the *unverified* row at all — a single interrupted fire reading
|
||||
**streak 2** inflates the job's apparent health.
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — first instance of this class
|
||||
- [[systems/scheduled-tasks]] — fleet ledger and per-window tallies
|
||||
- [[systems/mnemosyne]] — the interrupted job's purpose and settings
|
||||
- [[systems/hermes-agent]] — gateway lifecycle and restart history
|
||||
@@ -1408,3 +1408,49 @@ The two top-level **stale** files are `current-state.md` and
|
||||
`reports/communities-wa-social-media-audit-2026.md` — both dated historical records
|
||||
rather than drift. The vault repo is **local-only** (no remotes); it was committed,
|
||||
not pushed.
|
||||
|
||||
## [2026-10-27] audit | Nightly wiki audit — window 2026-10-26 00:00 → 2026-10-27 00:5x AWST: a planned restart orphaned a live cron run
|
||||
|
||||
- **A *planned* gateway restart orphaned a cron run — the second instance of the 2026-10-20 orphan class, and
|
||||
this time it was loud.** `gateway.log`: `Received SIGTERM as a planned gateway stop` at **14:33:35**
|
||||
(`under_systemd=yes`, `parent_name=systemd`, `loadavg_1m=2.15`). The drain then logged
|
||||
`Shutdown drain: 1 in-flight cron job(s) … drain took 1.35s, timed_out=True` → `interrupting remaining work`,
|
||||
and at 14:34:29 `Skipping .clean_shutdown marker — drain timed out with interrupted agents`. The victim was
|
||||
**`ea79a2107e36` (Mnemosyne Health Monitor)**, claimed **12:01:13** — **2 h 34 m** in. It closed **14:35:59
|
||||
`failed`** on a string this wiki had not seen: **`RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its
|
||||
durable fire claim ownership`**. Unlike the 10-20 orphan it raised **two `cron_incidents` rows** (14:33:37
|
||||
`ea79a2_c3ab2d784ba3` *"Cron execution unverified; reconcile `admissions/724a69108b…json`"*; 14:35:59
|
||||
`ea79a2_a4661554a3c2`, the RuntimeError) and **delivered** the failure. → new
|
||||
`incidents/2026-10-26-planned-restart-orphaned-cron-run.md`
|
||||
- **The Mnemosyne Watchdog did its job.** `96eb95ce12f2` red-flagged twice (14:46:12, 15:01:15,
|
||||
`Script exited with code 1`) reacting to the same log `ERROR`, then **self-resolved at 15:16:12** when the
|
||||
line aged out of its 15-minute window. Both rows (`96eb95_0c89bd373a0a`, `96eb95_e32174134546`) are closed.
|
||||
- **Window tally and fleet:** 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are
|
||||
this one interruption** (Health Monitor + the two watchdog reds); 2026-10-25 settled **172 / 2**. Fleet
|
||||
unchanged: **28 jobs, 25 enabled / 3 disabled**. **Non-clean `last_status` 5 → 4** (`c0a9fa314542` and
|
||||
`08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as `error`, streak 2). **Unclosed incident rows 2 → 3**.
|
||||
**Pins unchanged.** One gateway start (14:35:18, teardown 54.06 s). **17 sessions in window** (15 `cron`,
|
||||
1 `ios`, 1 `telegram`).
|
||||
- **One incident row closed — the 2026-10-16 503 burst is fully shut.** `c0a9fa_764558b5ae83` **auto-closed
|
||||
09:13:00** when the radar's 09:00 slot ran clean; no execution row carries the resource-pressure string on
|
||||
10-25 or 10-26. Status stays `monitoring` (the single-node exposure is unchanged). Updated
|
||||
`incidents/2026-10-16-omniroute-503-resource-pressure.md`.
|
||||
- **⚠ Two quiet degradations: a completed run that delivered nothing useful, and a broken delivery thread.**
|
||||
(a) The radar's 07:00 slot closed **`completed` at 09:10**, but its delivered text read **“RUN INCOMPLETE —
|
||||
tool-call budget exhausted”** — no Notion writes, no cover letters — so the day's backlog was executed by hand.
|
||||
(b) `fad5a86979d4`'s 03:08 slot completed but ended `delivery_failed`: **`Thread 34795 not found`** — the same
|
||||
dead thread the gateway's own 14:33:37 shutdown notice hit. → `runbooks/job-radar.md`
|
||||
- **The radar's cover-letter pipeline changed the same morning.** The PDF is now printed from the **canonical
|
||||
rendered HTML** via Chromium (`build_cover_letter_pdf_from_html.py`, created 10:37) instead of ReportLab, so a
|
||||
download matches the on-screen letter; the generation default moved to **`deepseek-paid-smart`** (10:47) and
|
||||
`job_dashboard_server.py`'s regenerate timeout went **120 s → 600 s** (10:48, server restarted).
|
||||
→ `runbooks/job-radar.md`
|
||||
- **Files:** new `incidents/2026-10-26-planned-restart-orphaned-cron-run.md`; updated
|
||||
`incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md`,
|
||||
`incidents/2026-10-16-omniroute-503-resource-pressure.md`, `runbooks/job-radar.md`,
|
||||
`systems/scheduled-tasks.md`, `current-state.md`, `index.md`, `log.md`. **Nothing removed or archived.**
|
||||
No secrets written.
|
||||
- **Unresolved:** why a *planned* stop bypasses the drain deadline (`timed_out=True` at 1.35 s of a 30 s budget);
|
||||
whether `failure_streak` should count the *unverified* row (one interruption currently reads streak 2); and the
|
||||
**`Thread 34795`** delivery target needs re-pointing, or the Monday 03:08 digest will keep falling back to the
|
||||
unthreaded chat.
|
||||
|
||||
+33
-3
@@ -3,9 +3,9 @@ title: Daily Marketing Job Radar Operations and Recovery
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-27
|
||||
updated: 2026-10-13
|
||||
review_after: 2026-10-25
|
||||
verified_on: 2026-10-13
|
||||
updated: 2026-10-27
|
||||
review_after: 2026-11-10
|
||||
verified_on: 2026-10-27
|
||||
version_applies_to: Hermes Agent cron dca8482e4f76
|
||||
confidence: high
|
||||
tags: [automation, cron, jobs, notion, dashboard, runbook]
|
||||
@@ -262,6 +262,36 @@ hermes cron run dca8482e4f76 # reproduce and capture the 403
|
||||
|
||||
Do **not** allow a silent fallback — the radar must surface a model failure rather than emit an empty "no jobs found" digest.
|
||||
|
||||
## Cover-letter PDF pipeline — printed from the dashboard HTML (changed 2026-10-26)
|
||||
|
||||
The PDF is now produced from the **canonical rendered HTML**, so a download matches the letter on screen.
|
||||
Previously ReportLab (`build_cover_letter_pdf.py`) laid the letter out independently of the template and
|
||||
produced a visibly different document — Anthony rejected exactly that mismatch.
|
||||
|
||||
- **Renderer:** `~/.hermes/scripts/build_cover_letter_pdf_from_html.py` (created 2026-10-26 10:37) drives
|
||||
**Chromium `--print-to-pdf`** over the canonical HTML and writes the PDF into
|
||||
`~/.hermes/radar-ref/drafts/`. **Do not go back to ReportLab.**
|
||||
- **Model:** the cover-letter generation default moved from the `cover-letter-writer` combo to
|
||||
**`deepseek-paid-smart`** on 2026-10-26 10:47 (`regenerate_cover_letter.py`, env override `CL_MODEL`).
|
||||
- **Dashboard timeout:** `job_dashboard_server.py` must allow **`timeout=600`** on the regenerate
|
||||
subprocess (set 2026-10-26 10:48, up from 120 s). One run is an LLM call + DOCX assembly + a Chromium
|
||||
print; it can exceed two minutes. The dashboard server was restarted to pick the change up.
|
||||
- All five artefacts (`.txt .json .html .docx .pdf`) must still exist and be nonempty, and the HTML must
|
||||
contain no unresolved `{{...}}` tokens — the HTML is now the source of truth the PDF follows.
|
||||
|
||||
### 2026-10-26 07:00 slot — `RUN INCOMPLETE`, and why the backlog was finished by hand
|
||||
|
||||
The 07:00 run started on schedule and closed **`completed` at 09:10**, but the report it delivered read
|
||||
**“RUN INCOMPLETE — tool-call budget exhausted”**: discovery and scoring consumed the run's tool budget, so
|
||||
the **Notion writes and the entire cover-letter stage never executed**. The same-day backlog was then run
|
||||
interactively (rows written for the outstanding strong matches, cover letters generated and verified).
|
||||
|
||||
- **A `completed` execution row is not a completed radar run.** Read the delivered text for the
|
||||
`RUN INCOMPLETE` marker before treating a slot as done.
|
||||
- Budget the browser-heavy stages (one tab in flight, close before opening the next — see the
|
||||
2026-10-12 hazard below) and keep headroom for the Notion + cover-letter writes, which sit last in the
|
||||
pipeline.
|
||||
|
||||
## Browser-layer hazard inside the run (2026-10-12)
|
||||
|
||||
This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
review_after: 2026-10-31
|
||||
updated: 2026-10-27
|
||||
verified_on: 2026-10-27
|
||||
review_after: 2026-11-03
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-10-26 00:1x AWST (live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`).
|
||||
- **Current as of 2026-10-27 00:1x AWST (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`).
|
||||
- **Current as of 2026-10-26 00:1x AWST** *(superseded by the 2026-10-27 entry above)* **(live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`).
|
||||
- **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page. *(Superseded by the 2026-10-26 entry above.)*
|
||||
- **Current as of 2026-10-24 00:2x AWST *(superseded by the 2026-10-25 entry above)* (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]]
|
||||
- **Current as of 2026-10-23 00:1x AWST *(superseded by the 2026-10-24 entry above)* (`jobs.json.updated_at 2026-10-23T00:07:25+08:00`) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. **A DB recount by `started_at` and by `claimed_at` puts 2026-10-21 at 168 / 1**, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Five incidents closed in-window and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (`0e519874d280`, closed 08:01:40), `06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so **unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`: `1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, `c0a9fa_764558b5ae83`, `0e753e_e7f8194a7d06`). **Non-clean `last_status` falls 9 → 5:** `1b23c042e17d` (`error`, streak 2 — **next slot Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` (`delivery_failed` — Mon 10-26 03:00), `066bb5191280` (`delivery_failed` — 11-01). All four cleared rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its first slot of 10-22. **Pins:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (`provider`/`model` null → global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). ⚠ **Stocktake anomaly:** `1a8e90ea2271` ran **03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, finished `completed` with delivery `suppressed` — and **committed nothing**. It left its regenerated `systems/skills-index.md` uncommitted (mtime 10-22 03:00, with a `## .Archive (41)` section reading **347**) and its diagnosed fixes unapplied → [[incidents/2026-08-15-skills-index-drift]]. **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop` (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models `mimo-v2.5`, `or-inclusionai_ling-2_6-1t`, `minimax/minimax-m3.1-flash`), 2 `acp` (0 messages), 1 `oneshot` memory-capture probe. → [[systems/skills-index]], [[incidents/2026-08-15-skills-index-drift]]
|
||||
|
||||
Reference in New Issue
Block a user