audit: 2026-10-28 nightly — radar budget overrun now a recorded failure (new incident page); radar split into two cron jobs; tailscale peer table re-verified; TTS provider corrected to the Mac mini speech server

Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76
(Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed
failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before
Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and
cover letters never ran (Notion DB held 451 entries, 0 added). Incident row
dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified).

Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5,
deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5,
topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3).

Also: the two incident rows the 2026-10-26 planned restart raised auto-closed
00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean
last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never
committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of
108) — committed here and index.md corrected; tailscale.md peer table rebuilt from
tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo
CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to
provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained
the voice pipeline section.

New: incidents/2026-10-27-job-radar-budget-exhausted.md
Nothing removed or archived. No secrets.
This commit is contained in:
Tony0410
2026-10-28 00:20:59 +08:00
parent a2dd0bd650
commit f880ba981d
13 changed files with 417 additions and 43 deletions
+1
View File
@@ -3,3 +3,4 @@ public/
package-lock.json package-lock.json
package.json package.json
*.log *.log
.audit-report-*.txt
+34 -3
View File
@@ -3,8 +3,8 @@ title: Current State
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-27 updated: 2026-10-28
verified_on: 2026-10-27 verified_on: 2026-10-28
review_after: 2026-11-03 review_after: 2026-11-03
confidence: high confidence: high
tags: [system] tags: [system]
@@ -12,7 +12,38 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
--- ---
# Current State # Current State
Updated: 2026-10-27 Updated: 2026-10-28
## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 00:2x AWST)
- **The radar's silent budget overrun became a recorded failure — and the radar is now two jobs.** Window
2026-10-27 settled **172 `completed` / 1 `failed`**. The failure is `dca8482e4f76` (Daily Marketing Job
Radar - Discovery): **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed **`failed`**, delivery `delivered`,
and the delivered report opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* —
discovery, verification, dedup and classification completed; **Notion ingestion, cover-letter generation and
the final cover-letter audit did not execute**. The prior day (10-26) had the same overrun but closed
`completed`, hiding it in a green row. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted`
11:03:59, **still open** at this audit (`failure_type: auth` — misclassified). → new page
[[incidents/2026-10-27-job-radar-budget-exhausted]]
- **Response: the radar was split into a two-stage pipeline the same day.** Stage 1 `dca8482e4f76` — Discovery
(`0 7 * * 1-5`, `deliver: local`, failures → topic 1030) writes a handoff and triggers stage 2; new stage 2
`5fc292ff0f3b` — Daily Marketing Job Radar - Write and Letters (`30 11,14 * * 1-5`, Telegram topic 1030,
created **2026-10-27T17:34:15**) lands the Notion entries, cover letters and audit. Two test fires of stage 2
on 10-27 (17:35:00 → 17:36:52, 17:38:38 → 17:43:42) both `completed` with delivery `suppressed`; the first
real 07:00 → 11:30 cycle had not yet run. **Fleet 28 → 29 jobs (26 / 3).** → [[runbooks/job-radar]]
- **The 2026-10-26 orphan class did not recur, and its two incident rows closed.** `ea79a2_a4661554a3c2` and
`ea79a2_c3ab2d784ba3` **auto-closed 00:38:32** (unclosed rows **3 → 2**, the 2 being the Facebook birthday
monthly scrape from 10-01 and the new radar row). No 10-27 run besides the radar's failed, so the watchdog
and every other job were clean all day.
- **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error`
(streak 1) and `08eb86f02d37` re-joined as `delivery_queued`. Still non-clean: `0e753e9720fd` (error),
`fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.**
- **Also updated:** `incidents/2026-10-27-job-radar-budget-exhausted.md` (new),
`incidents/2026-10-26-planned-restart-orphaned-cron-run.md` (recurrence check), `runbooks/job-radar.md`,
`systems/scheduled-tasks.md`, `systems/hermes-agent.md` (TTS provider correction),
`systems/conduit.md` (voice pipeline), `infrastructure/tailscale.md` (peer table re-verified),
`systems/skills-index.md` (regenerated 10-27, committed), `index.md`, `log.md`.
**Nothing removed or archived.** No secrets written.
## Recent Changes (2026-10-27, nightly audit; live probes 2026-10-27 00:1x–00:5x AWST) ## Recent Changes (2026-10-27, nightly audit; live probes 2026-10-27 00:1x–00:5x AWST)
@@ -3,7 +3,7 @@ title: Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemos
type: incident type: incident
status: active status: active
created: 2026-10-27 created: 2026-10-27
updated: 2026-10-27 updated: 2026-10-28
verified_on: 2026-10-27 verified_on: 2026-10-27
review_after: 2026-11-03 review_after: 2026-11-03
confidence: high confidence: high
@@ -101,3 +101,13 @@ disappears silently. What it did not do is *finish the work* or *avoid the inter
- [[systems/scheduled-tasks]] — fleet ledger and per-window tallies - [[systems/scheduled-tasks]] — fleet ledger and per-window tallies
- [[systems/mnemosyne]] — the interrupted job's purpose and settings - [[systems/mnemosyne]] — the interrupted job's purpose and settings
- [[systems/hermes-agent]] — gateway lifecycle and restart history - [[systems/hermes-agent]] — gateway lifecycle and restart history
## Update — 2026-10-28 (nightly audit)
- **No recurrence.** Every run in the 2026-10-27 window except the radar's budget overrun completed, and the
watchdog logged no alert for the day.
- **Both incident rows this fault raised closed automatically** — `ea79a2_a4661554a3c2` and
`ea79a2_c3ab2d784ba3` **auto-closed 2026-10-27T00:38:32+08:00** (unclosed rows fleet-wide fell 3 → 2).
- The `lost its durable fire claim ownership` string did not appear in any 10-27 execution row.
- Open question from the original write-up still stands: why a *planned* stop bypasses the drain deadline
(`timed_out=True` at 1.35 s of a 30 s budget).
@@ -0,0 +1,92 @@
---
title: Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response
type: incident
status: active
created: 2026-10-28
updated: 2026-10-28
verified_on: 2026-10-28
review_after: 2026-11-04
confidence: high
tags: [automation, cron, jobs, notion, silent-failure, scheduling]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-notion-api]
---
# Job Radar Stage 1 Exhausted Its Tool-Call Budget on 2026-10-27 — Notion Writes and Cover Letters Never Ran, and the Radar Was Split in Response
## Headline
On **2026-10-27** the Daily Marketing Job Radar (`dca8482e4f76`) ran **07:01:08 → 11:03:54 AWST**
(**4 h 2 m 46 s** wall clock), closed **`failed`**, and delivered a report that opens:
> **Daily Marketing Job Radar — run 2026-10-27 — INCOMPLETE (tool-call budget exhausted before Notion write stage)**
Discovery, verification, dedup and classification completed. **Notion ingestion, cover-letter generation and the
final cover-letter audit were not executed** — so the run is recorded as failed rather than successful.
Self-reported run time **10:47:57**; freshness cutoff **2026-10-20**.
This is the second consecutive budget overrun: the **2026-10-26** 07:00 slot ran 07:01:08 → 09:10:27
(**2 h 9 m 19 s**), closed **`completed`**, and *still* delivered the same `RUN INCOMPLETE` marker — the day's
backlog had to be finished by hand. 2026-10-27 is the first overrun recorded as **`failed`**, which is the
improvement: the failure reached the incident ledger instead of hiding inside a green row.
## Ledger state
| Field | Value |
|---|---|
| Execution row | `dca8482e4f76`, started `2026-10-27T07:01:08.265913+08:00`, finished `2026-10-27T11:03:54.143535+08:00`, `failed` |
| Delivery | `delivered` (the incomplete report reached Telegram topic 1030) |
| Incident row | `dca848_e93a538ade6b` — `state: alerted`, `failure_type: auth`, first/last seen `2026-10-27T11:03:53.715494+08:00`, alerted `11:03:59.013115` |
| Incident status | **open at the 2026-10-28 00:2x audit** — no ack, no close |
| `jobs.json` | `last_status: error`, `failure_streak: 1`, pin unchanged at `custom:omni-101` / `smart-route` |
⚠ `failure_type: auth` does **not** describe this fault. A tool-call budget exhaustion is being classified as an
auth failure; the field is not trustworthy as a diagnosis for this job.
## What the run actually did (from its own report)
- Live Notion DB `0f90ba2b-8b10-4d02-a62c-6f692d5b1168`: **451 entries**, `Date Added = 2026-10-27` → **0**.
- Board composition at run time: **Agent — Rhino 402, Chaos 12, Manual 2, blank 35**; **Source — SEEK 295,
LinkedIn 88** (row text truncated in the ledger).
## Response, same day: the radar is now two jobs
At **2026-10-27 17:34:15** the single radar job was split into a two-stage pipeline (see
[[runbooks/job-radar]] for the full contract):
| Stage | Job | Schedule | Delivery | Job id |
|---|---|---|---|---|
| 1 — Discovery | Daily Marketing Job Radar - Discovery — `DISCOVERY → VERIFY → DEDUP → SCORE`, no Notion write, no cover letters; writes a handoff file | `0 7 * * 1-5` | `local` (failures → Telegram topic 1030) | `dca8482e4f76` (unchanged) |
| 2 — Write and Letters | Daily Marketing Job Radar - Write and Letters — lands the handoff: Notion entries, cover letters, audit, report | `30 11,14 * * 1-5` | Telegram group `-1004321904721`, topic `1030` | `5fc292ff0f3b` (created 17:34:15) |
Stage 1 keeps its id, pin (`custom:omni-101` / `smart-route`) and its own tool budget; stage 2 is a separate job
whose prompt lands the handoff rather than repeating the discovery narrative. Two test fires of stage 2 ran the same
evening — **17:35:00 → 17:36:52** and **17:38:38 → 17:43:42** — both `completed`, both with delivery
`suppressed`, `last_status: ok`, streak 0.
Fleet size went **28 → 29 jobs** (26 enabled / 3 disabled).
## Why it matters
1. **The failure mode is partial work, not a lost run.** Four hours of browser-heavy discovery produced a
correct report and zero writes. Nothing in `jobs.json` distinguished it from a good day except
`last_status: error` on 10-27 (`completed` on 10-26, with the same incompleteness buried in the text).
2. **Splitting is the mitigation, not a fix.** Stage 2 inherits its own tool budget; if stage 1 still overruns
and never writes the handoff file, stage 2 has nothing to land — watch the 11:30 slot for that.
3. **The budget is consumed by discovery**, which is exactly the stage that now owns it alone.
## Open questions
1. **Verify one full cycle end-to-end** (stage 1 07:00 → stage 2 11:30): does stage 2 find stage 1's handoff
file, do the Notion writes and cover letters land, and is `MISSING_COVER_LETTER_COUNT=0` reported?
2. **Make budget exhaustion self-describing.** The classifier tagged it `auth`; a distinct budget/partial-run
type would put this on the same footing as any other failure.
3. **Do the two stages fit the same wall-clock envelope?** Stage 1 must finish inside its own budget *and* leave
the handoff before 11:30.
4. The incident row stays open until a clean stage-1 slot auto-closes it — confirm the 2026-10-28 07:00 slot
closes `dca848_e93a538ade6b`.
## Related
- [[runbooks/job-radar]] — the two-stage contract, recovery procedure and the 2026-10-26 overrun
- [[systems/scheduled-tasks]] — fleet ledger, job table and the 2026-10-27 window
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — the radar's previous failure mode
+5 -4
View File
File diff suppressed because one or more lines are too long
+17 -16
View File
@@ -3,8 +3,8 @@ title: Tailscale
type: infrastructure type: infrastructure
status: active status: active
created: 2026-07-26 created: 2026-07-26
updated: 2026-10-26 updated: 2026-10-28
verified_on: 2026-10-26 verified_on: 2026-10-28
confidence: high confidence: high
tags: [tailscale, vpn, networking, infrastructure] tags: [tailscale, vpn, networking, infrastructure]
--- ---
@@ -25,35 +25,36 @@ Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, seve
Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`. Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`.
Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**. Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**.
Re-verified 2026-10-21 03:0x: the 2026-10-19 table had drifted again — **`octop` (100.92.163.122), `springroll` (100.100.153.125), `openmausbot` and `openmausbot-vm` were missing from the table body** (only mentioned in prose), `omarchymac` had gone offline (online → idle/offline 3d), and every offline age was stale. Table rebuilt from `tailscale status --json`, keyed on DNS name rather than `HostName` (the JSON `HostName` for `freshrss` is `ct260`, and both `omniroute` nodes share the hostname `omniroute`). Container states cross-checked against live `pct list` on `pve` (100.96.100.82) and `m93p` (100.104.51.98): CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running. Re-verified 2026-10-21 03:0x: the 2026-10-19 table had drifted again — **`octop` (100.92.163.122), `springroll` (100.100.153.125), `openmausbot` and `openmausbot-vm` were missing from the table body** (only mentioned in prose), `omarchymac` had gone offline (online → idle/offline 3d), and every offline age was stale. Table rebuilt from `tailscale status --json`, keyed on DNS name rather than `HostName` (the JSON `HostName` for `freshrss` is `ct260`, and both `omniroute` nodes share the hostname `omniroute`). Container states cross-checked against live `pct list` on `pve` (100.96.100.82) and `m93p` (100.104.51.98): CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running.
Re-verified 2026-10-28 00:2x (nightly audit): State column rebuilt from `tailscale status --json` — **14 rows changed** (anthonys-imac: idle/offline 32d -> offline 34d; cachyos: idle/offline 52d -> offline 54d; cloudreve: idle/offline 93d — CT209 stopped -> offline 95d — CT209 stopped; conduit: idle/offline 49d -> offline 52d; elementsynapse: idle/offline 96d — CT555 stopped -> offline 98d — CT555 stopped; freshrss-new: idle/offline 69d -> offline 71d; m93p: online -> offline <1d; mattermost: idle/offline 36d — CT255 stopped -> offline 38d — CT255 stopped; omarchybook: idle/offline 4d -> offline 6d; omarchymac: idle/offline 8d -> offline 10d; omni: idle/offline 69d — CT601 removed from both hypervisors -> offline 71d — CT601 removed from both hypervisors; openmausbot: idle/offline 4d -> offline 6d; sftpgo: online — CT299 (pve) -> offline <1d — CT299 (pve); ubuntu-server-xfce: idle/offline 66d -> offline 68d). Header count corrected to **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself). Nodes in the table that are no longer in the tailnet: **none**. Live IPv4 peers absent from the table: **none**. **State transitions inside the audit window (not just age drift): `m93p` online → offline <1d, `sftpgo` (CT299) online → offline <1d.**
Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale status --json` — **`springroll-browser` (100.109.221.91) was missing entirely** and has been added. **`omarchybook` (100.107.9.100) and `openmausbot` (100.91.214.91) have both gone offline** (online → idle/offline 3d) and `openmausbot` is no longer a live peer despite its m93p LXC (CT1264) still running `tailscale` — the node itself has stopped connecting. Every offline age was refreshed (e.g. `mattermost` 31d → **35d**, `omni` 64d → **68d**, `elementsynapse` 91d → **95d**, `ubuntu-server-xfce` 60d → **64d**, `omarchymac` 3d → **7d**). `m93p` (100.104.51.98) now reads **idle/offline 13m ago** — the peer record is stale, but the host is plainly reachable (both `pct list` probes and the SSH session to CT201/CT460 during this audit succeeded), so treat it as a Tailscale-side liveness artefact, not a hypervisor outage. `homeassistant` still advertises `ExitNodeOption: true` but `ExitNode: false` — it **offers** exit-node capability; it is not currently serving as one (the old "active; offers exit node" wording overstated the live state). Container states cross-checked again against live `pct list` on both hypervisors, and each peer's container identity confirmed by running `tailscale ip -4` **inside** the candidate LXC on both hosts — the authoritative mapping, since peer names and LXC names do not always match. Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale status --json` — **`springroll-browser` (100.109.221.91) was missing entirely** and has been added. **`omarchybook` (100.107.9.100) and `openmausbot` (100.91.214.91) have both gone offline** (online → idle/offline 3d) and `openmausbot` is no longer a live peer despite its m93p LXC (CT1264) still running `tailscale` — the node itself has stopped connecting. Every offline age was refreshed (e.g. `mattermost` 31d → **35d**, `omni` 64d → **68d**, `elementsynapse` 91d → **95d**, `ubuntu-server-xfce` 60d → **64d**, `omarchymac` 3d → **7d**). `m93p` (100.104.51.98) now reads **idle/offline 13m ago** — the peer record is stale, but the host is plainly reachable (both `pct list` probes and the SSH session to CT201/CT460 during this audit succeeded), so treat it as a Tailscale-side liveness artefact, not a hypervisor outage. `homeassistant` still advertises `ExitNodeOption: true` but `ExitNode: false` — it **offers** exit-node capability; it is not currently serving as one (the old "active; offers exit node" wording overstated the live state). Container states cross-checked again against live `pct list` on both hypervisors, and each peer's container identity confirmed by running `tailscale ip -4` **inside** the candidate LXC on both hosts — the authoritative mapping, since peer names and LXC names do not always match.
| Node | IP | State | | Node | IP | State |
|---|---|---| |---|---|---|
| hermes | 100.118.5.51 | **self** — CT460 (pve) | | hermes | 100.118.5.51 | **self** — CT460 (pve) |
| agentzero | 100.106.54.94 | online — CT475 (pve) | | agentzero | 100.106.54.94 | online — CT475 (pve) |
| anthonys-imac | 100.112.123.124 | idle/offline 32d | | anthonys-imac | 100.112.123.124 | offline 34d |
| anthonys-ipad | 100.111.198.120 | online | | anthonys-ipad | 100.111.198.120 | online |
| anthonys-iphone | 100.100.116.67 | online | | anthonys-iphone | 100.100.116.67 | online |
| anthonys-mac-mini | 100.74.17.15 | online | | anthonys-mac-mini | 100.74.17.15 | online |
| cachyos | 100.124.208.15 | idle/offline 52d | | cachyos | 100.124.208.15 | offline 54d |
| clawtest | 100.122.189.81 | online — CT333 (pve) | | clawtest | 100.122.189.81 | online — CT333 (pve) |
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) | | cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
| cloudreve | 100.111.187.7 | idle/offline 93d — CT209 stopped | | cloudreve | 100.111.187.7 | offline 95d — CT209 stopped |
| conduit | 100.81.79.87 | idle/offline 49d | | conduit | 100.81.79.87 | offline 52d |
| docker | 100.96.150.69 | online — CT200 (pve) | | docker | 100.96.150.69 | online — CT200 (pve) |
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) | | docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
| elementsynapse | 100.75.250.37 | idle/offline 96d — CT555 stopped | | elementsynapse | 100.75.250.37 | offline 98d — CT555 stopped |
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) | | freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
| freshrss | 100.83.60.93 | online — CT1260 (m93p) | | freshrss | 100.83.60.93 | online — CT1260 (m93p) |
| freshrss-new | 100.110.180.115 | idle/offline 69d | | freshrss-new | 100.110.180.115 | offline 71d |
| gitea | 100.82.100.128 | online — CT221 (m93p) | | gitea | 100.82.100.128 | online — CT221 (m93p) |
| guest-vps | 100.117.14.84 | online | | guest-vps | 100.117.14.84 | online |
| hermesagent | 100.98.205.112 | online — CT450 (pve) | | hermesagent | 100.98.205.112 | online — CT450 (pve) |
| homeassistant | 100.68.139.5 | online | | homeassistant | 100.68.139.5 | online |
| litellm | 100.79.29.64 | online — CT500 (m93p) | | litellm | 100.79.29.64 | online — CT500 (m93p) |
| m93p | 100.104.51.98 | online | | m93p | 100.104.51.98 | offline <1d |
| mailflow | 100.115.60.92 | online — CT1262 (m93p) | | mailflow | 100.115.60.92 | online — CT1262 (m93p) |
| mattermost | 100.98.116.14 | idle/offline 36d — CT255 stopped | | mattermost | 100.98.116.14 | offline 38d — CT255 stopped |
| n8n | 100.84.124.30 | online | | n8n | 100.84.124.30 | online |
| nextcloud | 100.90.102.19 | online — CT270 (pve) | | nextcloud | 100.90.102.19 | online — CT270 (pve) |
| ntfy | 100.67.153.116 | online — CT1241 (m93p) | | ntfy | 100.67.153.116 | online — CT1241 (m93p) |
@@ -61,13 +62,13 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| octop | 100.92.163.122 | online | | octop | 100.92.163.122 | online |
| ollama | 100.108.44.119 | online — CT205 (pve) | | ollama | 100.108.44.119 | online — CT205 (pve) |
| omarchy | 100.73.227.103 | online | | omarchy | 100.73.227.103 | online |
| omarchybook | 100.107.9.100 | idle/offline 4d | | omarchybook | 100.107.9.100 | offline 6d |
| omarchymac | 100.84.180.69 | idle/offline 8d | | omarchymac | 100.84.180.69 | offline 10d |
| omarchymac-1 | 100.126.173.115 | online | | omarchymac-1 | 100.126.173.115 | online |
| omni | 100.93.204.4 | idle/offline 69d — CT601 removed from both hypervisors | | omni | 100.93.204.4 | offline 71d — CT601 removed from both hypervisors |
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute | | omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) | | omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
| openmausbot | 100.91.214.91 | idle/offline 4d | | openmausbot | 100.91.214.91 | offline 6d |
| openmausbot-vm | 100.77.107.30 | online | | openmausbot-vm | 100.77.107.30 | online |
| openweb | 100.69.230.13 | online — CT245 (m93p) | | openweb | 100.69.230.13 | online — CT245 (m93p) |
| paseo | 100.72.188.93 | online — CT1261 (m93p) | | paseo | 100.72.188.93 | online — CT1261 (m93p) |
@@ -76,11 +77,11 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| rclone | 100.107.116.17 | online — CT104 (m93p) | | rclone | 100.107.116.17 | online — CT104 (m93p) |
| runtipi | 100.96.244.39 | online — CT201 (pve) | | runtipi | 100.96.244.39 | online — CT201 (pve) |
| rustdesk | 100.95.227.57 | online — CT1008 (m93p) | | rustdesk | 100.95.227.57 | online — CT1008 (m93p) |
| sftpgo | 100.88.223.70 | online — CT299 (pve) | | sftpgo | 100.88.223.70 | offline <1d — CT299 (pve) |
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) | | social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
| springroll | 100.100.153.125 | online | | springroll | 100.100.153.125 | online |
| springroll-browser | 100.109.221.91 | online | | springroll-browser | 100.109.221.91 | online |
| ubuntu-server-xfce | 100.101.26.104 | idle/offline 66d | | ubuntu-server-xfce | 100.101.26.104 | offline 68d |
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) | | vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
| webserver | 100.82.40.23 | online — CT207 (pve) | | webserver | 100.82.40.23 | online — CT207 (pve) |
+52
View File
@@ -1454,3 +1454,55 @@ not pushed.
whether `failure_streak` should count the *unverified* row (one interruption currently reads streak 2); and the whether `failure_streak` should count the *unverified* row (one interruption currently reads streak 2); and the
**`Thread 34795`** delivery target needs re-pointing, or the Monday 03:08 digest will keep falling back to the **`Thread 34795`** delivery target needs re-pointing, or the Monday 03:08 digest will keep falling back to the
unthreaded chat. unthreaded chat.
## [2026-10-28] audit | Nightly wiki audit — window 2026-10-26 00:00 → 2026-10-27 00:5x AWST: a planned restart orphaned a live cron run
- **Window 2026-10-27: 172 `completed` / 1 `failed` (+ this audit `running`).** The failure is the radar's
budget overrun, and it is the story of the night → new page
[[incidents/2026-10-27-job-radar-budget-exhausted]].
- **The radar's tool-call budget was exhausted for the second consecutive day — and this time the run is
recorded as `failed`.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed` and
delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"*: discovery, verification,
dedup and classification completed; **Notion ingestion, cover-letter generation and the final cover-letter
audit never ran**. Notion DB `0f90ba2b-8b10-4d02-a62c-6f692d5b1168` held **451 entries** with **0** added
for 2026-10-27 (Agent — Rhino 402 / Chaos 12 / Manual 2 / blank 35; Source — SEEK 295 / LinkedIn 88).
Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** at this audit
(`failure_type: auth` — which does not describe a budget fault). The previous day's identical overrun closed
**`completed`** and was found only by reading the delivered text — that is why the 10-27 row matters.
- **Response, same day: the radar is now two jobs.** Stage 1 `dca8482e4f76` **Daily Marketing Job Radar -
Discovery** (`0 7 * * 1-5`, `deliver: local`, failures → Telegram topic 1030) does
`DISCOVERY → VERIFY → DEDUP → SCORE`, writes a handoff file and triggers stage 2. New stage 2
**`5fc292ff0f3b`** **Daily Marketing Job Radar - Write and Letters** (`30 11,14 * * 1-5`, Telegram topic
1030, created **2026-10-27T17:34:15**) lands the Notion entries, cover letters, missing-letter audit and
report. Two stage-2 test fires on 10-27 (**17:35:00 → 17:36:52**, **17:38:38 → 17:43:42**) both
`completed`, delivery `suppressed`, `last_status: ok`, streak 0. Pins unchanged (`custom:omni-101` /
`smart-route`). **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** → `runbooks/job-radar.md`
- **The 2026-10-26 orphan class did not recur; its two incident rows closed.** `ea79a2_a4661554a3c2` and
`ea79a2_c3ab2d784ba3` **auto-closed 2026-10-27T00:38:32+08:00** — unclosed rows fleet-wide **3 → 2** (the
2: `0e753e_e7f8194a7d06` Facebook birthday monthly scrape, open since 2026-10-01 09:00:55, and the new
`dca848_e93a538ade6b`). No 10-27 execution row carries the orphan strings and the watchdog alert log has no
10-27 entry. → `incidents/2026-10-26-planned-restart-orphaned-cron-run.md`
- **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error`
(streak 1) and `08eb86f02d37` re-joined as `delivery_queued`. Also non-clean: `0e753e9720fd` (error),
`fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed).
- **Two config corrections from live probes.** `tts.provider` is **`openai`** pointed at the **Mac mini speech
server** (`https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech`, model `phonon`, voice `alloy`) — the
page still claimed ElevenLabs from 2026-10-10 (change date not established, so the old bullet is marked
superseded rather than deleted). Live voice uses `voice.speech_server.url` =
`wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/`; `GET /v1/pool` → `{"size":2,"in_use":0}`, both
units `idle`, while `/v1/models` returns `{"detail":"Not Found"}`. `voice.auto_tts` is still `false`, and
`wake_word.enabled` is `true`. → `systems/hermes-agent.md`, `systems/conduit.md`
- **`systems/skills-index.md` was regenerated by the 2026-10-27 03:00 slot but never committed** — the
authoring tree was dirty with `updated: 2026-10-27` and a **302 skills / 69 categories** total (against
index.md's stale **306 / 70 of 2026-10-23**). Freshness at that run: **0 stale / 104 fresh / 4 no-date of
108 pages** (`freshness-20261027.txt`). Committed, and `index.md` corrected to 302.
- **`infrastructure/tailscale.md` peer table re-verified** against `tailscale status --json`; the node-count
header and any state changes are recorded in that page.
- **Files:** new `incidents/2026-10-27-job-radar-budget-exhausted.md`; updated
`incidents/2026-10-26-planned-restart-orphaned-cron-run.md`, `runbooks/job-radar.md`,
`systems/scheduled-tasks.md`, `systems/hermes-agent.md`, `systems/conduit.md`,
`infrastructure/tailscale.md`, `systems/skills-index.md`, `current-state.md`, `index.md`, `log.md`.
**Nothing removed or archived.** No secrets written.
- **Unresolved:** whether the two-stage split actually fixes the budget problem — the first real 07:00 → 11:30
cycle (stage 1 handoff → stage 2 writes) had not run at audit time; why a budget exhaustion is classified
`failure_type: auth`; and the 10-27 radar incident row will only close when a clean stage-1 slot runs.
+121
View File
@@ -0,0 +1,121 @@
# Wiki freshness audit — 2026-10-27
Threshold: >21 days since `updated`
## STALE (0)
## FRESH (104)
- [ 7d] CONTRIBUTING.md
- [ 1d] agents/hermes-chaos.md
- [ 1d] agents/hermes-production.md
- [ 7d] agents/index.md
- [ 16d] agents/nanobot.md
- [ 7d] agents/open-claw.md
- [ 16d] agents/qwenpaw.md
- [ 7d] agents/zeroclaw.md
- [ 7d] comparisons/camoufox-vs-chrome.md
- [ 7d] comparisons/hermes-vs-nanobot.md
- [ 7d] comparisons/memory-backends.md
- [ 7d] comparisons/model-routers.md
- [ 14d] comparisons/steel-vs-camoufox.md
- [ 7d] concepts/agent-memory.md
- [ 14d] concepts/browser-automation.md
- [ 7d] concepts/model-routing.md
- [ 7d] concepts/persistent-shells.md
- [ 7d] concepts/tool-calling.md
- [ 0d] current-state.md
- [ 7d] decisions/index.md
- [ 7d] decisions/keep-openrouter-as-fallback.md
- [ 7d] decisions/persistent-shell-enabled.md
- [ 7d] decisions/rollback-steel-to-camofox.md
- [ 7d] experiments/index.md
- [ 3d] incidents/2026-07-29-mnemosyne-relay.md
- [ 7d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 3d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 5d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 4d] incidents/2026-08-15-skills-index-drift.md
- [ 15d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 2d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 7d] incidents/2026-08-29-gmail-read-state.md
- [ 6d] incidents/2026-09-02-sftpgo-outage.md
- [ 16d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 21d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 1d] incidents/2026-10-03-ct450-hermes-update.md
- [ 1d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 21d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 3d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 18d] incidents/2026-10-09-memory-recall-relevance.md
- [ 11d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 0d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 8d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 0d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 5d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 2d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 1d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 1d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 1d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 0d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 5d] infrastructure/backups.md
- [ 7d] infrastructure/credentials-map.md
- [ 16d] infrastructure/docker-services.md
- [ 7d] infrastructure/domains-and-tunnels.md
- [ 1d] infrastructure/hosts.md
- [ 13d] infrastructure/network-map.md
- [ 13d] infrastructure/proxmox.md
- [ 1d] infrastructure/tailscale.md
- [ 15d] queries/retained-research-answers.md
- [ 2d] reports/communities-wa-social-media-audit-2026.md
- [ 6d] reports/uber-eats/2026-07.md
- [ 6d] reports/uber-eats/2026-08.md
- [ 6d] reports/uber-eats/2026-09.md
- [ 6d] reports/uber-eats/index.md
- [ 7d] runbooks/backup-wiki.md
- [ 7d] runbooks/diagnose-docker.md
- [ 7d] runbooks/gateway-resource-resilience.md
- [ 0d] runbooks/job-radar.md
- [ 7d] runbooks/multiplexer-setup.md
- [ 1d] runbooks/provider-health.md
- [ 7d] runbooks/recover-docker-service.md
- [ 7d] runbooks/restart-browser.md
- [ 7d] runbooks/restart-hermes.md
- [ 7d] runbooks/restore-openrouter.md
- [ 7d] runbooks/test-openrouter.md
- [ 7d] runbooks/troubleshoot-tool-loop.md
- [ 1d] runbooks/update-hermes-safely.md
- [ 14d] systems/browser-backend.md
- [ 7d] systems/cloudflare-tunnel.md
- [ 1d] systems/conduit.md
- [ 1d] systems/current-profile.md
- [ 7d] systems/guanaco.md
- [ 7d] systems/headless-server.md
- [ 1d] systems/hermes-agent.md
- [ 7d] systems/hermes-ui.md
- [ 15d] systems/home-assistant.md
- [ 2d] systems/homepage-dashboard.md
- [ 16d] systems/ignis.md
- [ 1d] systems/memory-backend.md
- [ 1d] systems/messaging-integrations.md
- [ 3d] systems/mnemosyne.md
- [ 1d] systems/model-providers.md
- [ 15d] systems/nanobot.md
- [ 7d] systems/nextcloud.md
- [ 7d] systems/obsidian.md
- [ 5d] systems/omniroute.md
- [ 7d] systems/opencode-go.md
- [ 7d] systems/openrouter.md
- [ 16d] systems/paseo.md
- [ 8d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 7d] systems/terminal-backend.md
- [ 7d] systems/tool-search.md
## NO updated DATE (4)
- SCHEMA.md
- experiments/active/example.md
- index.md
- log.md
TOTAL: 0 stale, 104 fresh, 4 no-date of 108 pages
+49 -8
View File
@@ -3,10 +3,10 @@ title: Daily Marketing Job Radar Operations and Recovery
type: runbook type: runbook
status: active status: active
created: 2026-07-27 created: 2026-07-27
updated: 2026-10-27 updated: 2026-10-28
review_after: 2026-11-10 review_after: 2026-11-10
verified_on: 2026-10-27 verified_on: 2026-10-28
version_applies_to: Hermes Agent cron dca8482e4f76 version_applies_to: Hermes Agent cron dca8482e4f76 (stage 1) + 5fc292ff0f3b (stage 2)
confidence: high confidence: high
tags: [automation, cron, jobs, notion, dashboard, runbook] tags: [automation, cron, jobs, notion, dashboard, runbook]
sources: [live-cron-output, live-notion-api, live-dashboard-api, hermes-docs] sources: [live-cron-output, live-notion-api, live-dashboard-api, hermes-docs]
@@ -22,11 +22,14 @@ Run a verified Western Australian marketing-job search, deduplicate against the
| Setting | Value | | Setting | Value |
|---|---| |---|---|
| Cron job | `dca8482e4f76` — Daily Marketing Job Radar | | Cron job — stage 1 | `dca8482e4f76` — Daily Marketing Job Radar - Discovery (Discovery → Verify → Dedup → Score; no Notion write, no cover letters) |
| Schedule | `0 7 * * 1-5` (07:00 AWST weekdays) | | Schedule — stage 1 | `0 7 * * 1-5` (07:00 AWST weekdays) |
| Delivery | Telegram group `-1004321904721`, topic `1030` | | Delivery — stage 1 | `local` (dashboard/CLI); failures → Telegram group `-1004321904721`, topic `1030` |
| Cron job — stage 2 | `5fc292ff0f3b` — Daily Marketing Job Radar - Write and Letters (created 2026-10-27 17:34:15) |
| Schedule — stage 2 | `30 11,14 * * 1-5` (11:30 and 14:30 AWST weekdays) |
| Delivery — stage 2 | Telegram group `-1004321904721`, topic `1030` |
| Skill | `job-search-automation` | | Skill | `job-search-automation` |
| Primary model | `custom:omni-101` / `gpt-5.6-terra` (upstream `command-code`) — ⚠ 403 `MODEL_NOT_IN_PLAN` since 2026-09-26 | | Primary model (both stages) | `custom:omni-101` / `smart-route` (live 2026-10-28) — the 2026-09-26 `gpt-5.6-terra` 403 is historical |
| First fallback | `opencode-go / mimo-v2.5` | | First fallback | `opencode-go / mimo-v2.5` |
| Dashboard | `https://hermes.kangaroo-eel.ts.net/jobs/` | | Dashboard | `https://hermes.kangaroo-eel.ts.net/jobs/` |
| Dashboard backend | `127.0.0.1:9099` | | Dashboard backend | `127.0.0.1:9099` |
@@ -279,6 +282,17 @@ produced a visibly different document — Anthony rejected exactly that mismatch
- All five artefacts (`.txt .json .html .docx .pdf`) must still exist and be nonempty, and the HTML must - All five artefacts (`.txt .json .html .docx .pdf`) must still exist and be nonempty, and the HTML must
contain no unresolved `{{...}}` tokens — the HTML is now the source of truth the PDF follows. contain no unresolved `{{...}}` tokens — the HTML is now the source of truth the PDF follows.
### 2026-10-27 07:00 slot — the same overrun, now recorded as `failed` (and the trigger for the split)
The 07:00 slot ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**) and closed **`failed`** on delivery
`delivered`. The report it sent opens *"Daily Marketing Job Radar — run 2026-10-27 — INCOMPLETE (tool-call
budget exhausted before Notion write stage)"*: discovery, verification, dedup and classification finished;
**Notion writes, cover letters and the final audit did not run**. Notion DB
`0f90ba2b-8b10-4d02-a62c-6f692d5b1168` held **451 entries** with **0** added for 2026-10-27 at run time.
Incident row `dca848_e93a538ade6b` opened 11:03:53 (`alerted` 11:03:59) and was **still open** at the
2026-10-28 audit. The split above is the response — see
[[incidents/2026-10-27-job-radar-budget-exhausted]] for the full ledger.
### 2026-10-26 07:00 slot — `RUN INCOMPLETE`, and why the backlog was finished by hand ### 2026-10-26 07:00 slot — `RUN INCOMPLETE`, and why the backlog was finished by hand
The 07:00 run started on schedule and closed **`completed` at 09:10**, but the report it delivered read The 07:00 run started on schedule and closed **`completed` at 09:10**, but the report it delivered read
@@ -287,11 +301,38 @@ the **Notion writes and the entire cover-letter stage never executed**. The same
interactively (rows written for the outstanding strong matches, cover letters generated and verified). interactively (rows written for the outstanding strong matches, cover letters generated and verified).
- **A `completed` execution row is not a completed radar run.** Read the delivered text for the - **A `completed` execution row is not a completed radar run.** Read the delivered text for the
`RUN INCOMPLETE` marker before treating a slot as done. `RUN INCOMPLETE` marker before treating a slot as done. A *failed* row with a delivered report is the honest
version of the same fault — the 2026-10-27 slot is the first time it reached the incident ledger.
- Budget the browser-heavy stages (one tab in flight, close before opening the next — see the - Budget the browser-heavy stages (one tab in flight, close before opening the next — see the
2026-10-12 hazard below) and keep headroom for the Notion + cover-letter writes, which sit last in the 2026-10-12 hazard below) and keep headroom for the Notion + cover-letter writes, which sit last in the
pipeline. pipeline.
## Two-stage split (changed 2026-10-27)
The single-job radar was split after two consecutive budget overruns (2026-10-26 `completed` but
`RUN INCOMPLETE`, 2026-10-27 closed `failed` — see
[[incidents/2026-10-27-job-radar-budget-exhausted]]). Discovery is the stage that eats the tool budget, so it
now owns a job of its own and the writes own a second one.
| Stage | Job id | Schedule | Writes | Delivers |
|---|---|---|---|---|
| 1 — Discovery | `dca8482e4f76` (existing id) | `0 7 * * 1-5` | a handoff file only | `local`; failures → topic 1030 |
| 2 — Write and Letters | `5fc292ff0f3b` (new) | `30 11,14 * * 1-5` | Notion entries, cover letters, final audit, report | Telegram topic 1030 |
- **Stage 1** does `DISCOVERY → VERIFY → DEDUP → SCORE`, writes the handoff file, and triggers stage 2. It must
**not** write to Notion and must **not** generate cover letters.
- **Stage 2** lands the handoff: Notion entries, cover letters, the missing-letter audit and the report. Its
prompt is not a copy of stage 1's — it reports what it wrote, not what it found.
- Stage 1 keeps the `custom:omni-101` / `smart-route` pin. Both jobs still inherit the profile-level
`fallback_providers` chain; Hermes stores no per-job fallback list.
- First fires: two test runs of stage 2 on 2026-10-27 (17:35:00 → 17:36:52 and 17:38:38 → 17:43:42), both
`completed` with delivery `suppressed`. **The first real 07:00 → 11:30 cycle had not yet run at the
2026-10-28 audit.**
**Recovery if stage 2 lands nothing:** check that stage 1 actually wrote its handoff file before running stage 2
by hand (`hermes cron run 5fc292ff0f3b`). A stage-2 run with no handoff is a no-op, not a failed search — do not
re-run stage 1's discovery just to satisfy it.
## Browser-layer hazard inside the run (2026-10-12) ## Browser-layer hazard inside the run (2026-10-12)
This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove This job is the most browser-heavy cron on the estate, and on **2026-10-12 (07:01 → 07:33)** it drove
+15 -2
View File
@@ -3,8 +3,8 @@ title: Conduit
type: system type: system
status: active status: active
created: 2026-10-24 created: 2026-10-24
updated: 2026-10-26 updated: 2026-10-28
verified_on: 2026-10-26 verified_on: 2026-10-28
review_after: 2026-11-08 review_after: 2026-11-08
confidence: high confidence: high
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling] tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling]
@@ -90,6 +90,19 @@ the call rings when the asking turn ends and plays the reason first. Limits: `mi
asserted the capability did not exist **without calling the tool or loading the skill** → asserted the capability did not exist **without calling the tool or loading the skill** →
[[incidents/2026-10-24-conduit-call-request-refused]]. [[incidents/2026-10-24-conduit-call-request-refused]].
## Voice pipeline (probed 2026-10-28)
Phone voice sessions ride the Mac mini speech server, which is also the Hermes TTS target:
- `voice.speech_server.url` = `wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/` — live-voice transport,
`wake_word.enabled: true`.
- `tts.provider: openai` → `https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech` (model `phonon`,
voice `alloy`) — see [[systems/hermes-agent]].
- **Health check:** `GET https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/pool` returned
`{"size": 2, "in_use": 0}` with both units `idle` on 2026-10-28 00:2x — a **2-unit pool**. `/v1/models` is
**not implemented** (`{"detail":"Not Found"}`); probe `/v1/pool`, not `/v1/models`.
- `voice.auto_tts: false` — assistant replies are not auto-spoken; audio appears only when requested.
## Memory integration ## Memory integration
The voice session (`conduit-voice`) reaches Mnemosyne through the plugin's own memory layer, not The voice session (`conduit-voice`) reaches Mnemosyne through the plugin's own memory layer, not
through Mnemosyne's `system_prompt_block()` — recall is wrapped in a per-request deadline through Mnemosyne's `system_prompt_block()` — recall is wrapped in a per-request deadline
+11 -2
View File
@@ -3,8 +3,8 @@ title: Hermes Agent
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-26 updated: 2026-10-28
verified_on: 2026-10-26 verified_on: 2026-10-28
confidence: high confidence: high
tags: [system, hermes, agent, llm] tags: [system, hermes, agent, llm]
sources: [live-hermes-version, live-config, live-skill-inventory] sources: [live-hermes-version, live-config, live-skill-inventory]
@@ -166,6 +166,15 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
- **TTS:** ElevenLabs is the configured provider; its package was missing from the live managed env - **TTS:** ElevenLabs is the configured provider; its package was missing from the live managed env
and was reinstalled 2026-10-10. `voice.auto_tts: false` - auto-speak is off by design, so TTS and was reinstalled 2026-10-10. `voice.auto_tts: false` - auto-speak is off by design, so TTS
output appears only when explicitly requested. output appears only when explicitly requested.
- ⚠ **Superseded — live config checked 2026-10-28** (change date not established): `tts.provider` is
**`openai`** pointed at the **Mac mini speech server** — `tts.providers.openai.base_url` =
`https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech`, `model: phonon`, `voice: alloy`. Live voice
(as opposed to one-shot TTS) runs over `voice.speech_server.url` =
`wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/`, with `wake_word.enabled: true`. A `command`-type
provider `ollama_local` (`~/.hermes/scripts/ollama-tts.sh`) remains configured as an alternative.
`tts.elevenlabs` (`eleven_flash_v2_5`) and `tts.gemini` (`gemini-2.5-flash-preview-tts`, voice `Charon`,
persona file `~/.hermes/hermes-tts-persona.txt`) are still present as provider blocks. `voice.auto_tts`
is still `false`. See [[systems/conduit]] for the voice path.
- **Mattermost platform unreachable** since 2026-10-10; the gateway keeps retrying on a 5-minute - **Mattermost platform unreachable** since 2026-10-10; the gateway keeps retrying on a 5-minute
cadence - [[systems/messaging-integrations]]. cadence - [[systems/messaging-integrations]].
- **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]]. - **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]].
+7 -5
View File
@@ -3,8 +3,8 @@ title: Scheduled Tasks
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-27 updated: 2026-10-28
verified_on: 2026-10-27 verified_on: 2026-10-28
review_after: 2026-11-03 review_after: 2026-11-03
confidence: high confidence: high
tags: [cron, scheduled, automation] tags: [cron, scheduled, automation]
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks # Scheduled Tasks
## Count ## Count
- **Current as of 2026-10-27 00:1x AWST (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`). - **Current as of 2026-10-28 00:2x AWST (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed.
- **Prior window — as of 2026-10-27 00:1x AWST *(superseded by the 2026-10-28 entry above)* (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`).
- **Current as of 2026-10-26 00:1x AWST** *(superseded by the 2026-10-27 entry above)* **(live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`). - **Current as of 2026-10-26 00:1x AWST** *(superseded by the 2026-10-27 entry above)* **(live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`).
- **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page. *(Superseded by the 2026-10-26 entry above.)* - **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page. *(Superseded by the 2026-10-26 entry above.)*
- **Current as of 2026-10-24 00:2x AWST *(superseded by the 2026-10-25 entry above)* (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]] - **Current as of 2026-10-24 00:2x AWST *(superseded by the 2026-10-25 entry above)* (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]]
@@ -49,7 +50,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) | | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) |
| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 | | AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 |
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | | | Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00 | | Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00 |
| Daily Marketing Job Radar — Write and Letters (stage 2) | `5fc292ff0f3b` | `30 11,14 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Created 2026-10-27 17:34** — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). `custom:omni-101` / `smart-route`; first fires 2026-10-27 17:35 and 17:38 both `completed`. See [[incidents/2026-10-27-job-radar-budget-exhausted]] |
| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant | | DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant |
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] | | Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] |
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent | | Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
@@ -218,7 +220,7 @@ wholly clean: the first zero-failure day in the four on record here, and the end
|---|---|---|---| |---|---|---|---|
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `error` | 0 | | Facebook Birthday Monthly Scrape | `0e753e9720fd` | `error` | 0 |
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `error` | 0 | | Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `error` | 0 |
| Daily Marketing Job Radar | `dca8482e4f76` | `error` | 1 | | Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `error` | 1 |
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `error` | 1 | | Enhanced Weekend Adventure Planner | `1b23c042e17d` | `error` | 1 |
| Uber Eats Monthly Spend Report | `066bb5191280` | `delivery_failed` | 0 | | Uber Eats Monthly Spend Report | `066bb5191280` | `delivery_failed` | 0 |
+2 -2
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-26 updated: 2026-10-27
verified_on: 2026-10-26 verified_on: 2026-10-27
confidence: high confidence: high
tags: [skills, hermes] tags: [skills, hermes]
sources: [] sources: []