diff --git a/current-state.md b/current-state.md index 3305c40..5d95eb4 100644 --- a/current-state.md +++ b/current-state.md @@ -3,8 +3,8 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-05 -verified_on: 2026-10-05 +updated: 2026-10-06 +verified_on: 2026-10-06 review_after: 2026-10-31 confidence: high tags: [system] @@ -12,7 +12,14 @@ sources: [live-config, live-cron-list, live-hermes-version] --- # Current State -Updated: 2026-10-05 +Updated: 2026-10-06 + +## Recent Changes (2026-10-06) +- **Dead model pins cleared across the whole fleet — failing set 7 → 0.** All seven jobs that had been failing since 2026-10-01 were repinned on 2026-10-05 23:16 → 2026-10-06 00:05. Each had been pinned to a raw model name that died upstream: `cliprox/terra` (HTTP 401 connection auth, 3 jobs), `ollama-cloud/qwen3.5:397b` (HTTP 410, retired 2026-09-25), `nvidia/deepseek-ai/deepseek-v4-flash` (HTTP 400, dropped from the live catalog, 2 jobs) and `command-code/gpt-5.6-terra` (HTTP 403 `MODEL_NOT_IN_PLAN`). They were moved to `custom:omni-101` on **routing aliases** — `free-stack` ×6, `smart-route` ×1 — because the fault was raw-pin brittleness, not any one model. Every `failure_streak` in `jobs.json` now reads 0. Three were verified by live run: `79f9bb41ecf5` (23:39), `06a0a1c8704f` (23:45) and `dca8482e4f76` (23:56, delivered to topic 1030 — its first success since 2026-09-25). The other four are repinned but not yet re-run, so they carry a stale `last_status: error` until their next scheduled slot. See [[incidents/2026-10-05-dead-model-pin-repin]]. +- **The `cliprox/terra` connection incident is closed for cron purposes.** No enabled job references that pin any more — the jobs were moved off it rather than the proxy connection being re-authenticated. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. +- **GitHub Trending Radar's revoked-token diagnosis is refuted.** `c0a9fa314542` ran on schedule at **2026-10-05 09:00 and completed cleanly** — a second consecutive success after the off-schedule 2026-09-30 19:22 run. No `getMe` → 401, and no BotFather token replacement is needed. The previous audit correctly refused to treat the claim as standing fact; it can now be retired. +- **A stale failure notice, not a live outage, opened the review.** Anthony acted on a `85edf6772bca` notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job succeeded on every run from 2026-09-30 17:53 onward, and its three incidents were resolved on 2026-09-30 17:55:13; the notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.** +- **Window tally** (`executions.db`, 2026-10-05 00:00 → 2026-10-06 00:15 AWST): 185 `completed`, 3 `failed` (all pre-repin, 03:00–07:00), 2 `unknown` (aborted first attempts of `79f9bb41ecf5` / `dca8482e4f76`, superseded by their successful re-runs) and 1 `running` (the nightly audit itself). No interactive sessions other than the repin session. ## Recent Changes (2026-10-05) - **Cron fleet re-verified 2026-10-05 00:01 AWST — failing set 6 → 7.** Job count unchanged at 30 (27 enabled / 3 disabled). The one new failure event in the audit window is `79f9bb41ecf5` (Weekly Review, Sun 18:00), which ran on schedule **2026-10-04 18:00** and failed with the *identical* `cliprox/terra` error the incident predicted: `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. That is the last untested job on the pin, so the provider-connection auth expiry is now **3 of 3 confirmed** — settled as a credential fault, not per-job. No review was produced; `delivery_outcome: delivered` again because the failure notice is the deliverable. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. diff --git a/incidents/2026-10-01-cliproxy-connection-auth-expired.md b/incidents/2026-10-01-cliproxy-connection-auth-expired.md index 6ee6764..8e457c0 100644 --- a/incidents/2026-10-01-cliproxy-connection-auth-expired.md +++ b/incidents/2026-10-01-cliproxy-connection-auth-expired.md @@ -1,20 +1,20 @@ --- title: OmniRoute cliprox/terra Connection Auth Expiry type: incident -status: open +status: resolved created: 2026-10-02 -updated: 2026-10-05 -verified_on: 2026-10-05 +updated: 2026-10-06 +verified_on: 2026-10-06 confidence: high -tags: [cron, omniroute, provider, auth, incident] +tags: [cron, omniroute, provider, auth, incident, resolved] sources: [live-jobs-json, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050] --- # OmniRoute `cliprox/terra` Connection Auth Expiry -**Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); still open -**Status:** Open — not self-recovering; **all 3 exposed jobs now confirmed broken** -**Impact:** 3 of 3 jobs on this pin have failed on it; no job on `cliprox/terra` has produced output since the credential lapsed +**Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); closed 2026-10-05 23:45 AWST +**Status:** **Resolved for cron** — all three exposed jobs were repinned off the pin; the connection credential itself was never re-authenticated +**Impact:** 3 of 3 jobs on this pin failed on it (2026-10-01 → 2026-10-04); no job on `cliprox/terra` produced output while the credential was lapsed. Zero jobs remain on the pin. ## Symptoms @@ -90,12 +90,29 @@ Corollary worth recording: a job can be recorded as **successfully delivered whi having failed**, because the failure notice itself is a deliverable. Read the execution `status`, not the delivery outcome, when judging whether a job worked. -## Fix +## Resolution — 2026-10-05 · closed for cron purposes + +All three jobs were **repinned off `cliprox/terra`** on 2026-10-05 (23:39–23:45 AWST) as +part of the fleet-wide dead-model-pin sweep. `06a0a1c8704f` and `1b23c042e17d` moved to +`custom:omni-101` / `free-stack`; `79f9bb41ecf5` moved to `free-stack` and was verified by +a live manual run at 23:39. No enabled job references this pin any more, every +`failure_streak` in `jobs.json` is 0, and the incident is closed **for cron purposes**. + +Note precisely what was and was not fixed: the underlying OmniRoute connection credential +for `cliprox/terra` was **never re-authenticated**. The jobs were moved off the pin +instead. Any future job that targets `cliprox/terra` — or any manual session that calls it +— will hit the same `HTTP 401 … connection(s) authentication expired` until someone +reconnects it in the OmniRoute dashboard. That is the one piece of this incident still +outstanding, and it is outside the cron fleet's blast radius. +See [[incidents/2026-10-05-dead-model-pin-repin]]. + +## Fix (as originally diagnosed) Re-authenticate the OmniRoute connection for `cliprox/terra`. Do **not** repin these jobs to another raw model as a first move — that treats a credential fault as a routing fault and will produce a second, different outage. If the connection cannot be restored, repin -all three jobs together to a routing alias (`smart-route`). +all three jobs together to a routing alias (`smart-route`) — which is what was ultimately +done. ## Distinguish From @@ -107,6 +124,7 @@ all three jobs together to a routing alias (`smart-route`). ## Related +- [[incidents/2026-10-05-dead-model-pin-repin]] - [[systems/scheduled-tasks]] - [[systems/model-providers]] - [[systems/omniroute]] diff --git a/incidents/2026-10-05-dead-model-pin-repin.md b/incidents/2026-10-05-dead-model-pin-repin.md new file mode 100644 index 0000000..774b240 --- /dev/null +++ b/incidents/2026-10-05-dead-model-pin-repin.md @@ -0,0 +1,114 @@ +--- +title: Seven Cron Jobs on Dead Model Pins Repinned to free-stack / smart-route +type: incident +status: resolved +created: 2026-10-06 +updated: 2026-10-06 +verified_on: 2026-10-06 +confidence: high +tags: [cron, model-pin, omniroute, incident, resolved] +sources: [live-jobs-json, cron-executions-db, cron-incidents-db, session-20260810_122140_d687279c] +--- + +# Seven Cron Jobs on Dead Model Pins Repinned to `free-stack` / `smart-route` + +**Window:** failures 2026-10-01 → 2026-10-05; remediated 2026-10-05 23:16 → 2026-10-06 00:05 AWST +**Status:** Resolved — all seven repinned, every `failure_streak` cleared to 0; three verified by live run +**Trigger:** an interactive review on 2026-10-05 23:16 that began from a *stale* cron failure notice + +## Trigger: a six-day-old alert + +The review started because Anthony replied to a cron failure notice for `85edf6772bca` +(**Tech & AI Newsletter Digest (Analytical)**) reading: + +``` +Restart-safe cron worker dispatch failed: cron external worker exited before ownership +acknowledgement (exit 1) +``` +``` +This job has failed 4 runs in a row — worth a review. +``` + +**That alert was stale and the job was healthy.** `85edf6772bca` had completed on every run +from 2026-09-30 17:53 through 2026-10-05 07:00, and all three of its incidents were already +`resolved` as of **2026-09-30 17:55:13**. The notice came from the **2026-09-29 07:00** run +(incident `85edf6_7ab60a6b6d4a`, first seen 2026-09-29 07:00:45) and had been sitting +unread in the topic for six days. + +It looked current because the *wording* matches the live 2026-09-28 managed-runtime +incident signature (`cron external worker exited before ownership acknowledgement`) — see +[[incidents/2026-09-28-hermes-managed-runtime-deps]]. + +**Rule: a failure notice in a topic is not evidence of a current failure.** Check the job's +`executions` row timestamp (and its `cron_incidents` state) before acting on a notice — +a resolved incident still leaves its old message behind. + +## What was actually broken — seven jobs, four dead pins + +The check prompted by the stale alert found the set the 2026-10-01 sweep had flagged, all +still pinned to upstreams that no longer work: + +| Job | Name | Dead pin | Error | +|---|---|---|---| +| `1a8e90ea2271` | wiki-stocktake-maintenance | `ollama-cloud / qwen3.5:397b` | HTTP 410 — retired 2026-09-25 | +| `dca8482e4f76` | Daily Marketing Job Radar | `command-code / gpt-5.6-terra` | 403 `MODEL_NOT_IN_PLAN` | +| `fad5a86979d4` | Facebook Birthday Weekly Registry Sync | `nvidia / deepseek-v4-flash` | HTTP 400 — absent from live catalog | +| `0e753e9720fd` | Facebook Birthday Monthly Scrape | `nvidia / deepseek-v4-flash` | HTTP 400 — same pin | +| `06a0a1c8704f` | AI Newsletter Curation for Patrick | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — connection auth expired | +| `1b23c042e17d` | Enhanced Weekend Adventure Planner | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — same pin | +| `79f9bb41ecf5` | Weekly Review | `custom:Omniroute:20128 / cliprox/terra` | HTTP 401 — same pin | + +Four distinct fault classes, none self-recovering: retired model (410), plan-gated model +(403), catalog-missing model (400 ×2 jobs), provider-connection credential expiry (401 ×3 jobs). +The 401 pair is documented separately in +[[incidents/2026-10-01-cliproxy-connection-auth-expired]]. + +## Fix — repin to routing aliases + +All seven moved to provider **`custom:omni-101`** (OmniRoute on the estate), using routing +aliases rather than another raw upstream model name: + +| Alias | Jobs | +|---|---| +| `free-stack` | `06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd` | +| `smart-route` | `dca8482e4f76` | + +The root cause was **raw-pin brittleness** — a raw model name dies silently when the upstream +retires it, the catalog drops it, or a plan changes. An alias absorbs that. This is the same +lesson as the 2026-08-04 free-DeepSeek outage. + +`free-stack` is in service again: it was on the broken-combo list as recently as 2026-08-14. +Its route was re-verified before being handed six jobs — do not assume the 2026-08 list +is still current. + +## Verification + +| Job | Evidence | Outcome | +|---|---|---| +| `79f9bb41ecf5` | manual run 2026-10-05 23:39 | succeeded; incident auto-resolved 23:39:12 | +| `06a0a1c8704f` | manual run 2026-10-05 23:45 | succeeded; incident auto-resolved 23:45:01 | +| `dca8482e4f76` | manual run 23:48 → completed 23:56 | delivered to `telegram:-1004321904721:1030` (message_id 9367); incident resolved 2026-10-06 00:05:07 | + +Fleet-wide re-read after the repin: **30 jobs, 27 enabled / 3 disabled, `failure_streak` 0 on +every job** — down from seven jobs carrying streaks up to ×11. + +Enabled fleet by pin after the fix: `smart-route` ×10, no-agent scripts ×8, `free-stack` ×6, +`deepseek-paid-smart` ×2, stale `custom:omni` ×1. + +## Residual + +- Four repinned jobs still carry the *pre-repin* `last_status: error` — `1b23c042e17d`, + `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`. Their streaks are already 0; the status + clears on the next scheduled run. **Do not read those stale statuses as live failures.** +- `066bb5191280` still records `delivery_failed` (bare `telegram` target — separate open issue). +- The `cliprox/terra` connection itself was never re-authenticated. That pin remains unusable. +- The daily stocktake (`1a8e90ea2271`) had been blocked since 2026-09-25; with the repin it can + run again at its 03:00 schedule, so `wiki-maintain.sh` no longer needs a manual run each day. + +## Related + +- [[systems/scheduled-tasks]] +- [[systems/model-providers]] +- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] +- [[incidents/2026-09-28-hermes-managed-runtime-deps]] +- [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]] diff --git a/index.md b/index.md index c7b3f09..18d0a58 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,6 @@ # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-05 +> model routing and automation. Last updated: 2026-10-06 ## Start here - [[current-state]] — Current systems status @@ -28,13 +28,14 @@ ### Integrations - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. -- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; 7 failing as of 2026-10-05) +- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; **0 failing as of 2026-10-06** — 7 dead-model-pin jobs repinned 2026-10-05) - [[systems/skills-index]] — Installed skills inventory (auto-regenerated daily; being refreshed by hand while the stocktake job is blocked) ## Incidents +- [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05) - [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic - [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented -- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired; **all 3 jobs sharing the pin now failed** (settled 2026-10-04; open 2026-10-01) +- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); **closed 2026-10-05** when all three were repinned off the pin - [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26) - [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min - [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (active); ~48h of turns not captured diff --git a/log.md b/log.md index 11392e2..6feae8d 100644 --- a/log.md +++ b/log.md @@ -334,3 +334,26 @@ Reviewed window: 2026-10-04 00:12 → 2026-10-05 00:01 AWST. - **Streak-number caveat recorded:** `hermes cron list`'s "N failures in a row" is the `failure_streak` counter in `jobs.json`, not a DB count. `~/.hermes/cron/executions.db` now retains only rows from **2026-09-30 onward** (1,001 rows total), so `executions.db` recounts undercount older streaks — the earlier DB-based "×7" for `1a8e90ea2271` undercounts against the live ×11. Quote the live counter and note the retention floor. - **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source — `wiki/systems/scheduled-tasks.md`, `wiki/incidents/2026-10-01-cliproxy-connection-auth-expired.md`, `current-state.md`, `index.md`, `log.md`. - No secrets written; all credential-shaped values remain referenced by connection ID only. + +## [2026-10-06] update | Nightly wiki audit — dead model pins repinned, failing set 7 → 0 + +Reviewed window: 2026-10-05 00:04 → 2026-10-06 00:15 AWST. + +**New** +- `incidents/2026-10-05-dead-model-pin-repin.md` — the seven-job dead-model-pin repin. Records the four dead pins (`cliprox/terra` HTTP 401 ×3, `ollama-cloud/qwen3.5:397b` HTTP 410 ×1, `nvidia/deepseek-ai/deepseek-v4-flash` HTTP 400 ×2, `command-code/gpt-5.6-terra` HTTP 403 ×1), the remediation window 2026-10-05 23:16 → 2026-10-06 00:05, and the deliberate choice of routing aliases (`free-stack` ×6, `smart-route` ×1 on `custom:omni-101`) over another raw model name. + +**Updated** +- `systems/scheduled-tasks.md` — Count block rewritten: **0 failing** (was 7), with all 30 `failure_streak` counters at 0; four jobs carry a stale pre-repin `last_status: error` that clears on their next scheduled run. Seven job-table rows updated (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`, `dca8482e4f76`), plus `c0a9fa314542`. New `### Resolved 2026-10-05` subsection in Known Issues; five resolved bullets removed from `### Open`, leaving bare-`telegram` targets, the two-wiki-tree drift, the work-mailbox block, the stale `custom:omni` key and the delivered-notice rule as the standing open set. +- `index.md` — new incident linked at the top of the Incidents list; cron-fleet line re-dated to 0 failing as of 2026-10-06; the cliprox incident line marked closed. +- `current-state.md` — new Recent Changes (2026-10-06) block; frontmatter dates bumped. + +**Findings worth the ink** +- **`dca8482e4f76` (Daily Marketing Job Radar) delivered to topic 1030 at 23:56** — its first successful run since 2026-09-25. The repin is verified by live run, not assumed. Same for `79f9bb41ecf5` (23:39) and `06a0a1c8704f` (23:45). +- **GitHub Trending Radar's revoked-token diagnosis is dead.** `c0a9fa314542` ran on schedule at 2026-10-05 09:00 and completed cleanly — two consecutive successes after 2026-09-30 19:22. No `getMe` → 401; no BotFather token swap needed. The 2026-10-05 audit correctly refused to record this as standing fact; it can now be retired outright. +- **`delivery_outcome: delivered` still does not mean the job worked.** Every pre-repin failure recorded `delivered`, because the failure notice is itself the deliverable. Judge on the execution `status` column. +- **A stale notice opened the review.** Anthony was prompted by a `85edf6772bca` notice that had sat in topic `346` since the 2026-09-29 07:00 run; the job succeeded on every run from 2026-09-30 17:53 and its incidents were resolved 2026-09-30 17:55:13. The notice's wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.** +- **Window tally** (`executions.db`, 2026-10-05 00:00 → 2026-10-06 00:15): 185 `completed`, 3 `failed` (all pre-repin, 03:00–07:00), 2 `unknown` (aborted first attempts of `79f9bb41ecf5` and `dca8482e4f76` at 23:33/23:48, superseded by their successful re-runs) and 1 `running` (this audit). + +**Vault mirror:** the new incident page plus `systems/scheduled-tasks.md`, `index.md`, `current-state.md` and `log.md` mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view. + +No secrets written; credential-shaped values referenced by connection ID only. diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 3e31bc1..5386a92 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,8 +3,8 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-05 -verified_on: 2026-10-05 +updated: 2026-10-06 +verified_on: 2026-10-06 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db] @@ -13,37 +13,32 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count -- **Total jobs:** 30 — 27 enabled, 3 disabled (verified from live `~/.hermes/cron/jobs.json`, 2026-10-05) -- **Failing (last run failed outright):** 7 — - - `dca8482e4f76` — 403 `MODEL_NOT_IN_PLAN`, streak 7 - - `1a8e90ea2271` — HTTP 410, model retired upstream, streak 11 - - `fad5a86979d4` — HTTP 400, model absent from the NVIDIA live catalog, streak 3 (last run 2026-09-30) - - `79f9bb41ecf5` — **new 2026-10-04** — HTTP 401, provider-connection auth expired; the third and final job on the `cliprox/terra` pin (see below) - - `1b23c042e17d` — **new 2026-10-02** — HTTP 401, provider-connection auth expired (same pin as `06a0a1c8704f`) - - `06a0a1c8704f` — **new 2026-10-01** — HTTP 401, provider-connection auth expired - - `0e753e9720fd` — **new 2026-10-01** — HTTP 400, same NVIDIA pin as `fad5a86979d4` -- **Failing (delivery only):** 1 — `066bb5191280` (2026-10-01): the report itself generated; the send failed against the DM thread. -- **Resolved since the last entry:** 1 — `1b23c042e17d`'s stale `delivery_failed` flag cleared exactly as predicted. Its 2026-10-02 run delivered to topic `102` (`delivery_outcome: delivered`); it then failed on the model pin, above. -- **`cliprox/terra` pin — settled 2026-10-04: 3 of 3 broken.** All three enabled jobs on that pin (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`) have now failed with the identical `HTTP 401: All 1 connection(s) authentication expired`. `79f9bb41ecf5` (Weekly Review) ran 2026-10-04 18:00 and failed as the last untested job — settling it as a provider-connection credential fault, not a per-job fluke. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. -- **Streak counting note:** the `N failures in a row` figure in `hermes cron list` comes from the `failure_streak` counter in `jobs.json`. `executions.db` now retains only rows from **2026-09-30 onward** (1,001 rows total), so DB-based recounts undercount older streaks. Live counters as of 2026-10-05: `1a8e90ea2271` ×11, `dca8482e4f76` ×7, `fad5a86979d4` ×3. -- **Re-verified 2026-10-05 00:01 AWST:** job count unchanged at 30 (27 enabled / 3 disabled). Failing set moved 6 → **7** — the only new failure event in the window is `79f9bb41ecf5`'s 2026-10-04 18:00 run. No interactive (non-cron) sessions occurred in the past 24h, so the only durable changes are cron-fleet. `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found`), confirmed live in `hermes cron list`. -- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) +- **Total jobs:** 30 — 27 enabled, 3 disabled (verified from live `~/.hermes/cron/jobs.json`, 2026-10-06) +- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven still carry a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. See [[incidents/2026-10-05-dead-model-pin-repin]]. + - Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04). +- **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window. +- **Repin applied 2026-10-05:** `free-stack` ×6 (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`) and `smart-route` ×1 (`dca8482e4f76`), all on `custom:omni-101`. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. `free-stack` is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs. +- **Verified live, not assumed:** `79f9bb41ecf5` succeeded on a manual run 2026-10-05 23:39 and `06a0a1c8704f` at 23:45 (both incidents auto-resolved); `dca8482e4f76` completed 23:56 and delivered to `telegram:-1004321904721:1030` (incident resolved 2026-10-06 00:05:07). The other four are repinned but not yet re-run — they are due on their normal schedules. +- **Streak counting note:** the `N failures in a row` figure in `hermes cron list` comes from the `failure_streak` counter in `jobs.json`. `executions.db` retains only rows from **2026-09-30 onward**, so DB-based recounts undercount older streaks. Live counters as of 2026-10-06: **all 0**. +- **Re-verified 2026-10-06 AWST:** job count unchanged at 30 (27 enabled / 3 disabled). `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found`). +- **A stale notice, not a live outage, drove this review.** The interactive session on 2026-10-05 23:16 began because Anthony replied to a `85edf6772bca` failure notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check `executions.started_at` (and the `cron_incidents` state) before acting on a notice. +- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06) ## Job Table | Name | ID | Schedule | Delivery | Skills | Script | Notes | |---|---|---|---|---|---|---| | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) | -| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ⚠ HTTP 401 ×1 (2026-10-01) — `cliprox/terra` provider connection auth expired. **All 3 jobs on this pin have now failed on it** (see `1b23c042e17d`, `79f9bb41ecf5`), so this is a confirmed credential fault. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]] | +| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 | | Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | | -| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ⚠ 403 `MODEL_NOT_IN_PLAN` ×7 — see Known Issues | +| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). A manual run completed at 23:56 and delivered to topic 1030 — first successful run since 2026-09-25. Live `last_status: ok`, streak 0 | | DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24 | -| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ⚠ Now HTTP 401 ×1 (2026-10-02) — `cliprox/terra` connection auth expired, same pin as `06a0a1c8704f` and `79f9bb41ecf5` (pin now 3/3 broken); no plan produced | +| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). Not yet re-run — carries a stale `last_status: error` until Fri 08:00; streak 0 | | Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent | -| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ⚠ HTTP 400 ×1 (2026-10-01) — `nvidia/deepseek-ai/deepseek-v4-flash` not in the active NVIDIA catalog; delivery also failed on the DM thread. Bare `telegram` → home DM; retarget to a group topic | -| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ⚠ HTTP 400 ×3 — `nvidia/deepseek-ai/deepseek-v4-flash` not in the active NVIDIA catalog; bare `telegram` target should move to a group topic | +| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). Streak 0; stale `last_status: error` until the 1st. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic | +| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). Streak 0; stale `last_status: error` until Mon 03:00. ⚠ Bare `telegram` target should move to a group topic | | FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | | -| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | Token-revocation claim **not reproduced at the last recorded run** (2026-09-30 19:22, off-schedule: completed and delivered). The revoked-token (`getMe` → 401) diagnosis comes from an earlier run; the next scheduled run Mon 2026-10-05 09:00 confirms whether it still holds | +| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ✅ **Token-revocation claim refuted.** The scheduled run on 2026-10-05 09:00 completed cleanly — a second consecutive success after the off-schedule 2026-09-30 19:22 run — so the `getMe` → 401 diagnosis no longer applies and no token replacement is needed. Still delivers to the bare home DM, which works | | Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | | | Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) | | Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent | @@ -58,10 +53,10 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] | Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) | | Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | — | | | Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (`configured thread_id 34795 for telegram:1793951355 was not found`) — the 2026-09 report generated fine, only the send failed. Bare `telegram` → home DM; retarget to a group topic | -| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | runs on `cliprox/terra`. ⚠ **failed 2026-10-04 18:00** with the pin's HTTP 401 (`All 1 connection(s) authentication expired`) — the last untested job on that pin, now confirming it 3/3 broken. No review produced. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]] | +| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the `cliprox/terra` pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ **Repinned 2026-10-05 23:39** → `custom:omni-101` / `free-stack` and **verified by a live manual run**; the incident auto-resolved. Now `last_status: ok`, streak 0 | | weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 | | Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes | -| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ⚠ HTTP 410 ×11 — `qwen3.5:397b` retired 2026-09-25; needs repinning. Regenerates skills-index, freshness audit, rebuilds Quartz, commits to Gitea. **The script itself is healthy** — run by hand on 2026-10-03 and completed cleanly | +| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). Resumes its 03:00 schedule, so `wiki-maintain.sh` no longer needs a daily manual run. Streak 0; stale `last_status: error` until 03:00. Regenerates skills-index, freshness audit, rebuilds Quartz, commits to Gitea — **the script itself was never broken** | | 🦝 Hermes Config Backup to Gitea (Daily) | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent | ## Removed Jobs @@ -80,6 +75,13 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] ## Known Issues +### Resolved 2026-10-05 +- **Dead model pins across 7 jobs — repinned to routing aliases (resolved 2026-10-05 23:16 → 2026-10-06 00:05):** Seven jobs were each pinned to a raw model name that had died upstream — `cliprox/terra` (HTTP 401, connection auth expired, 3 jobs), `ollama-cloud/qwen3.5:397b` (HTTP 410, retired 2026-09-25), `nvidia/deepseek-ai/deepseek-v4-flash` (HTTP 400, dropped from the live catalog, 2 jobs) and `command-code/gpt-5.6-terra` (HTTP 403 `MODEL_NOT_IN_PLAN`) — across 2026-10-01 → 2026-10-05. All seven were repinned to provider `custom:omni-101` using **routing aliases** — `free-stack` ×6, `smart-route` ×1 — deliberately, because the fault was raw-pin brittleness and re-pinning to another raw name would have re-created it. `failure_streak` is now 0 on all 30 jobs, down from 7 failing. Three were verified by live run: `79f9bb41ecf5` (23:39), `06a0a1c8704f` (23:45) and `dca8482e4f76` (23:56, delivered to topic 1030 — its first success since 2026-09-25). The remaining four are repinned but not yet re-run; the stale `last_status: error` field clears on each one's next scheduled run. See [[incidents/2026-10-05-dead-model-pin-repin]]. + - The `cliprox/terra` incident is **closed for cron purposes**: no enabled job references that pin any more. The underlying proxy connection was never re-authenticated — the jobs were moved off it instead. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. + - `wiki-stocktake-maintenance` resumes its 03:00 schedule, so `wiki-maintain.sh` no longer needs a daily manual run. +- **GitHub Trending Radar `c0a9fa314542` — token-revocation claim refuted (2026-10-05):** the scheduled Mon 09:00 run **completed cleanly** — its second consecutive success after the off-schedule 2026-09-30 19:22 run. The `getMe` → 401 revoked-token diagnosis no longer applies and no token replacement is needed. It still delivers to the bare home DM, which works. +- **A stale failure notice, not a live outage, opened the 2026-10-05 review:** the interactive session began because Anthony acted on a `85edf6772bca` failure notice sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.** + ### Resolved 2026-09-30 - **Telegram thread delivery failures — root cause found and fixed:** jobs `fad5a86979d4`, `1b23c042e17d`, `c8d52904e1d6` reported `configured thread_id 34795 for telegram:1793951355 was not found` / `Message thread not found`. The target was wrong, not the topic: a **bare `deliver: "telegram"` resolves to the home DM (`1793951355`)**, where a group forum topic ID cannot exist — so the send hard-failed and the message was lost (an earlier "benign, delivered without thread" reading was wrong). `1b23c042e17d` and `c8d52904e1d6` were retargeted to `telegram:-1004321904721:102` and verified end-to-end. See [[systems/messaging-integrations]]. - **Managed-runtime dependency failure (2026-09-28 → 30):** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause was `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2, run every 15 min by job `96eb95ce12f2`. All managed venvs verified. See [[incidents/2026-09-28-hermes-managed-runtime-deps]]. @@ -88,13 +90,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] - **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`. ### Open -- **Daily Marketing Job Radar — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 10-02, streak 7):** Job `dca8482e4f76` is pinned to `custom:omni-101` / `gpt-5.6-terra`; the upstream (`command-code`) rejects it with `GPT-5.6 Terra available in Pro and above plans`. Anthony's primary job pipeline — needs a model or provider change; it cannot self-recover. See [[runbooks/job-radar]]. -- **`cliprox/terra` provider connection auth expired — 3 of 3 jobs confirmed (2026-10-04):** Jobs `06a0a1c8704f` (2026-10-01), `1b23c042e17d` (2026-10-02) and `79f9bb41ecf5` (2026-10-04) all failed with `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Failure notices reached their topics; no output was produced. Every enabled job on this pin has now failed on it — a *credential/connection* problem at the proxy, not a missing model, and it will not self-recover until the connection is re-authenticated. **Re-authenticate the OmniRoute connection before repinning anything.** See [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. -- **A delivered notice is not a successful job:** all three `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output. -- **wiki-stocktake-maintenance — HTTP 410, model retired (streak 11):** Job `1a8e90ea2271` is pinned to `ollama-cloud / qwen3.5:397b`, which Ollama retired on 2026-09-25 (`qwen3.5:397b was retired at 2026-09-25 00:00:00 -0700 PDT`). It cannot self-recover — it needs a new model pin. **The job's script is fine:** `wiki-maintain.sh` was run by hand on 2026-10-03 and completed cleanly, regenerating [[systems/skills-index]] (344 skills / 70 categories) and refreshing the freshness audit. Only the scheduled wrapper is broken — so every non-model step of the daily stocktake needs a manual run until it is repinned. It failed again on 2026-10-04 03:00 (same 410). -- **Facebook Birthday scrapes — HTTP 400, model missing from catalog (streak 3 / 1):** Job `fad5a86979d4` (weekly) is pinned to `nvidia/deepseek-ai/deepseek-v4-flash`; the provider reports the model is `not available in the active live catalog for provider 'nvidia'`. **Confirmed as a shared-pin issue on 2026-10-01**, when sibling job `0e753e9720fd` (monthly) failed on the same pin at its own 09:00 run. Safest durable fix is repinning both to a routing alias (`smart-route`) instead of a raw model name. -- **GitHub Trending Radar — revoked Telegram bot token (claim now unverified):** The recorded diagnosis is that job `c0a9fa314542` builds its digest successfully, then fails the Telegram push (`getMe` → 401 Unauthorized), with the token in `/home/hermes/workspace/github-trending-radar/config.json` needing replacement from @BotFather. However its **last recorded run (2026-09-30 19:22, off-schedule) completed and delivered**, so the failure was not reproduced then. Treat "token revoked" as needing re-confirmation at the next scheduled run (Mon 2026-10-05 09:00) rather than as a standing fact. Digest content remains recoverable from the script's `--dry-run` output in the interim. -- **Stale provider key `custom:omni` (2 jobs):** the provider was removed from `custom_providers` (the live 2026-10-01 set is `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101`). Jobs `e530c89f2be0` and `d40085da631c` still name it; both run `smart-route`, and the default `custom` provider resolves to the same base URL (`http://100.78.63.20:20128`), so no failure has been observed — but the references should be normalised to `custom:omni-101`. +- **A delivered notice is not a successful job:** `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output. +- **Stale provider key `custom:omni` (1 enabled + 1 disabled):** the provider was removed from `custom_providers` (live set on 2026-10-01: `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101`). Re-verified 2026-10-06: `e530c89f2be0` (enabled) and `d40085da631c` (disabled) still name it; both run `smart-route`, and the default `custom` provider resolves to the same base URL (`http://100.78.63.20:20128`), so no failure has been observed — but the references should be normalised to `custom:omni-101`. The 2026-10-05 repin moved seven jobs to an explicit `custom:omni-101`, leaving these two as the only remaining `custom:omni` references. - **Bare `telegram` delivery targets (3 jobs) — demonstrably breaking sends:** `0e753e9720fd`, `fad5a86979d4`, `066bb5191280` deliver to the home DM rather than a group topic. As of 2026-10-01 this is no longer theoretical: `066bb5191280` (02:11) and `0e753e9720fd` (09:01) both recorded `delivery_outcome: failed` with `configured thread_id 34795 for telegram:1793951355 was not found`. Work generated correctly; the delivery did not. Retarget all three to `telegram:-1004321904721:`. See [[systems/messaging-integrations]]. - **Two wiki trees, drifted in both directions (NEW 2026-10-02):** `/home/hermes/workspace/agent-estate-wiki` is the canonical source (Gitea remote `Anthony/agent-estate-wiki`, Quartz build, `wiki-maintain.sh` target). `/home/hermes/wiki/agent-estate-wiki` is the **Obsidian/Ignis vault copy** the iPad reads — pages nested under `wiki/`, and **no git remote at all**. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged *backwards* (`systems/ignis.md`, `systems/nextcloud.md`, three `incidents/` pages exist only in the vault; `raw/stocktake/*`, `agents/`, `incidents/2026-09-02-sftpgo-outage.md` only in canonical) — a naive one-way sync deletes pages. See [[systems/hermes-agent]]. - **Work mailbox unreachable (non-cron; affects on-demand lookups):** the `work` Outlook account is configured with password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` fails with `login not supported`. No cron job polls it. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]].