wiki audit 2026-10-05: cliprox/terra pin settled 3/3 (Weekly Review failed 2026-10-04 18:00), failing set 6->7, streak-counter/executions.db-retention caveat
This commit is contained in:
+9
-3
@@ -3,8 +3,8 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
updated: 2026-10-05
|
||||
verified_on: 2026-10-05
|
||||
review_after: 2026-10-31
|
||||
confidence: high
|
||||
tags: [system]
|
||||
@@ -12,7 +12,13 @@ sources: [live-config, live-cron-list, live-hermes-version]
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-10-04
|
||||
Updated: 2026-10-05
|
||||
|
||||
## Recent Changes (2026-10-05)
|
||||
- **Cron fleet re-verified 2026-10-05 00:01 AWST — failing set 6 → 7.** Job count unchanged at 30 (27 enabled / 3 disabled). The one new failure event in the audit window is `79f9bb41ecf5` (Weekly Review, Sun 18:00), which ran on schedule **2026-10-04 18:00** and failed with the *identical* `cliprox/terra` error the incident predicted: `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. That is the last untested job on the pin, so the provider-connection auth expiry is now **3 of 3 confirmed** — settled as a credential fault, not per-job. No review was produced; `delivery_outcome: delivered` again because the failure notice is the deliverable. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||||
- **`1a8e90ea2271` failed again — 2026-10-04 03:00 (HTTP 410).** The `wiki-stocktake-maintenance` job remains pinned to the Ollama-retired `qwen3.5:397b`; no change, still needs a repin. Its script is healthy. See [[systems/scheduled-tasks]].
|
||||
- **No interactive (non-cron) sessions in the window** — all 10 sessions active in the past 24h were cron. The only durable changes are cron-fleet status, reflected above.
|
||||
- **Streak-number caveat for future audits:** the `N failures in a row` figure in `hermes cron list` is the `failure_streak` counter from `jobs.json`, not a DB-derived count. `~/.hermes/cron/executions.db` now retains only rows from **2026-09-30 onward** (1,001 rows), so DB recounts undercount older streaks. Live counters 2026-10-05: `1a8e90ea2271` ×11, `dca8482e4f76` ×7, `fad5a86979d4` ×3. Quote the live counter, and note the retention floor.
|
||||
|
||||
## Recent Changes (2026-10-04)
|
||||
- **Both Hermes instances updated on 2026-10-03, from opposite directions.** CT460 (production) was rebuilt to `v0.21.5+6169.gab86a20` (upstream `ab86a20a`, tree mtime 10:11 AWST) and CT450 (chaos) came up from `v0.19.0` to `v0.21.5+6170.ga510d6c` (upstream `a510d6c6`, tree mtime 10:22 AWST). Same semver token, different builds, separate state. CT450 now reports `Update available: 3 commits behind`. CT450 has no SSH — it is reached via `ssh root@100.96.100.82` + `pct exec 450`, and `hermes` is not on the default `pct exec` PATH. See [[incidents/2026-10-03-ct450-hermes-update]].
|
||||
|
||||
@@ -3,18 +3,18 @@ title: OmniRoute cliprox/terra Connection Auth Expiry
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-10-02
|
||||
updated: 2026-10-03
|
||||
verified_on: 2026-10-03
|
||||
updated: 2026-10-05
|
||||
verified_on: 2026-10-05
|
||||
confidence: high
|
||||
tags: [cron, omniroute, provider, auth, incident]
|
||||
sources: [live-jobs-json, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040]
|
||||
sources: [live-jobs-json, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050]
|
||||
---
|
||||
|
||||
# OmniRoute `cliprox/terra` Connection Auth Expiry
|
||||
|
||||
**Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); still open
|
||||
**Status:** Open — not self-recovering
|
||||
**Impact:** 2 of the 3 exposed jobs have now failed on this pin; 1 remains untested
|
||||
**Status:** Open — not self-recovering; **all 3 exposed jobs now confirmed broken**
|
||||
**Impact:** 3 of 3 jobs on this pin have failed on it; no job on `cliprox/terra` has produced output since the credential lapsed
|
||||
|
||||
## Symptoms
|
||||
|
||||
@@ -66,10 +66,25 @@ That is now **2 of 3** exposed jobs confirmed broken on this pin, which retires
|
||||
doubt about a one-job fluke. Its failure notice was delivered to topic `102`, so the
|
||||
run is recorded `delivery_outcome: delivered` even though no plan was produced.
|
||||
|
||||
`79f9bb41ecf5` (**Weekly Review**, Sun 18:00) is the last untested one — its only
|
||||
`79f9bb41ecf5` (**Weekly Review**, Sun 18:00) was the last untested one — its only
|
||||
recorded run was 2026-09-27 18:00, four days *before* the connection lapsed, and it
|
||||
succeeded. Its next scheduled run is **2026-10-04 18:00**, which will confirm whether
|
||||
the expiry is tenant-wide for this model or per-connection.
|
||||
succeeded. Its next scheduled run was **2026-10-04 18:00**.
|
||||
|
||||
### Third job confirmed — 2026-10-04 18:00 · pin settled 3/3
|
||||
|
||||
`79f9bb41ecf5` ran on schedule on 2026-10-04 18:00 and failed with the *identical*
|
||||
error again:
|
||||
|
||||
```
|
||||
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
|
||||
— please reconnect in the dashboard
|
||||
```
|
||||
|
||||
That settles it: **3 of 3** jobs on `cliprox/terra` have now failed on the same
|
||||
credential fault. There is no remaining doubt about scope — it is not per-job, and it
|
||||
will not self-recover. Failure streak for `79f9bb41ecf5` is 1; no review was produced.
|
||||
The run's `delivery_outcome` is again `delivered` (the failure notice is the
|
||||
deliverable), so the execution `status` is the only honest signal.
|
||||
|
||||
Corollary worth recording: a job can be recorded as **successfully delivered while
|
||||
having failed**, because the failure notice itself is a deliverable. Read the
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Anthony's Systems Wiki
|
||||
> Operational knowledge for Anthony's agents, infrastructure,
|
||||
> model routing and automation. Last updated: 2026-10-04
|
||||
> model routing and automation. Last updated: 2026-10-05
|
||||
|
||||
## Start here
|
||||
- [[current-state]] — Current systems status
|
||||
@@ -28,13 +28,13 @@
|
||||
|
||||
### Integrations
|
||||
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
|
||||
- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; 6 failing as of 2026-10-04)
|
||||
- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; 7 failing as of 2026-10-05)
|
||||
- [[systems/skills-index]] — Installed skills inventory (auto-regenerated daily; being refreshed by hand while the stocktake job is blocked)
|
||||
|
||||
## Incidents
|
||||
- [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic
|
||||
- [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented
|
||||
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired; **2 of 3 jobs sharing the pin now failed**, the third untested (open 2026-10-01)
|
||||
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired; **all 3 jobs sharing the pin now failed** (settled 2026-10-04; open 2026-10-01)
|
||||
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26)
|
||||
- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min
|
||||
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (active); ~48h of turns not captured
|
||||
|
||||
@@ -322,3 +322,15 @@ Reviewed window: 2026-10-03 00:12 → 2026-10-04 00:12 AWST.
|
||||
|
||||
**Note** — `mnemosyne doctor` on this host prints `vec0: present_but_unloadable` and marks the vector tiers unavailable. That is a Python-3.11-legacy-`venv` artefact, not the gateway's state. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`.
|
||||
|
||||
|
||||
## [2026-10-05] update | Nightly wiki audit — cliprox pin settled 3/3, failing set 6 → 7
|
||||
|
||||
Reviewed window: 2026-10-04 00:12 → 2026-10-05 00:01 AWST.
|
||||
|
||||
- **Updated `systems/scheduled-tasks.md`** — fleet re-verified against live `~/.hermes/cron/jobs.json` + `hermes cron list`: still 30 jobs (27 enabled, 3 disabled), but the failing set moved **6 → 7**. The one new failure event in the window is `79f9bb41ecf5` (**Weekly Review**, Sun 18:00), which ran on schedule **2026-10-04 18:00** and failed with the *identical* `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. It was the last untested job on the `cliprox/terra` pin, so that pin is now **3 of 3 confirmed broken** — a provider-connection credential fault, not a per-job fluke. Count block, job table rows (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`) and the Known Issues Open list all updated; frontmatter dates bumped.
|
||||
- **Updated `incidents/2026-10-01-cliproxy-connection-auth-expired.md`** — new "Third job confirmed — 2026-10-04 18:00 · pin settled 3/3" section; Impact line changed from "2 of 3 failed, 1 untested" to "3 of 3"; status line and frontmatter dates/sources bumped. Still **Open** — needs the OmniRoute connection re-authenticated before any repin.
|
||||
- **Updated `current-state.md`** — new **Recent Changes (2026-10-05)** block: the cliprox 3/3 settlement, `1a8e90ea2271`'s repeat 410 on 2026-10-04, confirmation that **no interactive sessions occurred in the window** (all 10 active sessions were cron), and a streak-counting caveat.
|
||||
- **Updated `index.md`** — last-updated → 2026-10-05; cron-fleet line now reads "7 failing as of 2026-10-05"; cliprox incident line now "all 3 jobs sharing the pin now failed".
|
||||
- **Streak-number caveat recorded:** `hermes cron list`'s "N failures in a row" is the `failure_streak` counter in `jobs.json`, not a DB count. `~/.hermes/cron/executions.db` now retains only rows from **2026-09-30 onward** (1,001 rows total), so `executions.db` recounts undercount older streaks — the earlier DB-based "×7" for `1a8e90ea2271` undercounts against the live ×11. Quote the live counter and note the retention floor.
|
||||
- **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source — `wiki/systems/scheduled-tasks.md`, `wiki/incidents/2026-10-01-cliproxy-connection-auth-expired.md`, `current-state.md`, `index.md`, `log.md`.
|
||||
- No secrets written; all credential-shaped values remain referenced by connection ID only.
|
||||
|
||||
+17
-16
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-04
|
||||
verified_on: 2026-10-04
|
||||
updated: 2026-10-05
|
||||
verified_on: 2026-10-05
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
@@ -13,31 +13,32 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Total jobs:** 30 — 27 enabled, 3 disabled (verified from live `~/.hermes/cron/jobs.json`, 2026-10-03)
|
||||
- **Failing (last run failed outright):** 6 —
|
||||
- **Total jobs:** 30 — 27 enabled, 3 disabled (verified from live `~/.hermes/cron/jobs.json`, 2026-10-05)
|
||||
- **Failing (last run failed outright):** 7 —
|
||||
- `dca8482e4f76` — 403 `MODEL_NOT_IN_PLAN`, streak 7
|
||||
- `1a8e90ea2271` — HTTP 410, model retired upstream, streak 7
|
||||
- `1a8e90ea2271` — HTTP 410, model retired upstream, streak 11
|
||||
- `fad5a86979d4` — HTTP 400, model absent from the NVIDIA live catalog, streak 3 (last run 2026-09-30)
|
||||
- `79f9bb41ecf5` — **new 2026-10-04** — HTTP 401, provider-connection auth expired; the third and final job on the `cliprox/terra` pin (see below)
|
||||
- `1b23c042e17d` — **new 2026-10-02** — HTTP 401, provider-connection auth expired (same pin as `06a0a1c8704f`)
|
||||
- `06a0a1c8704f` — **new 2026-10-01** — HTTP 401, provider-connection auth expired
|
||||
- `0e753e9720fd` — **new 2026-10-01** — HTTP 400, same NVIDIA pin as `fad5a86979d4`
|
||||
- **Failing (delivery only):** 1 — `066bb5191280` (2026-10-01): the report itself generated; the send failed against the DM thread.
|
||||
- **Resolved since the last entry:** 1 — `1b23c042e17d`'s stale `delivery_failed` flag cleared exactly as predicted. Its 2026-10-02 run delivered to topic `102` (`delivery_outcome: delivered`); it then failed on the model pin, above.
|
||||
- **Shared model pin at risk:** 3 enabled jobs run `cliprox/terra` — `06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`. **Two of the three have now failed on that pin.** The third (`79f9bb41ecf5`) last ran 2026-09-27 — before the credential lapsed — and is next due 2026-10-04 18:00. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||||
- **Streak recount (2026-10-03):** recounted directly from `~/.hermes/cron/executions.db`. `dca8482e4f76` 6 → 7 (one more daily run since), and `1a8e90ea2271` 8 → **7** — the ×8 carried in the 2026-10-02 entry was an overcount: that job has exactly 7 recorded runs and all 7 failed.
|
||||
- **Re-verified 2026-10-04 00:12 AWST — no change:** still 30 jobs (27 enabled / 3 disabled) and the same **6** failing jobs; no new failure event landed inside the audit window. `79f9bb41ecf5` (Weekly Review, the last untested `cliprox/terra` job) drops at **2026-10-04 18:00 AWST** — its result settles whether that pin is 3/3 broken. `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found; delivered without thread_id`), confirmed live in `hermes cron list`.
|
||||
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03)
|
||||
- **`cliprox/terra` pin — settled 2026-10-04: 3 of 3 broken.** All three enabled jobs on that pin (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`) have now failed with the identical `HTTP 401: All 1 connection(s) authentication expired`. `79f9bb41ecf5` (Weekly Review) ran 2026-10-04 18:00 and failed as the last untested job — settling it as a provider-connection credential fault, not a per-job fluke. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||||
- **Streak counting note:** the `N failures in a row` figure in `hermes cron list` comes from the `failure_streak` counter in `jobs.json`. `executions.db` now retains only rows from **2026-09-30 onward** (1,001 rows total), so DB-based recounts undercount older streaks. Live counters as of 2026-10-05: `1a8e90ea2271` ×11, `dca8482e4f76` ×7, `fad5a86979d4` ×3.
|
||||
- **Re-verified 2026-10-05 00:01 AWST:** job count unchanged at 30 (27 enabled / 3 disabled). Failing set moved 6 → **7** — the only new failure event in the window is `79f9bb41ecf5`'s 2026-10-04 18:00 run. No interactive (non-cron) sessions occurred in the past 24h, so the only durable changes are cron-fleet. `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found`), confirmed live in `hermes cron list`.
|
||||
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05)
|
||||
|
||||
## Job Table
|
||||
|
||||
| Name | ID | Schedule | Delivery | Skills | Script | Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) |
|
||||
| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ⚠ HTTP 401 ×1 (2026-10-01) — `cliprox/terra` provider connection auth expired; **a second job on this pin failed 2026-10-02** (`1b23c042e17d`), so this is a confirmed credential fault, not a fluke. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]] |
|
||||
| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ⚠ HTTP 401 ×1 (2026-10-01) — `cliprox/terra` provider connection auth expired. **All 3 jobs on this pin have now failed on it** (see `1b23c042e17d`, `79f9bb41ecf5`), so this is a confirmed credential fault. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]] |
|
||||
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
|
||||
| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ⚠ 403 `MODEL_NOT_IN_PLAN` ×7 — see Known Issues |
|
||||
| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24 |
|
||||
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ⚠ Now HTTP 401 ×1 (2026-10-02) — `cliprox/terra` connection auth expired, same pin as `06a0a1c8704f`; no plan produced |
|
||||
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ⚠ Now HTTP 401 ×1 (2026-10-02) — `cliprox/terra` connection auth expired, same pin as `06a0a1c8704f` and `79f9bb41ecf5` (pin now 3/3 broken); no plan produced |
|
||||
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
|
||||
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ⚠ HTTP 400 ×1 (2026-10-01) — `nvidia/deepseek-ai/deepseek-v4-flash` not in the active NVIDIA catalog; delivery also failed on the DM thread. Bare `telegram` → home DM; retarget to a group topic |
|
||||
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ⚠ HTTP 400 ×3 — `nvidia/deepseek-ai/deepseek-v4-flash` not in the active NVIDIA catalog; bare `telegram` target should move to a group topic |
|
||||
@@ -57,10 +58,10 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) |
|
||||
| Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | — | |
|
||||
| Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (`configured thread_id 34795 for telegram:1793951355 was not found`) — the 2026-09 report generated fine, only the send failed. Bare `telegram` → home DM; retarget to a group topic |
|
||||
| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | runs on `cliprox/terra`. Last run 2026-09-27 completed and delivered — **before** the connection lapsed. Next due 2026-10-04 18:00: the last untested job on this pin |
|
||||
| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | runs on `cliprox/terra`. ⚠ **failed 2026-10-04 18:00** with the pin's HTTP 401 (`All 1 connection(s) authentication expired`) — the last untested job on that pin, now confirming it 3/3 broken. No review produced. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]] |
|
||||
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
|
||||
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes |
|
||||
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ⚠ HTTP 410 ×7 — `qwen3.5:397b` retired 2026-09-25; needs repinning. Regenerates skills-index, freshness audit, rebuilds Quartz, commits to Gitea. **The script itself is healthy** — run by hand on 2026-10-03 and completed cleanly |
|
||||
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ⚠ HTTP 410 ×11 — `qwen3.5:397b` retired 2026-09-25; needs repinning. Regenerates skills-index, freshness audit, rebuilds Quartz, commits to Gitea. **The script itself is healthy** — run by hand on 2026-10-03 and completed cleanly |
|
||||
| 🦝 Hermes Config Backup to Gitea (Daily) | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
|
||||
|
||||
## Removed Jobs
|
||||
@@ -88,9 +89,9 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
|
||||
### Open
|
||||
- **Daily Marketing Job Radar — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 10-02, streak 7):** Job `dca8482e4f76` is pinned to `custom:omni-101` / `gpt-5.6-terra`; the upstream (`command-code`) rejects it with `GPT-5.6 Terra available in Pro and above plans`. Anthony's primary job pipeline — needs a model or provider change; it cannot self-recover. See [[runbooks/job-radar]].
|
||||
- **`cliprox/terra` provider connection auth expired — 2 of 3 jobs confirmed (streak 1 each):** Jobs `06a0a1c8704f` (2026-10-01) and `1b23c042e17d` (2026-10-02) both failed with `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Failure notices reached their topics; no output was produced. This is a *credential/connection* problem at the proxy, not a missing model — it will not self-recover until the connection is re-authenticated. `79f9bb41ecf5` is the remaining untested job on this pin (next run 2026-10-04 18:00 — re-check on 2026-10-05). Re-authenticate the OmniRoute connection before repinning anything. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||||
- **A delivered notice is not a successful job:** both `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output.
|
||||
- **wiki-stocktake-maintenance — HTTP 410, model retired (streak 7):** Job `1a8e90ea2271` is pinned to `ollama-cloud / qwen3.5:397b`, which Ollama retired on 2026-09-25 (`qwen3.5:397b was retired at 2026-09-25 00:00:00 -0700 PDT`). It cannot self-recover — it needs a new model pin. **The job's script is fine:** `wiki-maintain.sh` was run by hand on 2026-10-03 and completed cleanly, regenerating [[systems/skills-index]] (344 skills / 70 categories) and refreshing the freshness audit. Only the scheduled wrapper is broken — so every non-model step of the daily stocktake needs a manual run until it is repinned.
|
||||
- **`cliprox/terra` provider connection auth expired — 3 of 3 jobs confirmed (2026-10-04):** Jobs `06a0a1c8704f` (2026-10-01), `1b23c042e17d` (2026-10-02) and `79f9bb41ecf5` (2026-10-04) all failed with `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Failure notices reached their topics; no output was produced. Every enabled job on this pin has now failed on it — a *credential/connection* problem at the proxy, not a missing model, and it will not self-recover until the connection is re-authenticated. **Re-authenticate the OmniRoute connection before repinning anything.** See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||||
- **A delivered notice is not a successful job:** all three `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output.
|
||||
- **wiki-stocktake-maintenance — HTTP 410, model retired (streak 11):** Job `1a8e90ea2271` is pinned to `ollama-cloud / qwen3.5:397b`, which Ollama retired on 2026-09-25 (`qwen3.5:397b was retired at 2026-09-25 00:00:00 -0700 PDT`). It cannot self-recover — it needs a new model pin. **The job's script is fine:** `wiki-maintain.sh` was run by hand on 2026-10-03 and completed cleanly, regenerating [[systems/skills-index]] (344 skills / 70 categories) and refreshing the freshness audit. Only the scheduled wrapper is broken — so every non-model step of the daily stocktake needs a manual run until it is repinned. It failed again on 2026-10-04 03:00 (same 410).
|
||||
- **Facebook Birthday scrapes — HTTP 400, model missing from catalog (streak 3 / 1):** Job `fad5a86979d4` (weekly) is pinned to `nvidia/deepseek-ai/deepseek-v4-flash`; the provider reports the model is `not available in the active live catalog for provider 'nvidia'`. **Confirmed as a shared-pin issue on 2026-10-01**, when sibling job `0e753e9720fd` (monthly) failed on the same pin at its own 09:00 run. Safest durable fix is repinning both to a routing alias (`smart-route`) instead of a raw model name.
|
||||
- **GitHub Trending Radar — revoked Telegram bot token (claim now unverified):** The recorded diagnosis is that job `c0a9fa314542` builds its digest successfully, then fails the Telegram push (`getMe` → 401 Unauthorized), with the token in `/home/hermes/workspace/github-trending-radar/config.json` needing replacement from @BotFather. However its **last recorded run (2026-09-30 19:22, off-schedule) completed and delivered**, so the failure was not reproduced then. Treat "token revoked" as needing re-confirmation at the next scheduled run (Mon 2026-10-05 09:00) rather than as a standing fact. Digest content remains recoverable from the script's `--dry-run` output in the interim.
|
||||
- **Stale provider key `custom:omni` (2 jobs):** the provider was removed from `custom_providers` (the live 2026-10-01 set is `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101`). Jobs `e530c89f2be0` and `d40085da631c` still name it; both run `smart-route`, and the default `custom` provider resolves to the same base URL (`http://100.78.63.20:20128`), so no failure has been observed — but the references should be normalised to `custom:omni-101`.
|
||||
|
||||
Reference in New Issue
Block a user