audit(2026-10-26): runaway Gmail digest + dead Nous credential; model default reverted to OmniRoute; conduit 0.69.2 live; freshness floor still 2026-10-08
This commit is contained in:
@@ -69,14 +69,17 @@ Unified LLM router at `100.82.40.23:3001` (also `freellmapi.martinwa.org`). Veri
|
||||
- Exposes `/api/fallback/routing`, `/api/fallback/penalty-inspector`, `/api/fallback/token-usage`.
|
||||
This is the router that resolves Hermes's **default model** `freellmapi-auto` — distinct from OmniRoute's LKGP (below).
|
||||
|
||||
### OmniRoute — fallback / auxiliary
|
||||
Not used for Hermes's main chat path but handles specific auxiliary roles (`skills_hub`, `approval`, `mcp`) and serves as the fallback provider. Has a much larger advertised catalog (~3430 vs ~1512) but that catalog can diverge from what's actually resolvable — combo discovery vs. resolution can differ (documented known issue). See [[systems/omniroute]].
|
||||
### OmniRoute — ⚠ main chat path again since 2026-10-24 (was "fallback / auxiliary")
|
||||
**Re-verified 2026-10-26: this section is out of date.** Hermes's **default** is `custom:omni-101` /
|
||||
`smart-route` on the `omniroute-1` node (`http://100.78.63.20:20128`) — i.e. OmniRoute **is** the main chat
|
||||
path, as it was before the 2026-10-20 → 10-24 `nous` detour → [[systems/current-profile]].
|
||||
It also still handles auxiliary roles and appears in the `fallback_providers` chain.` Has a much larger advertised catalog (~3430 vs ~1512) but that catalog can diverge from what's actually resolvable — combo discovery vs. resolution can differ (documented known issue). See [[systems/omniroute]].
|
||||
|
||||
### OpenRouter — kept only as a fallback decision, not wired in
|
||||
**Verified:** absent from `custom_providers` in the live config. A decision page ([[decisions/keep-openrouter-as-fallback]]) records intent to retain it as a fallback option, but there's no live routing through it today — see [[systems/openrouter]] for the "previously used, largely replaced" history.
|
||||
|
||||
## Verdict
|
||||
Two routers are actually live (LiteLLM primary, OmniRoute fallback/auxiliary); OpenRouter is a retained *intent*, not a working path, as of 2026-07-22. If [[decisions/keep-openrouter-as-fallback]] is still desired, it needs an actual `custom_providers` entry — currently it's fallback-in-name-only.
|
||||
Two routers are live (**OmniRoute primary** for the default path and auxiliary slots since 2026-10-24 19:06; **LiteLLM** as the deeper fallback), OpenRouter is a retained *intent*, not a working path — re-verified 2026-10-26. If [[decisions/keep-openrouter-as-fallback]] is still desired, it needs an actual `custom_providers` entry — currently it's fallback-in-name-only.
|
||||
|
||||
## Related
|
||||
- [[systems/model-providers]]
|
||||
|
||||
@@ -18,12 +18,17 @@ How Hermes selects providers, applies combos, and enforces context limits.
|
||||
## Current Flow
|
||||
- Main path: `custom:litellm` → `http://litellm:4000/v1`
|
||||
- Default model: `freellmapi-auto`
|
||||
- OmniRoute used for auxiliary roles: vision, skills_hub, approval, mcp, title_generation, web_extract, compression
|
||||
- OmniRoute used for the **default chat path** (`custom:omni-101` / `smart-route`, since 2026-10-24 19:06) and for **all eight auxiliary slots** (`approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`, `title_generation`, `vision` — all `no-think/freeapi/auto` as of 2026-10-25 19:13)
|
||||
|
||||
> **Live `model:` block re-read 2026-10-20 — the default has moved.**
|
||||
> `model.default` is now **`meituan/longcat-2.5-preview:free`** (provider `nous`), not
|
||||
> `freellmapi-auto`. The block also carries `base_url`, `persist_switch_by_default` and `api_mode`.
|
||||
> **Live `model:` block re-read 2026-10-26 — the default has moved TWICE.**
|
||||
> `model.default` is **`smart-route`** on provider **`custom:omni-101`**
|
||||
> (`http://100.78.63.20:20128`), `api_mode: chat_completions`. It moved to `nous` /
|
||||
> `meituan/longcat-2.5-preview:free` on 2026-10-20 22:01 and **back** on **2026-10-24 19:06:10**.
|
||||
> A six-provider top-level `fallback_providers` chain is live again, headed by the **dead `nous`
|
||||
> credential** (`Invalid refresh token` since 2026-10-22).
|
||||
> → [[systems/current-profile]] · [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
>
|
||||
|
||||
> **The five `custom_providers` and their live base URLs:**
|
||||
>
|
||||
> | Name | Base URL | Host |
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
title: Nous Portal Refresh Token Invalid — Context Summaries Unavailable
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-10-26
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
review_after: 2026-11-09
|
||||
confidence: high
|
||||
tags: [hermes, router, memory, incident, unresolved]
|
||||
sources: [gateway-log, live-config, live-probe]
|
||||
---
|
||||
|
||||
# Nous Portal Refresh Token Invalid — Context Summaries Unavailable (2026-10-22 →)
|
||||
|
||||
`Nous Portal runtime credentials unavailable: Invalid refresh token. Run `hermes model` to re-authenticate.`
|
||||
|
||||
## Occurrences (gateway logs, counted 2026-10-26)
|
||||
| Day | Lines | Notable |
|
||||
|---|---|---|
|
||||
| 2026-10-22 | 1 | 11:32:20 `tui_gateway.server: llm.oneshot failed` |
|
||||
| 2026-10-24 | 3 | — |
|
||||
| 2026-10-25 | **20** | last at 14:53:04 — `agent.context_compressor: Failed to generate context summary` |
|
||||
| 2026-10-26 | 0 (to 00:1x) | none since the 19:27:13 restart |
|
||||
|
||||
## Effect
|
||||
- **Context compression failed outright.** Sessions could not summarise, so working context grew unchecked —
|
||||
one cron fire reached **148 k tokens** on 10-25 and ran 1 h 55 m.
|
||||
→ [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]
|
||||
- Every auxiliary LLM path pointed at `nous` failed with it.
|
||||
|
||||
## What changed — and what did not
|
||||
- **2026-10-25, `config.yaml` written 19:13 AWST:** all **eight** `auxiliary:` slots moved off `nous` to
|
||||
**`no-think/freeapi/auto` on `custom:omni-101`**, each with the chain
|
||||
`no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, `free_only: true`
|
||||
(compression timeout 90 s, vision timeout 180 s). → [[systems/model-providers]]
|
||||
- **The credential is still invalid.** The fault is **masked, not fixed** — no `hermes model`
|
||||
re-authentication has been run, and `nous` is still the **first** entry in `fallback_providers`.
|
||||
- Zero `invalid_grant` lines appear after the 2026-10-25 19:27:13 gateway start, so the masking is holding.
|
||||
|
||||
## Open
|
||||
- Re-authenticate the Nous Portal credential, or remove `nous` from `fallback_providers`.
|
||||
- Until then, a failover event will attempt a dead credential silently.
|
||||
|
||||
## Related
|
||||
- [[systems/model-providers]]
|
||||
- [[systems/current-profile]]
|
||||
- [[systems/hermes-agent]]
|
||||
@@ -3,7 +3,7 @@ title: Conduit Call Requests Refused by the Agent
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-10-25
|
||||
updated: 2026-10-25
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-25
|
||||
review_after: 2026-11-08
|
||||
confidence: high
|
||||
@@ -86,3 +86,10 @@ into [[incidents/2026-10-22-conduit-memory-start-timeout]].
|
||||
- [[systems/conduit]]
|
||||
- [[incidents/2026-10-22-conduit-memory-start-timeout]]
|
||||
- [[concepts/tool-calling]]
|
||||
|
||||
## Status update — 2026-10-26 00:1x AWST
|
||||
- **No further refusals in-window** (2026-10-24 12:56 → 2026-10-26 00:1x). The signature has not recurred.
|
||||
- `conduit-calls.json` → `history` is still **empty (0 entries)**, so the call path has gone unused, not fixed. The file *does* carry `watches`, `outcomes`, `ends` and `told` state.
|
||||
- The plugin **is** working for cards: `conduit_show` ×1 on 2026-10-25 14:37, delivered live.
|
||||
- Plugin advanced to **`0.69.2`** and the dashboard gateway restarted at **18:42:33**, so upstream commits are live — see [[systems/conduit]].
|
||||
- **Status stays open**: the request path is untested, not confirmed good.
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
---
|
||||
title: Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-10-26
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
review_after: 2026-11-09
|
||||
confidence: medium
|
||||
tags: [cron, automation, hermes, incident, unresolved]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-sessions, gateway-log, jobs-json]
|
||||
---
|
||||
|
||||
# Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop (2026-10-25)
|
||||
|
||||
**Windows:** 2026-10-25 08:31:14 → 10:26:37 AWST (runaway) and 17:19:36 → 19:51:59 AWST (five extra fires)
|
||||
**Job:** `08eb86f02d37` — *Gmail Action Digest*, daily `30 8 * * *`, pinned `custom:omni-101`
|
||||
|
||||
## What happened
|
||||
- The 10-25 **08:30 slot** claimed at 08:30:57, started **08:31:14** and did not finish until **10:26:37 —
|
||||
a 1 h 55 m 23 s run**, against **91 s** (10-23) and **104 s** (10-24) in the two preceding days. It closed
|
||||
`failed` on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed).`
|
||||
- The same job then fired **five more times** on 10-25 — 17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53 —
|
||||
for **six fires in the calendar day** against one configured daily slot.
|
||||
- `19:18:41 → 19:28:04` closed `failed` on a signature this ledger had not seen before:
|
||||
**`Cron admission missing: the owner never admitted this fire`** (`source: direct`, `pid 2473801`,
|
||||
`scheduled_instant` empty). The 2026-10-20 orphan was `Unclean gateway exit` — a second, distinct variant.
|
||||
- Every fire, including both failures, carries `delivery_outcome: delivered`, so **up to six Gmail Action
|
||||
Digests reached Anthony on 2026-10-25**. The 19:30:31 fire ran 21 m 28 s.
|
||||
- Restart context (`gateway.log`): `18:29:22` SIGTERM as a planned stop under systemd → `18:30:04` start;
|
||||
`19:26:16` *Stopping gateway for restart* → `19:27:13` start. Three starts on 10-25 (00:32:20, 18:30:04,
|
||||
19:27:13). The five extra fires bracket the evening pair. → [[systems/hermes-agent]]
|
||||
|
||||
## Root cause — hypothesised, not confirmed
|
||||
The 2026-10-25 14:17 interactive investigation found that the 08:31 fire's worker had made **70 `terminal`
|
||||
calls on unrelated Proxmox VM101 `/dev/sdb` backup work**, with run context growing **66 k → 148 k tokens**,
|
||||
and that **context compression was unavailable all day** (`invalid_grant`). Working hypothesis: the scheduled
|
||||
fire **adopted and continued an unrelated long-lived session** instead of starting its own. *Why* the
|
||||
scheduler admitted an unrelated session is unexplained and is the first thing to check.
|
||||
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
|
||||
## Impact
|
||||
- Six delivered digests in one day (duplicate content) — the user-visible symptom.
|
||||
- **Neither failure raised an `cron_incidents` row**; the two open rows are older (10-01, 10-19). A failed
|
||||
execution therefore does not reliably produce an incident.
|
||||
- The job now reads `last_status: delivery_queued` with an **empty** `last_error` (2026-10-26 00:1x) — a state
|
||||
that did not resolve to `ok`.
|
||||
- 2026-10-25 closed **172 `completed` / 2 `failed`**, both failures belonging to this job.
|
||||
|
||||
## Open questions / next actions
|
||||
- Why did a scheduled fire adopt an unrelated session? Check session-reuse across fires.
|
||||
- Record `Cron admission missing: the owner never admitted this fire` alongside the 2026-10-20 orphan
|
||||
signature and give both a recovery path on gateway restart.
|
||||
- Give scheduled runs a **wall-clock budget** — a 113× runtime creep was invisible anywhere except the
|
||||
execution row.
|
||||
- Does `delivery_queued` clear on the 2026-10-26 08:30 slot? Check after that fire.
|
||||
|
||||
## Related
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
- [[systems/hermes-agent]]
|
||||
@@ -1,6 +1,6 @@
|
||||
# Anthony's Systems Wiki
|
||||
> Operational knowledge for Anthony's agents, infrastructure,
|
||||
> model routing and automation. Last updated: 2026-10-25
|
||||
> model routing and automation. Last updated: 2026-10-26
|
||||
>
|
||||
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
|
||||
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
|
||||
@@ -32,13 +32,15 @@
|
||||
- [[systems/tool-search]] — Always-on lazy discovery for external tools
|
||||
|
||||
### Integrations
|
||||
- [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.69.0` (HEAD `f000bf0` — 22 commits on since the 10-24 audit)**, upstream `kaishi00/hermes-conduit-notifier`; gateway on `:9119`, process up since **2026-10-23 18:54:43** (so those upstream commits are **not** live); one registered device (iPhone app `2.0.0`; phone-status contact last 10-23 19:10, health ingestion to 10-24 09:59; store 569.9 MB / 1,520,229 events / 3,028 `buckets`). **Still carries the uncommitted in-tree patch — now five releases behind HEAD** (`conduit-calls.json → history` is **empty**, so the memory fix has no positive call evidence; the “two calls closed cleanly 00:03:23 / 00:06:57” reading was **wrong** — those were cron turn-ends). ⚠ **Known failure: three 10-24 requests to be phoned were refused by the agent, which never called the tool** → [[incidents/2026-10-24-conduit-call-request-refused]].
|
||||
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped** (peer offline 29d; host answers neither `:8065` nor ICMP).
|
||||
- [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.69.2` (HEAD `942ddb8`, 2026-10-24 15:35 — *"Pulse: list candidates, decide in code, journal them (#342)"*; 24 commits on since the `0.64.0` baseline, **0 behind `origin/main`**)**, upstream `kaishi00/hermes-conduit-notifier`. **Gateway on `:9119` restarted 2026-10-25 18:42:33 — so the upstream commits are now LIVE** (the 10-24 "not live" note is superseded). One registered device (iPhone app `2.0.0`). **Still carries the uncommitted in-tree `dashboard/plugin_api.py` patch**; `conduit-calls.json → history` is **empty**, so the memory fix still has no positive call evidence, but `conduit_show` ran once on 10-25 14:37 (card delivered). ⚠ **Known failure: three 10-24 requests to be phoned were refused by the agent, which never called the tool; no recurrence since** → [[incidents/2026-10-24-conduit-call-request-refused]].
|
||||
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26** (peer offline **36 d**, `tx 312 / rx 0`; host answers neither `:8065` nor ICMP). **268** connect timeouts on 10-25 vs 264 on 10-24; the reconnect **attempt counter reset to 54** at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost.
|
||||
- [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip.
|
||||
- [[systems/scheduled-tasks]] — **As of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
|
||||
- [[systems/scheduled-tasks]] — **As of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
|
||||
- [[systems/skills-index]] — Installed skills inventory (**306 skills / 70 categories** as of the **2026-10-23 03:07** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at that run: **2 stale / 94 fresh / 6 no-date of 102 pages** (`freshness-20261023.txt`). Note: `SKILL.md` files on disk still exceed the index (the generator de-dups one nested duplicate; see [[systems/scheduled-tasks]]). ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
|
||||
|
||||
## Incidents
|
||||
- [[incidents/2026-10-25-gmail-digest-runaway-and-refire]] — **The Gmail Action Digest ran 1 h 55 m and fired six times on 2026-10-25.** The 08:30 slot started 08:31:14 and closed **10:26:37 `failed`** on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed)` — against **91 s** (10-23) and **104 s** (10-24). The same job then fired five more times (17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53); the 19:18:41 fire died on a **new signature** — `Cron admission missing: the owner never admitted this fire` (distinct from the 10-20 orphan) — and all six fires, failures included, carry `delivery_outcome: delivered`, so up to **six digests reached Anthony**. Neither failure raised a `cron_incidents` row; the job now reads `last_status: delivery_queued` with an **empty** `last_error`. Hypothesis (unconfirmed): the 08:31 fire **adopted an unrelated long-lived session** — its worker made **70 `terminal` calls on VM101 `/dev/sdb` backup work** with context growing 66 k → 148 k tokens, while **context compression was unavailable all day** (`invalid_grant`). → [[systems/scheduled-tasks]], [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] — **`Invalid refresh token` — Nous Portal runtime credentials are dead, so context summarisation failed.** Gateway-log counts: **1** line 2026-10-22, **3** on 10-24, **20** on 10-25 (last 14:53:04, `agent.context_compressor: Failed to generate context summary`), **0** since the 19:27:13 restart. ⚠ **Masked, not fixed:** at **2026-10-25 19:13** all **eight** `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) moved off `nous` to **`no-think/freeapi/auto` on `custom:omni-101`** (fallback `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, `free_only: true`; compression timeout 90 s, vision 180 s). No `hermes model` re-authentication has been run and **`nous` is still the first entry in `fallback_providers`**, so failover tries a dead credential. → [[systems/model-providers]], [[systems/current-profile]]
|
||||
- [[incidents/2026-10-24-conduit-call-request-refused]] — **The agent refused to phone Anthony, three times, on 2026-10-24 (12:53–12:56).** He asked to be called to test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`); all three replies asserted the agent has **no calling capability**, with **zero tool calls** in the session. `conduit_call_user` (toolset `conduit`) exists, is registered for every session, is reachable via `tool_search`, and the plugin ships the `calling-the-user` skill; his call settings (`enabled` / `when_asked` / `decides`) were permissive and the device was paired. **Root cause: capability asserted instead of discovered** — the first reply came from `kilo-auto/free`, but the switch to `smart-route` repeated it. Same audit also **corrects the 10-22 incident's closing evidence**: `conduit-calls.json → ends` records turn-ends that fire a watch, not calls, so “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne **cron** sessions — `history` (real calls) is **empty** and no call has been placed since the memory fix. → [[systems/conduit]], [[incidents/2026-10-22-conduit-memory-start-timeout]]
|
||||
- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]]
|
||||
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]]
|
||||
|
||||
@@ -1264,3 +1264,49 @@ No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe st
|
||||
still **2** (`0e753e_e7f8194a7d06`, `c0a9fa_764558b5ae83`) — identical to the page's 2026-10-25
|
||||
entry. No drift, nothing to change.
|
||||
- **No secrets written.** Job IDs, incident IDs, IPs, counts, paths and timestamps only.
|
||||
|
||||
## [2026-10-26] audit | Nightly wiki audit — window 2026-10-25 00:00 → 2026-10-26 00:5x AWST: a runaway digest and a dead Nous credential
|
||||
|
||||
- **Fleet was quiet but one job was not.** Live `executions.db`: 2026-10-25 settled **172 `completed` / 2 `failed`**;
|
||||
**both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times** in the day (08:31:14 →
|
||||
10:26:37 runaway at **1 h 55 m** vs 91 s / 104 s the two prior days; then 17:19:36, 18:40:06, 19:18:41, 19:30:31,
|
||||
19:38:53). New signature on the 19:18:41 fire: **`Cron admission missing: the owner never admitted this fire`**.
|
||||
All six fires — failures included — carry `delivery_outcome: delivered`. **Neither failure created a `cron_incidents`
|
||||
row**; the page's open-incident count is unchanged at 2. → new `incidents/2026-10-25-gmail-digest-runaway-and-refire.md`
|
||||
- **Root cause of the missing context compression found: the Nous Portal credential is dead.**
|
||||
`Nous Portal runtime credentials unavailable: Invalid refresh token` — **1** line 2026-10-22, **3** on 10-24,
|
||||
**20** on 10-25 (last 14:53:04, `agent.context_compressor: Failed to generate context summary`), **0** since the
|
||||
19:27:13 restart. → new `incidents/2026-10-22-nous-portal-refresh-token-invalid.md`
|
||||
- **⚠ The global model default moved BACK to OmniRoute — and the last three audits recorded the stale value.**
|
||||
A 2026-10-24 19:06:10 switch note reads *"model was just switched from meituan/longcat-2.5-preview:free to
|
||||
`smart-route` via omni-101"*; live config confirms `custom:omni-101` / `smart-route` (`api_mode:
|
||||
chat_completions`), `config.yaml` last written 2026-10-25 19:13. A **six-provider top-level `fallback_providers`
|
||||
chain is live again** (headed by the dead `nous` credential), so the 2026-10-21 "no fallback chain" record is
|
||||
stale too. Consequence: the 8 unpinned jobs are back on OmniRoute → **the fleet is a single upstream path again**.
|
||||
Corrected on `systems/current-profile.md`, `systems/model-providers.md`, `concepts/model-routing.md`,
|
||||
`comparisons/model-routers.md`, `systems/omniroute.md`.
|
||||
- **Auxiliary routing captured.** At 2026-10-25 19:13 **all eight** `auxiliary:` slots (approval, compression,
|
||||
curator, mcp, profile_describer, skills_hub, title_generation, vision) moved off `nous` to
|
||||
**`no-think/freeapi/auto` on `custom:omni-101`** with `free_only: true` and a `freeapi/deepseek-v4-flash` →
|
||||
`kg/kilo-auto/free` chain. ⚠ Masking, not fixing — `nous` remains first in `fallback_providers`. New section on
|
||||
`systems/model-providers.md`.
|
||||
- **`systems/scheduled-tasks.md` — new 2026-10-26 entry added**; the 2026-10-25 entry marked superseded. Fleet
|
||||
unchanged (28 jobs / 25 enabled / 3 disabled, all 28 IDs present); non-clean `last_status` **4 → 5**
|
||||
(`08eb86f02d37` joined as `delivery_queued` with an empty `last_error`); pins unchanged.
|
||||
- **`systems/conduit.md` re-probed live.** Version **`0.69.2`** in `plugin.yaml` *and* `dashboard/manifest.json`
|
||||
(both), HEAD **`942ddb8`** (2026-10-24 15:35:42, *"Pulse: list candidates, decide in code, journal them (#342)"*),
|
||||
**0 behind `origin/main`**, 24 commits on since the `0.64.0` baseline, working tree still carrying the
|
||||
uncommitted `dashboard/plugin_api.py` patch. **The dashboard gateway restarted 2026-10-25 18:42:33, so the
|
||||
upstream commits are now live** — the previous "not live" note is superseded. `conduit-calls.json → history`
|
||||
is still **empty**; `conduit_show` ran once on 10-25 14:37. Status on
|
||||
`incidents/2026-10-24-conduit-call-request-refused.md` **stays open** (untested, not fixed).
|
||||
- **`systems/messaging-integrations.md`** — CT255 still `stopped` (Mattermost offline **36 d**, `tx 312 / rx 0`);
|
||||
268 timeouts on 10-25 vs 264 on 10-24; reconnect counter reset to 54 by the 19:27 restart (per-process, not
|
||||
outage duration).
|
||||
- **`systems/hermes-agent.md`** — gateway restart record added (three starts on 2026-10-25: 00:32:20, 18:30:04,
|
||||
19:27:13; pid 2476414 up since 19:26:33) plus the credential fault and its restart-resets-counters caveat.
|
||||
- **`index.md`** — header date, the two new incident links, and rewritten `conduit`, `messaging-integrations` and
|
||||
`scheduled-tasks` summaries.
|
||||
- **No secrets written.** Job IDs, incident IDs, base URLs, counts, paths and timestamps only.
|
||||
- **Not changed:** `current-state.md` (its dated sections are historical records), and the dead `custom:omni`
|
||||
provider entry (still named by 2 jobs — Anthony's call, not the wiki's).
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
# Wiki freshness audit — 2026-10-26
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (8)
|
||||
|
||||
- [ 22d] agents/hermes-chaos.md (updated 2026-10-04)
|
||||
- [ 22d] agents/hermes-production.md (updated 2026-10-04)
|
||||
- [ 22d] incidents/2026-10-03-ct450-hermes-update.md (updated 2026-10-04)
|
||||
- [ 22d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md (updated 2026-10-04)
|
||||
- [ 22d] infrastructure/hosts.md (updated 2026-10-04)
|
||||
- [ 22d] runbooks/provider-health.md (updated 2026-10-04)
|
||||
- [ 22d] runbooks/update-hermes-safely.md (updated 2026-10-04)
|
||||
- [ 22d] systems/memory-backend.md (updated 2026-10-04)
|
||||
|
||||
## FRESH (95)
|
||||
|
||||
- [ 6d] CONTRIBUTING.md
|
||||
- [ 6d] agents/index.md
|
||||
- [ 15d] agents/nanobot.md
|
||||
- [ 6d] agents/open-claw.md
|
||||
- [ 15d] agents/qwenpaw.md
|
||||
- [ 6d] agents/zeroclaw.md
|
||||
- [ 6d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 6d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 6d] comparisons/memory-backends.md
|
||||
- [ 6d] comparisons/model-routers.md
|
||||
- [ 13d] comparisons/steel-vs-camoufox.md
|
||||
- [ 6d] concepts/agent-memory.md
|
||||
- [ 13d] concepts/browser-automation.md
|
||||
- [ 6d] concepts/model-routing.md
|
||||
- [ 6d] concepts/persistent-shells.md
|
||||
- [ 6d] concepts/tool-calling.md
|
||||
- [ 1d] current-state.md
|
||||
- [ 6d] decisions/index.md
|
||||
- [ 6d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 6d] decisions/persistent-shell-enabled.md
|
||||
- [ 6d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 6d] experiments/index.md
|
||||
- [ 2d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 6d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 2d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 4d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 3d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 14d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 1d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 6d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 5d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 15d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 20d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 20d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 2d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 17d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 10d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 4d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 7d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 5d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 4d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 1d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 0d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 0d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 0d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 4d] infrastructure/backups.md
|
||||
- [ 6d] infrastructure/credentials-map.md
|
||||
- [ 15d] infrastructure/docker-services.md
|
||||
- [ 6d] infrastructure/domains-and-tunnels.md
|
||||
- [ 12d] infrastructure/network-map.md
|
||||
- [ 12d] infrastructure/proxmox.md
|
||||
- [ 1d] infrastructure/tailscale.md
|
||||
- [ 14d] queries/retained-research-answers.md
|
||||
- [ 1d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 5d] reports/uber-eats/2026-07.md
|
||||
- [ 5d] reports/uber-eats/2026-08.md
|
||||
- [ 5d] reports/uber-eats/2026-09.md
|
||||
- [ 5d] reports/uber-eats/index.md
|
||||
- [ 6d] runbooks/backup-wiki.md
|
||||
- [ 6d] runbooks/diagnose-docker.md
|
||||
- [ 6d] runbooks/gateway-resource-resilience.md
|
||||
- [ 13d] runbooks/job-radar.md
|
||||
- [ 6d] runbooks/multiplexer-setup.md
|
||||
- [ 6d] runbooks/recover-docker-service.md
|
||||
- [ 6d] runbooks/restart-browser.md
|
||||
- [ 6d] runbooks/restart-hermes.md
|
||||
- [ 6d] runbooks/restore-openrouter.md
|
||||
- [ 6d] runbooks/test-openrouter.md
|
||||
- [ 6d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 13d] systems/browser-backend.md
|
||||
- [ 6d] systems/cloudflare-tunnel.md
|
||||
- [ 0d] systems/conduit.md
|
||||
- [ 0d] systems/current-profile.md
|
||||
- [ 6d] systems/guanaco.md
|
||||
- [ 6d] systems/headless-server.md
|
||||
- [ 0d] systems/hermes-agent.md
|
||||
- [ 6d] systems/hermes-ui.md
|
||||
- [ 14d] systems/home-assistant.md
|
||||
- [ 1d] systems/homepage-dashboard.md
|
||||
- [ 15d] systems/ignis.md
|
||||
- [ 0d] systems/messaging-integrations.md
|
||||
- [ 2d] systems/mnemosyne.md
|
||||
- [ 0d] systems/model-providers.md
|
||||
- [ 14d] systems/nanobot.md
|
||||
- [ 6d] systems/nextcloud.md
|
||||
- [ 6d] systems/obsidian.md
|
||||
- [ 4d] systems/omniroute.md
|
||||
- [ 6d] systems/opencode-go.md
|
||||
- [ 6d] systems/openrouter.md
|
||||
- [ 15d] systems/paseo.md
|
||||
- [ 7d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 6d] systems/terminal-backend.md
|
||||
- [ 6d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (4)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- index.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 8 stale, 95 fresh, 4 no-date of 107 pages
|
||||
+14
-5
@@ -3,8 +3,8 @@ title: Conduit
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-10-24
|
||||
updated: 2026-10-25
|
||||
verified_on: 2026-10-25
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
review_after: 2026-11-08
|
||||
confidence: high
|
||||
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling]
|
||||
@@ -21,9 +21,17 @@ be started from the phone where the agent can speak and listen (`conduit-voice`
|
||||
Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[systems/hermes-ui]]
|
||||
(the browser desktop), [[systems/push-notifications]] (the older `hermes-push` bridge).
|
||||
|
||||
## Runtime checks (2026-10-26 00:1x AWST)
|
||||
- **Working tree:** M dashboard/plugin_api.py — the uncommitted memory-timeout patch is **still present**.
|
||||
- **`conduit-calls.json` → `history` entries:** **0** — the call-request path remains unused
|
||||
→ [[incidents/2026-10-24-conduit-call-request-refused]].
|
||||
- **Tool use in-window:** `conduit_show` ×1 — the plugin works for cards; the open incident concerns *placing calls*.
|
||||
|
||||
## Where it lives
|
||||
- **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push`
|
||||
- **Version:** `0.69.0` (`plugin.yaml` and `dashboard/manifest.json`), git HEAD `f000bf0` —
|
||||
- **Version (re-verified 2026-10-26):** `0.69.2` (plugin.yaml) / `0.69.2` (dashboard manifest), git HEAD `942ddb8` — *Plugin and relay fixes left over from earlier reviews (0.69.2) (#342)*, committed 2026-10-24 15:35:42 +0000. **0 commits behind `origin/main`**; **24 commits on from the `0.64.0` / `eaa14be` baseline** of the 2026-10-24 audit (upstream `#324`–`#342`: Pulse cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway restart, `channels.yaml`, App Store builds following the archived plugin version).
|
||||
- **⚠ Superseded provenance:** the 2026-10-24 audit read `0.69.0` / `f000bf0`; two further commits landed on
|
||||
2026-10-24 15:35. Git HEAD was previously recorded as: `- **Version:** `0.69.0` (`plugin.yaml` and `dashboard/manifest.json`), git HEAD `f000bf0` —
|
||||
*“Pulse: list candidates, decide in code, journal them (#337)”*. **22 commits on from the
|
||||
`0.64.0` / `eaa14be` recorded at the 2026-10-24 audit** (upstream `#324`–`#338`: Pulse
|
||||
cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway
|
||||
@@ -41,8 +49,9 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste
|
||||
`conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`.
|
||||
|
||||
## Current state (verified 2026-10-25 00:2x–00:5x AWST)
|
||||
- **Dashboard gateway:** listening on `0.0.0.0:9119`; process started **2026-10-23 18:54:43** and
|
||||
**not restarted since** — so it is still the process that picked up the memory-timeout patch, and
|
||||
- **Dashboard gateway (re-verified 2026-10-26):** listening on `0.0.0.0:9119`; process started **2026-10-25
|
||||
18:42:33** — **restarted since the 2026-10-24 entry, so the upstream commits are now live** (the recorded
|
||||
*"not restarted since / not live"* note is superseded, not wrong at the time) — so it is still the process that picked up the memory-timeout patch, and
|
||||
the `0.64.0` → `0.69.0` upstream commits are **not** live.
|
||||
- **Device:** one registered — **iPhone** (`id 70aa63f3…`, `kid c24e8215…`), app version `2.0.0`,
|
||||
features `chat-cards`, `info-cards`, `live-cards`, `maps-navigate`, `media-cards`, `show-cards`;
|
||||
|
||||
+32
-22
@@ -3,9 +3,9 @@ title: Current Hermes Profile
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-21
|
||||
verified_on: 2026-10-21
|
||||
review_after: 2026-10-28
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
review_after: 2026-11-09
|
||||
confidence: high
|
||||
tags: [system, hermes, profile]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026-10-21]
|
||||
@@ -18,27 +18,37 @@ sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026
|
||||
- **Location:** implicit — the top-level `~/.hermes/config.yaml`. There is **no** `~/.hermes/profiles/default/` directory (verified 2026-10-02); `~/.hermes/profiles/` holds only named sub-profiles.
|
||||
- **Other profiles present:** `ops` only (`~/.hermes/profiles/ops/`) — the Tusk elephant persona, with its own SOUL.md, skills, plugins and memories (last modified 2026-09-30). Other profiles do **not** affect this session; see the profile-isolation note below.
|
||||
|
||||
## Key Settings (model routing re-verified against live config 2026-10-21; remaining items as recorded 2026-10-02)
|
||||
## Key Settings (model routing + auxiliary slots re-verified against live config 2026-10-26; remaining items as recorded 2026-10-02)
|
||||
|
||||
### ⚠ Model default moved OFF OmniRoute (2026-10-20)
|
||||
- **Provider:** **`nous`** — `model.base_url: https://inference-api.nousresearch.com/v1`, `api_mode: chat_completions`.
|
||||
- **Model default:** **`meituan/longcat-2.5-preview:free`**
|
||||
- **When:** `config.yaml` last written **2026-10-20 22:01:38 AWST**. The move was **not** in place at the
|
||||
2026-10-09 19:42 snapshot and **was** in place by 2026-10-20 11:00. Previous values (2026-10-02 record):
|
||||
provider `custom` → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`, provider id `custom:omni-101`),
|
||||
default `smart-route`.
|
||||
- **The `model` block now carries only** `base_url`, `default`, `provider`, `api_mode`, `persist_switch_by_default`.
|
||||
**The 6-provider fallback chain recorded at the 2026-10-02 snapshot is gone** — there is no `fallbacks` key under
|
||||
`model`, so an unreachable `nous` is no longer covered by the old `custom` / `or-inclusionai_ling-2_6-1t` →
|
||||
`custom:litellm` → `litellm:4000` chain.
|
||||
- **⚠ Cron pins did NOT move with it.** **20 of the 28 scheduled jobs still pin OmniRoute explicitly**
|
||||
(`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). **The 8 jobs with no pin are
|
||||
the ones whose upstream changed** — they now call `nous` direct instead of the router. The fleet therefore
|
||||
runs **two upstream paths for the first time**. → [[systems/scheduled-tasks]]
|
||||
- **Routing posture:** no fallback chain under `model`; per-job pins are the only routing control that remains.
|
||||
### ⚠ Model default moved BACK ONTO OmniRoute (reverted 2026-10-24 19:06)
|
||||
- **Provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`),
|
||||
`api_mode: chat_completions`.
|
||||
- **Model default:** **`smart-route`** (paid alias — not a free combo).
|
||||
- **When:** a session switch note at **2026-10-24 19:06:10 AWST** reads *"model was just switched from
|
||||
meituan/longcat-2.5-preview:free to smart-route via omni-101."* The 2026-10-21 config snapshot still read
|
||||
`nous` / `meituan/longcat-2.5-preview:free`, so the revert landed between **2026-10-20 22:01** and
|
||||
**2026-10-24 19:06**; `config.yaml` was last written **2026-10-25 19:13**. ⚠ The 2026-10-22, 10-24 and
|
||||
10-25 audits all recorded the 10-20 move as current — this section corrects them. The revert coincides with
|
||||
the **invalid Nous Portal refresh token** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
- **The `model` block carries** `base_url`, `default`, `provider`, `api_mode`, `persist_switch_by_default` —
|
||||
still **no `fallbacks` key under `model`**.
|
||||
- **⚠ A top-level six-provider `fallback_providers` chain is live again** (re-verified 2026-10-26):
|
||||
`nous` / `meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` →
|
||||
`custom:guacano` / `deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` →
|
||||
`kilocode` / `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6-1t`.
|
||||
The 2026-10-21 record that *no* fallback chain exists is **stale**. ⚠ The first entry is the **dead `nous`
|
||||
credential** — a fallover would try it first.
|
||||
- **⚠ Cron pins did NOT need to move.** **18 of the 28 scheduled jobs pin OmniRoute explicitly**
|
||||
(`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). The 8 unpinned jobs follow
|
||||
the global default, which is OmniRoute again — so the fleet is back to a **single upstream path**, not the
|
||||
two-path split recorded on 2026-10-21. → [[systems/scheduled-tasks]]
|
||||
- **Routing posture:** top-level `fallback_providers` chain + per-job pins are the routing controls;
|
||||
`model.fallbacks` does not exist.
|
||||
- ⚠ The `custom:omni` provider key is **no longer configured** (only 2 jobs still name it) — see
|
||||
[[systems/model-providers]].
|
||||
|
||||
### Other settings (recorded 2026-10-02 — not re-verified 2026-10-21)
|
||||
- **Auxiliary vision model:** `stepfun/step-3.7-flash:free` on `nous`
|
||||
### Other settings (recorded 2026-10-02 — auxiliary line re-verified 2026-10-26)
|
||||
- **Auxiliary slots (all 8 — `approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`, `title_generation`, `vision`):** `no-think/freeapi/auto` on `custom:omni-101`; per-slot fallbacks `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`; `free_only: true`; compression timeout 90 s, vision timeout 180 s. Re-placed 2026-10-25 19:13 — previously vision was `stepfun/step-3.7-flash:free` on `nous`. → [[systems/model-providers]]
|
||||
- **Switch persistence:** `persist_switch_by_default: false`
|
||||
- **Timezone:** `Australia/Perth`
|
||||
- **Approvals mode:** `off` (`destructive_slash_confirm: false`, `mcp_reload_confirm: false`)
|
||||
|
||||
+19
-2
@@ -3,8 +3,8 @@ title: Hermes Agent
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
confidence: high
|
||||
tags: [system, hermes, agent, llm]
|
||||
sources: [live-hermes-version, live-config, live-skill-inventory]
|
||||
@@ -169,3 +169,20 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
|
||||
- **Mattermost platform unreachable** since 2026-10-10; the gateway keeps retrying on a 5-minute
|
||||
cadence - [[systems/messaging-integrations]].
|
||||
- **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]].
|
||||
|
||||
## 2026-10-26 00:1x AWST — gateway restart record and the dead Nous credential
|
||||
|
||||
- **Three gateway starts on 2026-10-25**: `00:32:20`, `18:30:04`, `19:27:13` AWST (after `2026-10-24 12:24:39`).
|
||||
The `18:29:22` shutdown carried `signal=SIGTERM under_systemd=yes`, `loadavg_1m=3.79` — a planned stop, not a
|
||||
crash; `19:26:16` logged *"Stopping gateway for restart..."* (the plugin self-update path). Current gateway
|
||||
pid **2476414**, up since **19:26:33** (≈4 h 45 m at audit time).
|
||||
- **Consequences of the 19:27 start:** the Conduit dashboard gateway restarted at 18:42:33 and so picked up the
|
||||
plugin's upstream commits → [[systems/conduit]]; the Mattermost reconnect attempt counter reset to 54
|
||||
→ [[systems/messaging-integrations]]; the `invalid_grant` spam stopped.
|
||||
- ⚠ **The Nous Portal credential is dead** — `Nous Portal runtime credentials unavailable: Invalid refresh token`.
|
||||
Gateway-log counts: **1 line 2026-10-22, 3 on 10-24, 20 on 10-25** (last 14:53:04,
|
||||
`agent.context_compressor: Failed to generate context summary`), **0 since the 19:27:13 start**. Context
|
||||
compression was unavailable for most of 10-25 and contributed to the digest runaway.
|
||||
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
- **Restart hygiene:** a gateway restart resets in-process counters (Mattermost retries, cron admission
|
||||
bookkeeping). Never read a counter as an outage duration.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Messaging Integrations
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-19
|
||||
verified_on: 2026-10-19
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
confidence: high
|
||||
tags: [messaging, telegram, discord, email, integrations]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
|
||||
@@ -113,3 +113,15 @@ This pass replaced the 10-11 observation's "cause not determined" with a definit
|
||||
|
||||
**Open item:** start CT255, or retire the platform block. The 5-minute reconnect loop continues until
|
||||
one of those happens — restarting the adapter while the host is stopped achieves nothing.
|
||||
|
||||
## 2026-10-26 00:1x AWST — Mattermost still down; attempt counter reset by the 10-25 restart
|
||||
|
||||
- **Cause unchanged: CT255 (`mattermost`) is `stopped`.** `tailscale status` shows the peer **offline, last seen
|
||||
36 d ago**, `tx 312 / rx 0` — `rx` has read 0 in every audit since 2026-10-11, i.e. no packet has come back.
|
||||
- **Reconnect loop still running:** **268** `mattermost connect timed out` lines on 2026-10-25 vs **264** on
|
||||
2026-10-24 (≈1 per 5 min, as designed); **2** so far on 2026-10-26.
|
||||
- **The attempt counter reset to 54** with the 19:27:13 gateway restart (`Reconnecting mattermost (attempt 54)`
|
||||
at 2026-10-26 00:07:06). The previous reset was the 2026-10-17 05:10 restart. ⚠ The counter measures **this
|
||||
process's** attempts, not the outage's length — do not quote it as downtime.
|
||||
- **Impact unchanged:** messaging path only; no cron job delivers via Mattermost.
|
||||
- **Not attempted:** `pct start 255` — starting a service on the PVE host is Anthony's call.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Model Providers
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-23
|
||||
verified_on: 2026-10-23
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
confidence: high
|
||||
tags: [model, provider, inference, routing]
|
||||
sources: [live-probe, config, web-search]
|
||||
@@ -20,27 +20,56 @@ Verified from live `~/.hermes/config.yaml` → `custom_providers`, 2026-10-01.
|
||||
|
||||
| Provider | URL | Models | Role |
|
||||
|---|---|---|---|
|
||||
| `custom:omni-101` | `http://100.78.63.20:20128` | 5,384 | **Primary.** OmniRoute on the estate (`omni-101`). 15 of 30 cron jobs reference it |
|
||||
| `custom:omni-101` | `http://100.78.63.20:20128` | 5,384 | **Primary.** OmniRoute on the estate (`omni-101`). 18 of 28 cron jobs reference it (2026-10-26) |
|
||||
| `custom:Omniroute:20128` | `http://omniroute:20128/v1` | 5,398 | Secondary OmniRoute host — legacy naming, kept alongside `omni-101` |
|
||||
| `custom:litellm` | `http://litellm:4000/v1` | 1,728 | Fallback inference (slower path, 6–26s/call) |
|
||||
| `custom:command-code` | `https://api.commandcode.ai/provider/v1` | 71 | DeepSeek Flash primary via the `deepseek-flash-mix` combo (added 2026-08-19); also the source of the `MODEL_NOT_IN_PLAN` 403 on `gpt-5.6-terra` |
|
||||
| `custom:guacano` | `http://ollama:11000/v1` | 46 | Guanaco LLM proxy/router on CT205 (Ollama stack). See [[systems/guanaco]] |
|
||||
| `custom:omni` | *(removed)* | — | **No longer configured.** It once served auxiliary roles (vision, skills, approval, mcp, web_extract, compression). Two cron jobs still name it as their provider key — see [[systems/scheduled-tasks]] |
|
||||
|
||||
**Built-in (non-`custom_providers`) routes still in use as fallbacks:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`. `charm-hyper` (Charmbracelet, `hyper.charm.land/v1`, free tier 100 credits/mo) is no longer in the live stack.
|
||||
**Built-in (non-`custom_providers`) routes still in use as fallbacks:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`. `charm-hyper` (Charmbracelet, `hyper.charm.land/v1`, free tier 100 credits/mo) is no longer in the live stack. **The live `fallback_providers` chain (2026-10-26) is `nous` / `meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` → `custom:guacano` / `deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` → `kilocode` / `poolside/laguna-m.1:free` → `custom` / `or-inclusionai_ling-2_6-1t`.**
|
||||
|
||||
**Default provider:** **`nous`** → `https://inference-api.nousresearch.com/v1`. **Default model:** **`meituan/longcat-2.5-preview:free`**.
|
||||
The default moved OFF OmniRoute on 2026-10-20 — see below. The `custom` provider key is no longer the default.
|
||||
**Default provider:** **`custom:omni-101`** → `http://100.78.63.20:20128`. **Default model:** **`smart-route`**.
|
||||
⚠ **The default moved BACK to OmniRoute on 2026-10-24 19:06:10 AWST** (session switch note: *"model was just
|
||||
switched from meituan/longcat-2.5-preview:free to smart-route via omni-101"*). The `nous` direct window ran
|
||||
2026-10-20 22:01 → 2026-10-24 19:06 and the 2026-10-22/10-24/10-25 audit entries recorded it as current —
|
||||
**stale**. `config.yaml` last written 2026-10-25 19:13.
|
||||
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
|
||||
Provider names that appear in job configs but not in `custom_providers` do **not** necessarily fail: the default `custom` block resolves to the same base URL, so a stale provider key can go unnoticed. Prefer routing aliases (`smart-route`) over raw upstream model names in pins.
|
||||
|
||||
## Default Model
|
||||
- **`meituan/longcat-2.5-preview:free`** on **`nous`** (`https://inference-api.nousresearch.com/v1`, `api_mode: chat_completions`) — the live default since 2026-10-20. Verified against live `~/.hermes/config.yaml` 2026-10-23.
|
||||
- **No fallback chain under `model`.** The 6-provider fallback recorded at the 2026-10-02 snapshot is gone; `model` now carries only `base_url`, `default`, `provider`, `api_mode`, `persist_switch_by_default`. An unreachable `nous` has no automatic failover — per-job pins are the only routing control that remains.
|
||||
- **`smart-route`** on **`custom:omni-101`** (`http://100.78.63.20:20128`, `api_mode: chat_completions`) — the
|
||||
live default since **2026-10-24 19:06:10 AWST**, re-verified against live `~/.hermes/config.yaml`
|
||||
**2026-10-26 00:1x**.
|
||||
- **⚠ A six-provider top-level `fallback_providers` chain is live again** (re-verified 2026-10-26): `nous` /
|
||||
`meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` → `custom:guacano` /
|
||||
`deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` → `kilocode` /
|
||||
`poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6-1t`. The
|
||||
2026-10-21 claim that **no** fallback chain exists is **stale**; the `model` block itself still carries only
|
||||
`base_url`, `default`, `provider`, `api_mode`, `persist_switch_by_default`.
|
||||
- ⚠ **The first fallback entry is the dead `nous` credential** (`Invalid refresh token`, since 2026-10-22)
|
||||
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
- Anthony has explicitly **rejected** `free-lkgp` as a default; stable defaults are `smart-route` / `deepseek-paid-smart` (paid, not free)
|
||||
- `resilient-free` was the default after the 2026-08-14 OmniRoute factory reset, but that combo is now in the broken set
|
||||
- Previous default `auto/best-free` was misconfigured — had no combo definition and routed to a paid `gpt-5.6-sol`
|
||||
- **20 of 28 scheduled jobs still pin OmniRoute explicitly** (`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). The fleet now runs two upstream paths. → [[systems/scheduled-tasks]]
|
||||
- **18 of 28 scheduled jobs pin OmniRoute explicitly** (`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). The remaining 8 follow the global default, which is **also** OmniRoute — so the fleet is back to a **single upstream path**. → [[systems/scheduled-tasks]]
|
||||
|
||||
## Auxiliary Role Routing (re-verified live 2026-10-26)
|
||||
**All eight `auxiliary:` slots** — `approval`, `compression`, `curator`, `mcp`, `profile_describer`,
|
||||
`skills_hub`, `title_generation`, `vision` — are **`no-think/freeapi/auto` on `custom:omni-101`**, each with
|
||||
the chain `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` and `free_only: true`.
|
||||
**Compression timeout 90 s; vision timeout 180 s.**
|
||||
|
||||
- Re-placed **2026-10-25 19:13 AWST**, replacing the `nous` / `stepfun/step-3.7-flash:free` and `custom:omni`
|
||||
aux paths recorded on 2026-10-02 and 2026-10-25 (17:50).
|
||||
- ⚠ The move was made **because the Nous Portal credential is dead** (`Invalid refresh token`), so it **masks
|
||||
rather than fixes** that fault → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
- Probe results from that session (17:50–19:13, not re-run 2026-10-26): `freeapi/auto` healthy for text and
|
||||
vision; `kg/kilo-auto/free` returns **empty content on image input**; `stepfun/step-3.7-flash:free` → 401
|
||||
*no active credentials*; freeapi pinned models in cooldown (429). ⚠ A **vision fallback returning empty
|
||||
content** is a silent failure mode — check `freeapi/auto` itself, not the fallback order, if image calls
|
||||
go blank.
|
||||
|
||||
## Retired / Unavailable Models (2026-09-25 → 2026-10-01)
|
||||
Pins that fail at request time, with the exact upstream error. Each is an open cron-repin in [[systems/scheduled-tasks]].
|
||||
|
||||
@@ -47,7 +47,11 @@ sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
burst, not afterwards.
|
||||
|
||||
## Purpose
|
||||
Secondary provider/router for Hermes auxiliary roles and some agent routing.
|
||||
Provider/router for Hermes — ⚠ **updated 2026-10-26: the `omniroute-1` node is the DEFAULT chat path again.**
|
||||
Hermes's default provider is `custom:omni-101` → `http://100.78.63.20:20128` with model `smart-route`
|
||||
(reverted 2026-10-24 19:06; the 2026-10-20 → 10-24 window pointed the default at `nous`).
|
||||
The CT600 provider key `custom:omni` (`http://omniroute:20128/v1`) remains auxiliary-only.
|
||||
→ [[systems/current-profile]]
|
||||
|
||||
## Current Configuration
|
||||
- **Hermes provider name:** `custom:omni` (maps to provider `omniroute:20128`)
|
||||
@@ -56,7 +60,7 @@ Secondary provider/router for Hermes auxiliary roles and some agent routing.
|
||||
- **Runs on:** Proxmox CT600 (`omniroute`, Tailscale `100.88.81.19`, onboot=1) — verified running 2026-10-11
|
||||
- **DB path:** `/opt/omniroute/storage.sqlite` (was `/var/lib/omniroute/storage.sqlite`)
|
||||
- **Backup:** pre-migration config saved at `~/.hermes/config.yaml.bak.omniroute-removal`
|
||||
- **Miss:** Hermes default provider path does not use OmniRoute for main chat; used for auxiliary roles instead
|
||||
- **Correction (2026-10-26):** the earlier *"Hermes default provider path does not use OmniRoute for main chat"* note is **stale** — the default is OmniRoute again (`custom:omni-101`, model `smart-route`). The **CT600** key `custom:omni` is still auxiliary-only.
|
||||
- **Auxiliary roles in Hermes config:** `skills_hub`, `approval`, `mcp`
|
||||
- **Key prefix visible in config:** `sk-5c67fbdc0...` — full value not stored in wiki
|
||||
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-25
|
||||
verified_on: 2026-10-25
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
review_after: 2026-10-31
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page.
|
||||
- **Current as of 2026-10-26 00:1x AWST (live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`).
|
||||
- **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page. *(Superseded by the 2026-10-26 entry above.)*
|
||||
- **Current as of 2026-10-24 00:2x AWST *(superseded by the 2026-10-25 entry above)* (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]]
|
||||
- **Current as of 2026-10-23 00:1x AWST *(superseded by the 2026-10-24 entry above)* (`jobs.json.updated_at 2026-10-23T00:07:25+08:00`) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. **A DB recount by `started_at` and by `claimed_at` puts 2026-10-21 at 168 / 1**, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Five incidents closed in-window and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (`0e519874d280`, closed 08:01:40), `06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so **unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`: `1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, `c0a9fa_764558b5ae83`, `0e753e_e7f8194a7d06`). **Non-clean `last_status` falls 9 → 5:** `1b23c042e17d` (`error`, streak 2 — **next slot Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` (`delivery_failed` — Mon 10-26 03:00), `066bb5191280` (`delivery_failed` — 11-01). All four cleared rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its first slot of 10-22. **Pins:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (`provider`/`model` null → global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). ⚠ **Stocktake anomaly:** `1a8e90ea2271` ran **03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, finished `completed` with delivery `suppressed` — and **committed nothing**. It left its regenerated `systems/skills-index.md` uncommitted (mtime 10-22 03:00, with a `## .Archive (41)` section reading **347**) and its diagnosed fixes unapplied → [[incidents/2026-08-15-skills-index-drift]]. **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop` (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models `mimo-v2.5`, `or-inclusionai_ling-2_6-1t`, `minimax/minimax-m3.1-flash`), 2 `acp` (0 messages), 1 `oneshot` memory-capture probe. → [[systems/skills-index]], [[incidents/2026-08-15-skills-index-drift]]
|
||||
- **Current as of 2026-10-22 00:1x AWST *(superseded by the 2026-10-23 entry above)* (`jobs.json.updated_at 2026-10-22T00:11:31+08:00`) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen.** Window 2026-10-21: **171 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`), which died at 07:40:48 after **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string, which appears in no 2026-10-21 row → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` stays at 9** (7 `error` / 2 `delivery_failed`) but the composition moved: **`08eb86f02d37` cleared** (its 08:30 slot ran clean; incident `08eb86_472c7b609c80` **resolved 2026-10-21 08:37:21**) while **`dca8482e4f76` re-entered** (streak 1). Survivors, with next run: `dca8482e4f76` (1, **10-22 07:00**), `0e519874d280` (2, **10-22 08:00**), `06a0a1c8704f` (1, **10-22 10:00**), `c8d52904e1d6` (1, **10-22 18:00**), `1b23c042e17d` (2, Fri 10-23 08:00), `c0a9fa314542` (1, Mon 10-26 09:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 11-01 02:00). **Incident ledger: 1 opened / 1 closed, unclosed stays 9** (8 `alerted` / 1 `detected`). **Pins:** 17 of the 28 jobs pin `custom:omni-101` (10 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`), **7 carry no pin at all** (`provider`/`model` null → global default), 1 pins `custom:omni` / `smart-route`. **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. **12 sessions in the window, all `cron` — no interactive sessions.** ⚠ **Infrastructure changed in-window: Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]], [[infrastructure/backups]].
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-25
|
||||
verified_on: 2026-10-25
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
confidence: high
|
||||
tags: [skills, hermes]
|
||||
sources: []
|
||||
|
||||
Reference in New Issue
Block a user