Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4126093644 | ||
|
|
123a74cb0b |
@@ -3,8 +3,8 @@ title: Hermes Chaos
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-15
|
created: 2026-08-15
|
||||||
updated: 2026-10-04
|
updated: 2026-10-26
|
||||||
verified_on: 2026-10-04
|
verified_on: 2026-10-26
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, chaos, ct450, experimental, agent]
|
tags: [hermes, chaos, ct450, experimental, agent]
|
||||||
sources: [live-probe, config-inspection]
|
sources: [live-probe, config-inspection]
|
||||||
@@ -28,15 +28,15 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod
|
|||||||
## Current State
|
## Current State
|
||||||
|
|
||||||
- **Status:** Active (experimental)
|
- **Status:** Active (experimental)
|
||||||
- **Hermes version:** `v0.21.5+6170.ga510d6c` (2026.9.24), upstream `a510d6c6` — **updated from v0.19.0 on 2026-10-03**, first update since July; 3 commits behind upstream. Python 3.14.7. See [[incidents/2026-10-03-ct450-hermes-update]].
|
- **Hermes version:** `v0.22.3` (2026.9.24), local `5b147ab036d6` — **the same build as CT460** (updated 2026-10-21). **0 commits behind `origin/main`** (verified 2026-10-26; the old "3 commits behind" claim is stale), though `hermes update --check` reports stable **v0.22.5** available. Python 3.14.7. See [[incidents/2026-10-03-ct450-hermes-update]].
|
||||||
- **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]].
|
- **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]].
|
||||||
- **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01)
|
- **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01)
|
||||||
- **Duplicate cron jobs:** 11 paused (no post-pause execution records since 2026-08-01)
|
- **Cron jobs:** **25 — 13 enabled / 12 disabled** (verified live in CT450's `jobs.json` 2026-10-26). **9 job names are shared with CT460**; 8 of those are disabled here and enabled on CT460, while `Important-mail monitor` is the reverse (enabled on CT450, disabled on CT460)
|
||||||
- **Production counterparts:** CT460 has 11 enabled jobs with successful latest runs
|
- **Production counterparts:** CT460 has **28 jobs / 25 enabled** — the two fleets no longer share a job count, and the CT450 duplicates are paused
|
||||||
|
|
||||||
## Providers & Routing
|
## Providers & Routing
|
||||||
|
|
||||||
- **Default model:** `deepseek-flash-mix` via `custom:omniroute` (`http://100.88.81.19:20128/v1`, key via `$HERMES_CUSTOM_100_88_81_19_20128_API_KEY`)
|
- **Default model:** **`smart-route`** via `custom` → `http://100.78.63.20:20128` (the `omni-101` node) — verified live in CT450's `config.yaml` 2026-10-26. ⚠ The old claim of `deepseek-flash-mix` on `custom:omniroute` (`100.88.81.19`) no longer matches the config
|
||||||
- **Fallback chain** (fixed 2026-08-15 — OmniRoute fallback entries removed, they were non-functional):
|
- **Fallback chain** (fixed 2026-08-15 — OmniRoute fallback entries removed, they were non-functional):
|
||||||
1. `kilo-stepfun-step-3_7-flash-free` via `litellm_4000`
|
1. `kilo-stepfun-step-3_7-flash-free` via `litellm_4000`
|
||||||
2. `nvidia/nemotron-3-ultra:free` via `nous`
|
2. `nvidia/nemotron-3-ultra:free` via `nous`
|
||||||
|
|||||||
+11
-11
@@ -3,8 +3,8 @@ title: Hermes Production
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-15
|
created: 2026-08-15
|
||||||
updated: 2026-10-04
|
updated: 2026-10-26
|
||||||
verified_on: 2026-10-04
|
verified_on: 2026-10-26
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, production, ct460, agent]
|
tags: [hermes, production, ct460, agent]
|
||||||
sources: [live-probe, config-inspection]
|
sources: [live-probe, config-inspection]
|
||||||
@@ -19,15 +19,15 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
|||||||
## Location
|
## Location
|
||||||
|
|
||||||
- **Container:** LXC CT460 (`hermes`)
|
- **Container:** LXC CT460 (`hermes`)
|
||||||
- **Tailscale IP:** `100.96.100.82`
|
- **Tailscale IP:** `100.118.5.51` (⚠ `100.96.100.82` is the **PVE hypervisor** `pve`, not CT460 — corrected 2026-10-26)
|
||||||
- **Hostname:** `hermes`
|
- **Hostname:** `hermes`
|
||||||
- **Port:** 8082 (HTTP), 9119 (gateway)
|
- **Ports:** 9119 (gateway/API), 8085 (webhook) — ⚠ the old **8082** claim was dead: no 8082 listener on CT460 as of 2026-10-26
|
||||||
- **Memory:** 14GB (upgraded from 5096MB on 2026-08-07)
|
- **Memory:** 14GB (upgraded from 5096MB on 2026-08-07)
|
||||||
|
|
||||||
## Current State
|
## Current State
|
||||||
|
|
||||||
- **Status:** Active
|
- **Status:** Active
|
||||||
- **Hermes version:** `v0.21.5+6169.gab86a20` (2026.9.24), upstream `ab86a20a`; install tree rebuilt 2026-10-03 10:11 AWST — verified live 2026-10-04
|
- **Hermes version:** `v0.22.3` (2026.9.24), local `5b147ab036d6` — updated 2026-10-21 22:47→22:57 AWST; **verified live 2026-10-26**. (`hermes update --check` reports stable **v0.22.5** available)
|
||||||
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
|
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
|
||||||
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
|
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
|
||||||
- **Config:** `~/.hermes/config.yaml`
|
- **Config:** `~/.hermes/config.yaml`
|
||||||
@@ -43,10 +43,10 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
|||||||
|
|
||||||
## Providers & Routing
|
## Providers & Routing
|
||||||
|
|
||||||
- **Default provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute) — as attested 2026-09-30; the `custom:omni` provider key no longer exists
|
- **Default provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`, CT101/m93p) — verified live 2026-10-26. The `custom:omni` provider key does not exist in `custom_providers`, but **2 cron jobs still pin it** (see [[systems/scheduled-tasks]])
|
||||||
- **Default model:** `smart-route` (previously `freellmapi-auto`)
|
- **Default model:** `smart-route` (previously `freellmapi-auto`)
|
||||||
- **Fallback providers:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`, `custom / litellm:4000`
|
- **Fallback providers** (live chain, re-verified 2026-10-26): `nous` / `meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` → `custom:guacano` / `deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` → `kilocode` / `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6_1t`. ⚠ The **first** entry is the dead `nous` credential — a fallover tries it first
|
||||||
- **Auxiliary curator model:** `openai-codex / gpt-5.6-terra`
|
- **Auxiliary models:** all eight `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) are **`no-think/freeapi/auto` on `custom:omni-101`** with a `freeapi/deepseek-v4-flash` → `kg/kilo-auto/free` fallback chain (moved 2026-10-25 19:13)
|
||||||
- See [[systems/model-providers]] for full breakdown.
|
- See [[systems/model-providers]] for full breakdown.
|
||||||
|
|
||||||
## Memory
|
## Memory
|
||||||
@@ -58,7 +58,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
|||||||
|
|
||||||
## Scheduled Tasks
|
## Scheduled Tasks
|
||||||
|
|
||||||
- **Cron jobs:** 30 (27 enabled / 3 disabled; **6 failing** as of 2026-10-04)
|
- **Cron jobs:** **28 — 25 enabled / 3 disabled** (verified live `jobs.json` 2026-10-26). **5 non-clean `last_status`**: `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`), `08eb86f02d37` (`delivery_queued`, empty `last_error`), `c0a9fa314542` (`error`, streak 1)
|
||||||
- Delivery targets: Telegram (primary), Discord, origin, local
|
- Delivery targets: Telegram (primary), Discord, origin, local
|
||||||
- Notable: Birthday checker, job radar, newsletter digests, mnemosyne health/watchdog/backup
|
- Notable: Birthday checker, job radar, newsletter digests, mnemosyne health/watchdog/backup
|
||||||
- See [[systems/scheduled-tasks]] for full table.
|
- See [[systems/scheduled-tasks]] for full table.
|
||||||
@@ -66,7 +66,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
|||||||
## Messaging
|
## Messaging
|
||||||
|
|
||||||
- Telegram, Discord, Slack, Email, Mattermost, ntfy
|
- Telegram, Discord, Slack, Email, Mattermost, ntfy
|
||||||
- **Mattermost is currently unreachable** — the gateway logs `mattermost connect timed out after 30s, next retry in 300s` (live, 2026-10-04) and the Mattermost host does not answer. Treat Mattermost delivery as down until probed.
|
- **Mattermost is currently unreachable** — the gateway logs `mattermost connect timed out after 30s, next retry in 300s` (re-probed live 2026-10-26) and the Mattermost host does not answer: CT255 is `stopped` on `pve` and its Tailscale peer has been offline **36d**. Treat Mattermost delivery as down until probed.
|
||||||
- See [[systems/messaging-integrations]] for details.
|
- See [[systems/messaging-integrations]] for details.
|
||||||
|
|
||||||
## Key Differences from Chaos
|
## Key Differences from Chaos
|
||||||
@@ -74,7 +74,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
|||||||
| Aspect | Production (CT460) | Chaos (CT450) |
|
| Aspect | Production (CT460) | Chaos (CT450) |
|
||||||
|--------|-------------------|---------------|
|
|--------|-------------------|---------------|
|
||||||
| Purpose | Stable daily operations | Experimental/testing |
|
| Purpose | Stable daily operations | Experimental/testing |
|
||||||
| Cron jobs | 30 (27 enabled / 3 disabled) | 11 paused duplicates |
|
| Cron jobs | 28 (25 enabled / 3 disabled) | 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460 |
|
||||||
| Config changes | Requires verification | Safe to experiment |
|
| Config changes | Requires verification | Safe to experiment |
|
||||||
| Memory | Production Mnemosyne | Separate state |
|
| Memory | Production Mnemosyne | Separate state |
|
||||||
|
|
||||||
|
|||||||
@@ -69,14 +69,17 @@ Unified LLM router at `100.82.40.23:3001` (also `freellmapi.martinwa.org`). Veri
|
|||||||
- Exposes `/api/fallback/routing`, `/api/fallback/penalty-inspector`, `/api/fallback/token-usage`.
|
- Exposes `/api/fallback/routing`, `/api/fallback/penalty-inspector`, `/api/fallback/token-usage`.
|
||||||
This is the router that resolves Hermes's **default model** `freellmapi-auto` — distinct from OmniRoute's LKGP (below).
|
This is the router that resolves Hermes's **default model** `freellmapi-auto` — distinct from OmniRoute's LKGP (below).
|
||||||
|
|
||||||
### OmniRoute — fallback / auxiliary
|
### OmniRoute — ⚠ main chat path again since 2026-10-24 (was "fallback / auxiliary")
|
||||||
Not used for Hermes's main chat path but handles specific auxiliary roles (`skills_hub`, `approval`, `mcp`) and serves as the fallback provider. Has a much larger advertised catalog (~3430 vs ~1512) but that catalog can diverge from what's actually resolvable — combo discovery vs. resolution can differ (documented known issue). See [[systems/omniroute]].
|
**Re-verified 2026-10-26: this section is out of date.** Hermes's **default** is `custom:omni-101` /
|
||||||
|
`smart-route` on the `omniroute-1` node (`http://100.78.63.20:20128`) — i.e. OmniRoute **is** the main chat
|
||||||
|
path, as it was before the 2026-10-20 → 10-24 `nous` detour → [[systems/current-profile]].
|
||||||
|
It also still handles auxiliary roles and appears in the `fallback_providers` chain.` Has a much larger advertised catalog (~3430 vs ~1512) but that catalog can diverge from what's actually resolvable — combo discovery vs. resolution can differ (documented known issue). See [[systems/omniroute]].
|
||||||
|
|
||||||
### OpenRouter — kept only as a fallback decision, not wired in
|
### OpenRouter — kept only as a fallback decision, not wired in
|
||||||
**Verified:** absent from `custom_providers` in the live config. A decision page ([[decisions/keep-openrouter-as-fallback]]) records intent to retain it as a fallback option, but there's no live routing through it today — see [[systems/openrouter]] for the "previously used, largely replaced" history.
|
**Verified:** absent from `custom_providers` in the live config. A decision page ([[decisions/keep-openrouter-as-fallback]]) records intent to retain it as a fallback option, but there's no live routing through it today — see [[systems/openrouter]] for the "previously used, largely replaced" history.
|
||||||
|
|
||||||
## Verdict
|
## Verdict
|
||||||
Two routers are actually live (LiteLLM primary, OmniRoute fallback/auxiliary); OpenRouter is a retained *intent*, not a working path, as of 2026-07-22. If [[decisions/keep-openrouter-as-fallback]] is still desired, it needs an actual `custom_providers` entry — currently it's fallback-in-name-only.
|
Two routers are live (**OmniRoute primary** for the default path and auxiliary slots since 2026-10-24 19:06; **LiteLLM** as the deeper fallback), OpenRouter is a retained *intent*, not a working path — re-verified 2026-10-26. If [[decisions/keep-openrouter-as-fallback]] is still desired, it needs an actual `custom_providers` entry — currently it's fallback-in-name-only.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[systems/model-providers]]
|
- [[systems/model-providers]]
|
||||||
|
|||||||
@@ -18,12 +18,17 @@ How Hermes selects providers, applies combos, and enforces context limits.
|
|||||||
## Current Flow
|
## Current Flow
|
||||||
- Main path: `custom:litellm` → `http://litellm:4000/v1`
|
- Main path: `custom:litellm` → `http://litellm:4000/v1`
|
||||||
- Default model: `freellmapi-auto`
|
- Default model: `freellmapi-auto`
|
||||||
- OmniRoute used for auxiliary roles: vision, skills_hub, approval, mcp, title_generation, web_extract, compression
|
- OmniRoute used for the **default chat path** (`custom:omni-101` / `smart-route`, since 2026-10-24 19:06) and for **all eight auxiliary slots** (`approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`, `title_generation`, `vision` — all `no-think/freeapi/auto` as of 2026-10-25 19:13)
|
||||||
|
|
||||||
> **Live `model:` block re-read 2026-10-20 — the default has moved.**
|
> **Live `model:` block re-read 2026-10-26 — the default has moved TWICE.**
|
||||||
> `model.default` is now **`meituan/longcat-2.5-preview:free`** (provider `nous`), not
|
> `model.default` is **`smart-route`** on provider **`custom:omni-101`**
|
||||||
> `freellmapi-auto`. The block also carries `base_url`, `persist_switch_by_default` and `api_mode`.
|
> (`http://100.78.63.20:20128`), `api_mode: chat_completions`. It moved to `nous` /
|
||||||
|
> `meituan/longcat-2.5-preview:free` on 2026-10-20 22:01 and **back** on **2026-10-24 19:06:10**.
|
||||||
|
> A six-provider top-level `fallback_providers` chain is live again, headed by the **dead `nous`
|
||||||
|
> credential** (`Invalid refresh token` since 2026-10-22).
|
||||||
|
> → [[systems/current-profile]] · [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||||
>
|
>
|
||||||
|
|
||||||
> **The five `custom_providers` and their live base URLs:**
|
> **The five `custom_providers` and their live base URLs:**
|
||||||
>
|
>
|
||||||
> | Name | Base URL | Host |
|
> | Name | Base URL | Host |
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: CT450 Hermes Update to v0.21.5
|
|||||||
type: incident
|
type: incident
|
||||||
status: resolved
|
status: resolved
|
||||||
created: 2026-10-04
|
created: 2026-10-04
|
||||||
updated: 2026-10-04
|
updated: 2026-10-26
|
||||||
verified_on: 2026-10-04
|
verified_on: 2026-10-26
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, ct450, chaos, update, incident]
|
tags: [hermes, ct450, chaos, update, incident]
|
||||||
sources: [live-probe, pve-100.96.100.82, session-2026-10-03]
|
sources: [live-probe, pve-100.96.100.82, session-2026-10-03]
|
||||||
@@ -48,6 +48,9 @@ Two gotchas that cost time:
|
|||||||
- `clang` present; no stale locks present.
|
- `clang` present; no stale locks present.
|
||||||
- CT460 production untouched — separate config, skills, cron and memory state.
|
- CT460 production untouched — separate config, skills, cron and memory state.
|
||||||
|
|
||||||
|
## Current state (re-verified 2026-10-26)
|
||||||
|
Both instances have since moved on: **CT460 and CT450 now both run `v0.22.3` (2026.9.24), local `5b147ab036d6`** — the same build. CT450 is **0 commits behind `origin/main`** (the "3 commits behind" note below is superseded), though `hermes update --check` reports stable **v0.22.5** available on both. CT450's default model is now **`smart-route`** on `custom` → `http://100.78.63.20:20128`. The **CT450 Mattermost timeout remains** — CT255 is `stopped` on `pve` and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.
|
||||||
|
|
||||||
## Follow-ups
|
## Follow-ups
|
||||||
- CT450 reports `Update available: 3 commits behind`. Harmless; re-run `hermes update` if the chaos box should track upstream exactly.
|
- CT450 reports `Update available: 3 commits behind`. Harmless; re-run `hermes update` if the chaos box should track upstream exactly.
|
||||||
- CT450's Mattermost connection is failing (`mattermost connect timed out after 30s`, retried every 300s) — an infrastructure condition (Mattermost host unreachable), not an update regression. See [[agents/hermes-chaos]].
|
- CT450's Mattermost connection is failing (`mattermost connect timed out after 30s`, retried every 300s) — an infrastructure condition (Mattermost host unreachable), not an update regression. See [[agents/hermes-chaos]].
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Mnemosyne Vector Recall Ran FTS-Only
|
|||||||
type: incident
|
type: incident
|
||||||
status: resolved
|
status: resolved
|
||||||
created: 2026-10-04
|
created: 2026-10-04
|
||||||
updated: 2026-10-04
|
updated: 2026-10-26
|
||||||
verified_on: 2026-10-04
|
verified_on: 2026-10-26
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [mnemosyne, memory, embeddings, fastembed, recall, incident]
|
tags: [mnemosyne, memory, embeddings, fastembed, recall, incident]
|
||||||
sources: [live-probe, check_embedding_coverage.py, session-2026-10-03]
|
sources: [live-probe, check_embedding_coverage.py, session-2026-10-03]
|
||||||
@@ -49,6 +49,22 @@ Embeddings generated (24h): 13 (working: 7, episodic: 6)
|
|||||||
## Benign noise
|
## Benign noise
|
||||||
`[E:onnxruntime:Default, env.cc:228 ThreadMain] pthread_setaffinity_np failed for thread: … mask: {0,}` — a container CPU-affinity warning emitted on every model load. Not an error; do not chase it.
|
`[E:onnxruntime:Default, env.cc:228 ThreadMain] pthread_setaffinity_np failed for thread: … mask: {0,}` — a container CPU-affinity warning emitted on every model load. Not an error; do not chase it.
|
||||||
|
|
||||||
|
## Current state (re-verified 2026-10-26)
|
||||||
|
The 2026-10-03 follow-up is **satisfied**: `fastembed` 0.8.1 is importable in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`, `d41b16c8b6674454a4a819bd4acced1f`) as well as 0.8.0 in the legacy venv, and `FASTEMBED_CACHE_PATH` is pinned in `~/.hermes/.env`.
|
||||||
|
|
||||||
|
Current `check_embedding_coverage.py` output:
|
||||||
|
```
|
||||||
|
Memories with embeddings: 1899 / 2350 (80.8%)
|
||||||
|
Working memory: 815 / 842 (96.8%)
|
||||||
|
Episodic memory: 1084 / 1090 (99.4%)
|
||||||
|
vec_episodes_rowids: 1086
|
||||||
|
vec_working_rowids: 828
|
||||||
|
Embeddings generated (24h): 15 (working: 9, episodic: 6)
|
||||||
|
```
|
||||||
|
Both tiers are **above the 95 % working-coverage reindex threshold**, so the 6-hourly
|
||||||
|
monitor job `b63ed5817212` (`last_status: ok`) has nothing to fire on. The 2026-10-04
|
||||||
|
figures below are superseded.
|
||||||
|
|
||||||
## Follow-ups
|
## Follow-ups
|
||||||
- Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%.
|
- Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%.
|
||||||
- Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.
|
- Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.
|
||||||
|
|||||||
@@ -0,0 +1,48 @@
|
|||||||
|
---
|
||||||
|
title: Nous Portal Refresh Token Invalid — Context Summaries Unavailable
|
||||||
|
type: incident
|
||||||
|
status: open
|
||||||
|
created: 2026-10-26
|
||||||
|
updated: 2026-10-26
|
||||||
|
verified_on: 2026-10-26
|
||||||
|
review_after: 2026-11-09
|
||||||
|
confidence: high
|
||||||
|
tags: [hermes, router, memory, incident, unresolved]
|
||||||
|
sources: [gateway-log, live-config, live-probe]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Nous Portal Refresh Token Invalid — Context Summaries Unavailable (2026-10-22 →)
|
||||||
|
|
||||||
|
`Nous Portal runtime credentials unavailable: Invalid refresh token. Run `hermes model` to re-authenticate.`
|
||||||
|
|
||||||
|
## Occurrences (gateway logs, counted 2026-10-26)
|
||||||
|
| Day | Lines | Notable |
|
||||||
|
|---|---|---|
|
||||||
|
| 2026-10-22 | 1 | 11:32:20 `tui_gateway.server: llm.oneshot failed` |
|
||||||
|
| 2026-10-24 | 3 | — |
|
||||||
|
| 2026-10-25 | **20** | last at 14:53:04 — `agent.context_compressor: Failed to generate context summary` |
|
||||||
|
| 2026-10-26 | 0 (to 00:1x) | none since the 19:27:13 restart |
|
||||||
|
|
||||||
|
## Effect
|
||||||
|
- **Context compression failed outright.** Sessions could not summarise, so working context grew unchecked —
|
||||||
|
one cron fire reached **148 k tokens** on 10-25 and ran 1 h 55 m.
|
||||||
|
→ [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]
|
||||||
|
- Every auxiliary LLM path pointed at `nous` failed with it.
|
||||||
|
|
||||||
|
## What changed — and what did not
|
||||||
|
- **2026-10-25, `config.yaml` written 19:13 AWST:** all **eight** `auxiliary:` slots moved off `nous` to
|
||||||
|
**`no-think/freeapi/auto` on `custom:omni-101`**, each with the chain
|
||||||
|
`no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, `free_only: true`
|
||||||
|
(compression timeout 90 s, vision timeout 180 s). → [[systems/model-providers]]
|
||||||
|
- **The credential is still invalid.** The fault is **masked, not fixed** — no `hermes model`
|
||||||
|
re-authentication has been run, and `nous` is still the **first** entry in `fallback_providers`.
|
||||||
|
- Zero `invalid_grant` lines appear after the 2026-10-25 19:27:13 gateway start, so the masking is holding.
|
||||||
|
|
||||||
|
## Open
|
||||||
|
- Re-authenticate the Nous Portal credential, or remove `nous` from `fallback_providers`.
|
||||||
|
- Until then, a failover event will attempt a dead credential silently.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/model-providers]]
|
||||||
|
- [[systems/current-profile]]
|
||||||
|
- [[systems/hermes-agent]]
|
||||||
@@ -3,7 +3,7 @@ title: Conduit Call Requests Refused by the Agent
|
|||||||
type: incident
|
type: incident
|
||||||
status: open
|
status: open
|
||||||
created: 2026-10-25
|
created: 2026-10-25
|
||||||
updated: 2026-10-25
|
updated: 2026-10-26
|
||||||
verified_on: 2026-10-25
|
verified_on: 2026-10-25
|
||||||
review_after: 2026-11-08
|
review_after: 2026-11-08
|
||||||
confidence: high
|
confidence: high
|
||||||
@@ -86,3 +86,10 @@ into [[incidents/2026-10-22-conduit-memory-start-timeout]].
|
|||||||
- [[systems/conduit]]
|
- [[systems/conduit]]
|
||||||
- [[incidents/2026-10-22-conduit-memory-start-timeout]]
|
- [[incidents/2026-10-22-conduit-memory-start-timeout]]
|
||||||
- [[concepts/tool-calling]]
|
- [[concepts/tool-calling]]
|
||||||
|
|
||||||
|
## Status update — 2026-10-26 00:1x AWST
|
||||||
|
- **No further refusals in-window** (2026-10-24 12:56 → 2026-10-26 00:1x). The signature has not recurred.
|
||||||
|
- `conduit-calls.json` → `history` is still **empty (0 entries)**, so the call path has gone unused, not fixed. The file *does* carry `watches`, `outcomes`, `ends` and `told` state.
|
||||||
|
- The plugin **is** working for cards: `conduit_show` ×1 on 2026-10-25 14:37, delivered live.
|
||||||
|
- Plugin advanced to **`0.69.2`** and the dashboard gateway restarted at **18:42:33**, so upstream commits are live — see [[systems/conduit]].
|
||||||
|
- **Status stays open**: the request path is untested, not confirmed good.
|
||||||
|
|||||||
@@ -0,0 +1,61 @@
|
|||||||
|
---
|
||||||
|
title: Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop
|
||||||
|
type: incident
|
||||||
|
status: open
|
||||||
|
created: 2026-10-26
|
||||||
|
updated: 2026-10-26
|
||||||
|
verified_on: 2026-10-26
|
||||||
|
review_after: 2026-11-09
|
||||||
|
confidence: medium
|
||||||
|
tags: [cron, automation, hermes, incident, unresolved]
|
||||||
|
sources: [live-sqlite-cron-executions, live-sqlite-sessions, gateway-log, jobs-json]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop (2026-10-25)
|
||||||
|
|
||||||
|
**Windows:** 2026-10-25 08:31:14 → 10:26:37 AWST (runaway) and 17:19:36 → 19:51:59 AWST (five extra fires)
|
||||||
|
**Job:** `08eb86f02d37` — *Gmail Action Digest*, daily `30 8 * * *`, pinned `custom:omni-101`
|
||||||
|
|
||||||
|
## What happened
|
||||||
|
- The 10-25 **08:30 slot** claimed at 08:30:57, started **08:31:14** and did not finish until **10:26:37 —
|
||||||
|
a 1 h 55 m 23 s run**, against **91 s** (10-23) and **104 s** (10-24) in the two preceding days. It closed
|
||||||
|
`failed` on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed).`
|
||||||
|
- The same job then fired **five more times** on 10-25 — 17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53 —
|
||||||
|
for **six fires in the calendar day** against one configured daily slot.
|
||||||
|
- `19:18:41 → 19:28:04` closed `failed` on a signature this ledger had not seen before:
|
||||||
|
**`Cron admission missing: the owner never admitted this fire`** (`source: direct`, `pid 2473801`,
|
||||||
|
`scheduled_instant` empty). The 2026-10-20 orphan was `Unclean gateway exit` — a second, distinct variant.
|
||||||
|
- Every fire, including both failures, carries `delivery_outcome: delivered`, so **up to six Gmail Action
|
||||||
|
Digests reached Anthony on 2026-10-25**. The 19:30:31 fire ran 21 m 28 s.
|
||||||
|
- Restart context (`gateway.log`): `18:29:22` SIGTERM as a planned stop under systemd → `18:30:04` start;
|
||||||
|
`19:26:16` *Stopping gateway for restart* → `19:27:13` start. Three starts on 10-25 (00:32:20, 18:30:04,
|
||||||
|
19:27:13). The five extra fires bracket the evening pair. → [[systems/hermes-agent]]
|
||||||
|
|
||||||
|
## Root cause — hypothesised, not confirmed
|
||||||
|
The 2026-10-25 14:17 interactive investigation found that the 08:31 fire's worker had made **70 `terminal`
|
||||||
|
calls on unrelated Proxmox VM101 `/dev/sdb` backup work**, with run context growing **66 k → 148 k tokens**,
|
||||||
|
and that **context compression was unavailable all day** (`invalid_grant`). Working hypothesis: the scheduled
|
||||||
|
fire **adopted and continued an unrelated long-lived session** instead of starting its own. *Why* the
|
||||||
|
scheduler admitted an unrelated session is unexplained and is the first thing to check.
|
||||||
|
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||||
|
|
||||||
|
## Impact
|
||||||
|
- Six delivered digests in one day (duplicate content) — the user-visible symptom.
|
||||||
|
- **Neither failure raised an `cron_incidents` row**; the two open rows are older (10-01, 10-19). A failed
|
||||||
|
execution therefore does not reliably produce an incident.
|
||||||
|
- The job now reads `last_status: delivery_queued` with an **empty** `last_error` (2026-10-26 00:1x) — a state
|
||||||
|
that did not resolve to `ok`.
|
||||||
|
- 2026-10-25 closed **172 `completed` / 2 `failed`**, both failures belonging to this job.
|
||||||
|
|
||||||
|
## Open questions / next actions
|
||||||
|
- Why did a scheduled fire adopt an unrelated session? Check session-reuse across fires.
|
||||||
|
- Record `Cron admission missing: the owner never admitted this fire` alongside the 2026-10-20 orphan
|
||||||
|
signature and give both a recovery path on gateway restart.
|
||||||
|
- Give scheduled runs a **wall-clock budget** — a 113× runtime creep was invisible anywhere except the
|
||||||
|
execution row.
|
||||||
|
- Does `delivery_queued` clear on the 2026-10-26 08:30 slot? Check after that fire.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/scheduled-tasks]]
|
||||||
|
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||||
|
- [[systems/hermes-agent]]
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
# Anthony's Systems Wiki
|
# Anthony's Systems Wiki
|
||||||
> Operational knowledge for Anthony's agents, infrastructure,
|
> Operational knowledge for Anthony's agents, infrastructure,
|
||||||
> model routing and automation. Last updated: 2026-10-25
|
> model routing and automation. Last updated: 2026-10-26 (stocktake — 8 stale pages corrected)
|
||||||
>
|
>
|
||||||
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
|
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
|
||||||
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
|
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
|
||||||
@@ -32,13 +32,15 @@
|
|||||||
- [[systems/tool-search]] — Always-on lazy discovery for external tools
|
- [[systems/tool-search]] — Always-on lazy discovery for external tools
|
||||||
|
|
||||||
### Integrations
|
### Integrations
|
||||||
- [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.69.0` (HEAD `f000bf0` — 22 commits on since the 10-24 audit)**, upstream `kaishi00/hermes-conduit-notifier`; gateway on `:9119`, process up since **2026-10-23 18:54:43** (so those upstream commits are **not** live); one registered device (iPhone app `2.0.0`; phone-status contact last 10-23 19:10, health ingestion to 10-24 09:59; store 569.9 MB / 1,520,229 events / 3,028 `buckets`). **Still carries the uncommitted in-tree patch — now five releases behind HEAD** (`conduit-calls.json → history` is **empty**, so the memory fix has no positive call evidence; the “two calls closed cleanly 00:03:23 / 00:06:57” reading was **wrong** — those were cron turn-ends). ⚠ **Known failure: three 10-24 requests to be phoned were refused by the agent, which never called the tool** → [[incidents/2026-10-24-conduit-call-request-refused]].
|
- [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.69.2` (HEAD `942ddb8`, 2026-10-24 15:35 — *"Pulse: list candidates, decide in code, journal them (#342)"*; 24 commits on since the `0.64.0` baseline, **0 behind `origin/main`**)**, upstream `kaishi00/hermes-conduit-notifier`. **Gateway on `:9119` restarted 2026-10-25 18:42:33 — so the upstream commits are now LIVE** (the 10-24 "not live" note is superseded). One registered device (iPhone app `2.0.0`). **Still carries the uncommitted in-tree `dashboard/plugin_api.py` patch**; `conduit-calls.json → history` is **empty**, so the memory fix still has no positive call evidence, but `conduit_show` ran once on 10-25 14:37 (card delivered). ⚠ **Known failure: three 10-24 requests to be phoned were refused by the agent, which never called the tool; no recurrence since** → [[incidents/2026-10-24-conduit-call-request-refused]].
|
||||||
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped** (peer offline 29d; host answers neither `:8065` nor ICMP).
|
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26** (peer offline **36 d**, `tx 312 / rx 0`; host answers neither `:8065` nor ICMP). **268** connect timeouts on 10-25 vs 264 on 10-24; the reconnect **attempt counter reset to 54** at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost.
|
||||||
- [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip.
|
- [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip.
|
||||||
- [[systems/scheduled-tasks]] — **As of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
|
- [[systems/scheduled-tasks]] — **As of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
|
||||||
- [[systems/skills-index]] — Installed skills inventory (**306 skills / 70 categories** as of the **2026-10-23 03:07** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at that run: **2 stale / 94 fresh / 6 no-date of 102 pages** (`freshness-20261023.txt`). Note: `SKILL.md` files on disk still exceed the index (the generator de-dups one nested duplicate; see [[systems/scheduled-tasks]]). ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
|
- [[systems/skills-index]] — Installed skills inventory (**306 skills / 70 categories** as of the **2026-10-23 03:07** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at that run: **2 stale / 94 fresh / 6 no-date of 102 pages** (`freshness-20261023.txt`). Note: `SKILL.md` files on disk still exceed the index (the generator de-dups one nested duplicate; see [[systems/scheduled-tasks]]). ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
|
||||||
|
|
||||||
## Incidents
|
## Incidents
|
||||||
|
- [[incidents/2026-10-25-gmail-digest-runaway-and-refire]] — **The Gmail Action Digest ran 1 h 55 m and fired six times on 2026-10-25.** The 08:30 slot started 08:31:14 and closed **10:26:37 `failed`** on `RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed)` — against **91 s** (10-23) and **104 s** (10-24). The same job then fired five more times (17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53); the 19:18:41 fire died on a **new signature** — `Cron admission missing: the owner never admitted this fire` (distinct from the 10-20 orphan) — and all six fires, failures included, carry `delivery_outcome: delivered`, so up to **six digests reached Anthony**. Neither failure raised a `cron_incidents` row; the job now reads `last_status: delivery_queued` with an **empty** `last_error`. Hypothesis (unconfirmed): the 08:31 fire **adopted an unrelated long-lived session** — its worker made **70 `terminal` calls on VM101 `/dev/sdb` backup work** with context growing 66 k → 148 k tokens, while **context compression was unavailable all day** (`invalid_grant`). → [[systems/scheduled-tasks]], [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||||
|
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] — **`Invalid refresh token` — Nous Portal runtime credentials are dead, so context summarisation failed.** Gateway-log counts: **1** line 2026-10-22, **3** on 10-24, **20** on 10-25 (last 14:53:04, `agent.context_compressor: Failed to generate context summary`), **0** since the 19:27:13 restart. ⚠ **Masked, not fixed:** at **2026-10-25 19:13** all **eight** `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) moved off `nous` to **`no-think/freeapi/auto` on `custom:omni-101`** (fallback `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, `free_only: true`; compression timeout 90 s, vision 180 s). No `hermes model` re-authentication has been run and **`nous` is still the first entry in `fallback_providers`**, so failover tries a dead credential. → [[systems/model-providers]], [[systems/current-profile]]
|
||||||
- [[incidents/2026-10-24-conduit-call-request-refused]] — **The agent refused to phone Anthony, three times, on 2026-10-24 (12:53–12:56).** He asked to be called to test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`); all three replies asserted the agent has **no calling capability**, with **zero tool calls** in the session. `conduit_call_user` (toolset `conduit`) exists, is registered for every session, is reachable via `tool_search`, and the plugin ships the `calling-the-user` skill; his call settings (`enabled` / `when_asked` / `decides`) were permissive and the device was paired. **Root cause: capability asserted instead of discovered** — the first reply came from `kilo-auto/free`, but the switch to `smart-route` repeated it. Same audit also **corrects the 10-22 incident's closing evidence**: `conduit-calls.json → ends` records turn-ends that fire a watch, not calls, so “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne **cron** sessions — `history` (real calls) is **empty** and no call has been placed since the memory fix. → [[systems/conduit]], [[incidents/2026-10-22-conduit-memory-start-timeout]]
|
- [[incidents/2026-10-24-conduit-call-request-refused]] — **The agent refused to phone Anthony, three times, on 2026-10-24 (12:53–12:56).** He asked to be called to test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`); all three replies asserted the agent has **no calling capability**, with **zero tool calls** in the session. `conduit_call_user` (toolset `conduit`) exists, is registered for every session, is reachable via `tool_search`, and the plugin ships the `calling-the-user` skill; his call settings (`enabled` / `when_asked` / `decides`) were permissive and the device was paired. **Root cause: capability asserted instead of discovered** — the first reply came from `kilo-auto/free`, but the switch to `smart-route` repeated it. Same audit also **corrects the 10-22 incident's closing evidence**: `conduit-calls.json → ends` records turn-ends that fire a watch, not calls, so “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne **cron** sessions — `history` (real calls) is **empty** and no call has been placed since the memory fix. → [[systems/conduit]], [[incidents/2026-10-22-conduit-memory-start-timeout]]
|
||||||
- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]]
|
- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]]
|
||||||
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]]
|
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]]
|
||||||
|
|||||||
+21
-14
@@ -3,8 +3,8 @@ title: Known Hosts and Paths
|
|||||||
type: infrastructure
|
type: infrastructure
|
||||||
status: partial
|
status: partial
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-04
|
updated: 2026-10-26
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-26
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [infrastructure, hosts, containers, paths]
|
tags: [infrastructure, hosts, containers, paths]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -16,12 +16,15 @@ sources: []
|
|||||||
|
|
||||||
| Host | Role | Address | Verified |
|
| Host | Role | Address | Verified |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| localhost (this host) | Primary Hermes runtime, Xvfb, file storage | `:102` display | Yes |
|
| localhost (this host) | Primary Hermes runtime, Xvfb, file storage | `:99` display (`Xvfb :99 -screen 0 1600x1200x24`) | Yes (live probe 2026-10-26) |
|
||||||
| litellm | LiteLLM proxy | `http://litellm:4000/v1` | Yes (from config) |
|
| litellm | LiteLLM proxy | `http://litellm:4000/v1` | Yes (from config) |
|
||||||
| omni | OmniRoute proxy | `http://omni:20128/v1` | Yes (from config, 2026-08-13) |
|
| `custom:Omniroute:20128` | OmniRoute proxy (CT600 node `omniroute`) | `http://omniroute:20128/v1` → `100.88.81.19:20128` | Yes (live probe 2026-10-26: 6,480 models) |
|
||||||
|
| `custom:omni-101` | OmniRoute proxy (CT101 node `omniroute-1`) — **the live default provider** | `http://100.78.63.20:20128` | Yes (live probe 2026-10-26) |
|
||||||
| CT201 | Steel Browser, Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.166` (LAN) | Yes (live probe) |
|
| CT201 | Steel Browser, Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.166` (LAN) | Yes (live probe) |
|
||||||
| CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net` | Yes |
|
| CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net`, `100.118.5.51` (Tailscale) | Yes |
|
||||||
| CT450 (hermesagent) | Hermes Agent **v0.21.5+6170.ga510d6c** (updated 2026-10-03 from v0.19.0), hermes-webui, hermes-ui (Vite) | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes |
|
| PVE `pve` | Proxmox VE hypervisor hosting CT460 and the other `pve` guests | `100.96.100.82` (Tailscale), `pve.kangaroo-eel.ts.net` | Yes (live `pct list` 2026-10-26) |
|
||||||
|
| PVE `m93p` | Second, unclustered Proxmox VE 9.2.10 hypervisor | `100.104.51.98` (Tailscale) | Yes (live `pct list` 2026-10-26) |
|
||||||
|
| CT450 (hermesagent) | Hermes Agent **v0.22.3** (2026.9.24), local `5b147ab036d6` — same build as CT460, hermes-webui, hermes-ui (Vite) | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes |
|
||||||
|
|
||||||
## Project Paths
|
## Project Paths
|
||||||
|
|
||||||
@@ -30,15 +33,16 @@ sources: []
|
|||||||
| `/home/hermes` | Hermes home | `~/.hermes/` contains config, skills, cache |
|
| `/home/hermes` | Hermes home | `~/.hermes/` contains config, skills, cache |
|
||||||
| `/home/hermes/workspace` | Workspace | default terminal cwd |
|
| `/home/hermes/workspace` | Workspace | default terminal cwd |
|
||||||
| `/home/hermes/.local/bin/hermes` | Hermes binary | |
|
| `/home/hermes/.local/bin/hermes` | Hermes binary | |
|
||||||
| `/home/hermes/.hermes/config.yaml` | Config | version 33 |
|
| `/home/hermes/.hermes/config.yaml` | Config | `_config_version: 51` |
|
||||||
| `/home/hermes/.hermes/skills/` | Skills dir | 66 skills installed |
|
| `/home/hermes/.hermes/skills/` | Skills dir | 302 skills across 69 categories (`.archive/` excluded) |
|
||||||
| `/home/hermes/.hermes/cron/` | Cron jobs | 25 active jobs |
|
| `/home/hermes/.hermes/cron/` | Cron jobs | 28 jobs — 25 enabled / 3 disabled |
|
||||||
| `/home/hermes/.hermes/mnemosyne/` | Mnemosyne DB | backups/, data/, logs/, models/ |
|
| `/home/hermes/.hermes/mnemosyne/` | Mnemosyne DB | backups/, data/, logs/, models/ |
|
||||||
| `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | cua-driver | v0.9.0 |
|
| `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | cua-driver | v0.9.0 |
|
||||||
| `~/.camofox/` | Camofox profiles | managed persistence |
|
| `~/.camofox/` | Camofox profiles | managed persistence |
|
||||||
| `~/.camoufox/` | Camoufox profiles | multiple profiles present |
|
| `~/.camoufox/` | Camoufox profiles | multiple profiles present |
|
||||||
| `/home/hermes/.hermes/browser_screenshots/` | Browser captures | |
|
| `/home/hermes/.hermes/browser_screenshots/` | Browser captures | |
|
||||||
| `/home/hermes/wiki/` | Wiki | This knowledge base |
|
| `/home/hermes/workspace/agent-estate-wiki/` | Wiki — **authoring tree** (source of truth, pushed to Gitea) |
|
||||||
|
| `/home/hermes/wiki/agent-estate-wiki/` | Wiki — **Obsidian/Quartz vault mirror** (local-only git; synced per audit, never pushed) |
|
||||||
| `/home/hermes/obsidian-vault/` | Obsidian vault | personal notes (50 - Research etc.) |
|
| `/home/hermes/obsidian-vault/` | Obsidian vault | personal notes (50 - Research etc.) |
|
||||||
| `/home/hermes/hermes-webui/` | Hermes WebUI | web UI source |
|
| `/home/hermes/hermes-webui/` | Hermes WebUI | web UI source |
|
||||||
| `/home/hermes/hermes-ui/` | Hermes Desktop browser (Vite dev) | present on CT460 and CT450 port 5174 |
|
| `/home/hermes/hermes-ui/` | Hermes Desktop browser (Vite dev) | present on CT460 and CT450 port 5174 |
|
||||||
@@ -55,12 +59,15 @@ sources: []
|
|||||||
- `/home/hermes/workspace/readlater/docker-compose.yml`
|
- `/home/hermes/workspace/readlater/docker-compose.yml`
|
||||||
|
|
||||||
## Homelab References
|
## Homelab References
|
||||||
- **PVE host:** `pve.kangaroo-eel.ts.net` (100.96.100.82) per session notes — Unverified on this host
|
- **PVE host `pve`:** `pve.kangaroo-eel.ts.net` (100.96.100.82) — **verified** via live `pct list` 2026-10-26 (17 LXCs)
|
||||||
- **CT600:** OmniRoute server (retired → CT601 `omni`, 2026-08-13) — Verified via `pct list`
|
- **PVE host `m93p`:** 100.104.51.98, Proxmox VE 9.2.10 — **verified** via live `pct list` 2026-10-26 (19 LXCs). Guest IDs collide between the two hosts
|
||||||
|
- **CT600 (`omniroute`):** OmniRoute server, `100.88.81.19` — **running and live**; `omniroute` is the hostname that resolves to it
|
||||||
|
- **CT601 (`omni`):** the old CT601 OmniRoute guest **no longer exists** on either hypervisor (`pct config 601` → no such config). The `omni` Tailscale peer (100.93.204.4) is its orphaned node, offline 69d
|
||||||
|
- **CT101 (`omniroute`, m93p):** second OmniRoute node, `100.78.63.20` — running
|
||||||
|
|
||||||
## Unverified
|
## Unverified
|
||||||
- CT600, CT333, and other remote Proxmox containers/VMs referenced in session notes but not confirmed from this host
|
- Individual LXCs are verified only when probed directly; the two `pct list` snapshots above are the current authority for what is running
|
||||||
- PVE host identity not confirmed from this host without explicit confirmation
|
- Guests not present in those `pct list` outputs are **not** running on either hypervisor — do not assume existence from older session notes
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[docker-services]]
|
- [[docker-services]]
|
||||||
|
|||||||
+27
-23
@@ -3,8 +3,8 @@ title: Tailscale
|
|||||||
type: infrastructure
|
type: infrastructure
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-26
|
created: 2026-07-26
|
||||||
updated: 2026-10-25
|
updated: 2026-10-26
|
||||||
verified_on: 2026-10-25
|
verified_on: 2026-10-26
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [tailscale, vpn, networking, infrastructure]
|
tags: [tailscale, vpn, networking, infrastructure]
|
||||||
---
|
---
|
||||||
@@ -20,7 +20,7 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access
|
|||||||
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
|
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
|
||||||
|
|
||||||
## Tailscale Node Inventory (verified 2026-10-25)
|
## Tailscale Node Inventory (verified 2026-10-25)
|
||||||
Full node list from `tailscale status` — **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
|
Full node list from `tailscale status` — **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
|
||||||
Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong).
|
Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong).
|
||||||
Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`.
|
Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`.
|
||||||
Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**.
|
Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**.
|
||||||
@@ -29,43 +29,45 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
|||||||
|
|
||||||
| Node | IP | State |
|
| Node | IP | State |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| hermes | 100.118.5.51 | **self** |
|
| hermes | 100.118.5.51 | **self** — CT460 (pve) |
|
||||||
| agentzero | 100.106.54.94 | online — CT475 (pve) |
|
| agentzero | 100.106.54.94 | online — CT475 (pve) |
|
||||||
| anthonys-imac | 100.112.123.124 | idle/offline 31d |
|
| anthonys-imac | 100.112.123.124 | idle/offline 32d |
|
||||||
| anthonys-ipad | 100.111.198.120 | online |
|
| anthonys-ipad | 100.111.198.120 | online |
|
||||||
| anthonys-iphone | 100.100.116.67 | online |
|
| anthonys-iphone | 100.100.116.67 | online |
|
||||||
| anthonys-mac-mini | 100.74.17.15 | online |
|
| anthonys-mac-mini | 100.74.17.15 | online |
|
||||||
| cachyos | 100.124.208.15 | idle/offline 51d |
|
| cachyos | 100.124.208.15 | idle/offline 52d |
|
||||||
| clawtest | 100.122.189.81 | online — CT333 (pve) |
|
| clawtest | 100.122.189.81 | online — CT333 (pve) |
|
||||||
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
|
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
|
||||||
| cloudreve | 100.111.187.7 | idle/offline 92d — CT209 stopped |
|
| cloudreve | 100.111.187.7 | idle/offline 93d — CT209 stopped |
|
||||||
| conduit | 100.81.79.87 | idle/offline 48d |
|
| conduit | 100.81.79.87 | idle/offline 49d |
|
||||||
| docker | 100.96.150.69 | online — CT200 (pve) |
|
| docker | 100.96.150.69 | online — CT200 (pve) |
|
||||||
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
|
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
|
||||||
| elementsynapse | 100.75.250.37 | idle/offline 95d — CT555 stopped |
|
| elementsynapse | 100.75.250.37 | idle/offline 96d — CT555 stopped |
|
||||||
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
|
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
|
||||||
| freshrss | 100.83.60.93 | online — CT1260 (m93p) |
|
| freshrss | 100.83.60.93 | online — CT1260 (m93p) |
|
||||||
| freshrss-new | 100.110.180.115 | idle/offline 68d |
|
| freshrss-new | 100.110.180.115 | idle/offline 69d |
|
||||||
| gitea | 100.82.100.128 | online — CT221 (m93p) |
|
| gitea | 100.82.100.128 | online — CT221 (m93p) |
|
||||||
| guest-vps | 100.117.14.84 | online |
|
| guest-vps | 100.117.14.84 | online |
|
||||||
| hermesagent | 100.98.205.112 | online — CT450 (pve) |
|
| hermesagent | 100.98.205.112 | online — CT450 (pve) |
|
||||||
| homeassistant | 100.68.139.5 | online |
|
| homeassistant | 100.68.139.5 | online |
|
||||||
| litellm | 100.79.29.64 | online — CT500 (m93p) |
|
| litellm | 100.79.29.64 | online — CT500 (m93p) |
|
||||||
| m93p | 100.104.51.98 | idle/offline 13m ago |
|
| m93p | 100.104.51.98 | online |
|
||||||
| mailflow | 100.115.60.92 | online — CT1262 (m93p) |
|
| mailflow | 100.115.60.92 | online — CT1262 (m93p) |
|
||||||
| mattermost | 100.98.116.14 | idle/offline 35d — CT255 stopped |
|
| mattermost | 100.98.116.14 | idle/offline 36d — CT255 stopped |
|
||||||
| n8n | 100.84.124.30 | online — CT215 (pve) |
|
| n8n | 100.84.124.30 | online |
|
||||||
| nextcloud | 100.90.102.19 | online — CT270 (pve) |
|
| nextcloud | 100.90.102.19 | online — CT270 (pve) |
|
||||||
| ntfy | 100.67.153.116 | online — CT1241 (m93p) |
|
| ntfy | 100.67.153.116 | online — CT1241 (m93p) |
|
||||||
|
| octoagent | 100.117.10.68 | online |
|
||||||
| octop | 100.92.163.122 | online |
|
| octop | 100.92.163.122 | online |
|
||||||
| ollama | 100.108.44.119 | online — CT205 (pve) |
|
| ollama | 100.108.44.119 | online — CT205 (pve) |
|
||||||
| omarchy | 100.73.227.103 | online |
|
| omarchy | 100.73.227.103 | online |
|
||||||
| omarchybook | 100.107.9.100 | idle/offline 3d |
|
| omarchybook | 100.107.9.100 | idle/offline 4d |
|
||||||
| omarchymac | 100.84.180.69 | idle/offline 7d |
|
| omarchymac | 100.84.180.69 | idle/offline 8d |
|
||||||
| omni | 100.93.204.4 | idle/offline 68d — CT601 stopped |
|
| omarchymac-1 | 100.126.173.115 | online |
|
||||||
|
| omni | 100.93.204.4 | idle/offline 69d — CT601 removed from both hypervisors |
|
||||||
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
|
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
|
||||||
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
|
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
|
||||||
| openmausbot | 100.91.214.91 | idle/offline 3d |
|
| openmausbot | 100.91.214.91 | idle/offline 4d |
|
||||||
| openmausbot-vm | 100.77.107.30 | online |
|
| openmausbot-vm | 100.77.107.30 | online |
|
||||||
| openweb | 100.69.230.13 | online — CT245 (m93p) |
|
| openweb | 100.69.230.13 | online — CT245 (m93p) |
|
||||||
| paseo | 100.72.188.93 | online — CT1261 (m93p) |
|
| paseo | 100.72.188.93 | online — CT1261 (m93p) |
|
||||||
@@ -78,7 +80,7 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
|||||||
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
|
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
|
||||||
| springroll | 100.100.153.125 | online |
|
| springroll | 100.100.153.125 | online |
|
||||||
| springroll-browser | 100.109.221.91 | online |
|
| springroll-browser | 100.109.221.91 | online |
|
||||||
| ubuntu-server-xfce | 100.101.26.104 | idle/offline 64d |
|
| ubuntu-server-xfce | 100.101.26.104 | idle/offline 66d |
|
||||||
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
|
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
|
||||||
| webserver | 100.82.40.23 | online — CT207 (pve) |
|
| webserver | 100.82.40.23 | online — CT207 (pve) |
|
||||||
|
|
||||||
@@ -102,7 +104,7 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
|||||||
| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) |
|
| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) |
|
||||||
| `/evening-grid.html` | `http://127.0.0.1:9099/evening-grid.html` | Evening grid |
|
| `/evening-grid.html` | `http://127.0.0.1:9099/evening-grid.html` | Evening grid |
|
||||||
|
|
||||||
Verified against `tailscale serve status` on 2026-10-25 — the CT460 route table above matched live exactly (12 paths, no drift).
|
Re-verified against `tailscale serve status` on **2026-10-26** — the CT460 route table above matched live exactly (**12 paths on :443**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; no drift).
|
||||||
|
|
||||||
## CT460 Funnel and 9443 Routes
|
## CT460 Funnel and 9443 Routes
|
||||||
| Port | Target | Notes |
|
| Port | Target | Notes |
|
||||||
@@ -110,16 +112,18 @@ Verified against `tailscale serve status` on 2026-10-25 — the CT460 route tabl
|
|||||||
| `:8443/webhook` | `http://127.0.0.1:8085/webhook` | Public webhook Funnel (`# Funnel on:` confirmed) |
|
| `:8443/webhook` | `http://127.0.0.1:8085/webhook` | Public webhook Funnel (`# Funnel on:` confirmed) |
|
||||||
| `:9443` (tailnet only) `/` | `http://127.0.0.1:5173` | Vite dev server on port 5173 exposed via a **tailnet-only** Serve on port 9443 — was missing from this page, added 2026-10-11 |
|
| `:9443` (tailnet only) `/` | `http://127.0.0.1:5173` | Vite dev server on port 5173 exposed via a **tailnet-only** Serve on port 9443 — was missing from this page, added 2026-10-11 |
|
||||||
|
|
||||||
|
Re-verified 2026-10-26 (stocktake): the node table was rebuilt again from `tailscale status --json` — **`octoagent` (100.117.10.68) and `omarchymac-1` (100.126.173.115) were both missing entirely** and have been added; `m93p` (100.104.51.98) now reads **online** (it was "idle/offline 13m ago" — a stale Tailscale peer record, and the host was plainly reachable throughout); every offline age was refreshed (`mattermost` 35d → **36d**, `omni` 68d → **69d**, `elementsynapse` 95d → **96d**, `ubuntu-server-xfce` 64d → **66d**, `omarchymac` 7d → **8d**, `omarchybook` 3d → **4d**, `openmausbot` 3d → **4d**). The `omni` row now reads **"CT601 removed from both hypervisors"** rather than "CT601 stopped" — live `pct list` on both `pve` and `m93p` shows **no CT601 at all**, so the old guest is gone, not merely halted. Container identity re-confirmed from **inside** each LXC via `pct exec <vmid> -- tailscale ip -4` on both hypervisors; every peer whose LXC has no Tailscale IP was checked against `pct list` (`CT105`/`CT1264` run but do not connect — `conduit` and `openmausbot` offline; `CT209`/`CT255`/`CT555` are `stopped`). Serve tables re-probed per host with `pct exec 201 -- tailscale serve status`.
|
||||||
|
|
||||||
## CT201 Serve Routes
|
## CT201 Serve Routes
|
||||||
Verified live on 2026-10-25 via `pct exec 201 -- tailscale serve status` on `pve` (100.96.100.82). The previous version of this table was wrong in two ways and has been corrected.
|
Verified live on 2026-10-26 via `pct exec 201 -- tailscale serve status` on `pve` (100.96.100.82). The previous version of this table was wrong in two ways and has been corrected.
|
||||||
|
|
||||||
| Host / Port | Path | Target | Notes |
|
| Host / Port | Path | Target | Notes |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Homepage asset proxy front door — **currently 502** (upstream 8082 has no listener) |
|
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Homepage asset proxy front door — **502** (re-probed 2026-10-26; upstream 8082 has no listener) |
|
||||||
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` container — **200** |
|
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` container — **200** |
|
||||||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Same asset proxy — **currently 502** |
|
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Same asset proxy — **502** (re-probed 2026-10-26) |
|
||||||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` — **200** |
|
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` — **200** |
|
||||||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/steel` | `http://127.0.0.1:5173` | Steel Browser UI — **200** on :443. ⚠ **`/steel` is NOT served on :8443** — it 502s there |
|
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/steel` | `http://127.0.0.1:5173` | Steel Browser UI — **200** on :443 (re-probed 2026-10-26). ⚠ **`/steel` is NOT served on :8443** — it 502s there |
|
||||||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/steel-api` | `http://127.0.0.1:3000` | Steel Browser API (path prefix not stripped — use direct IP for API calls) |
|
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/steel-api` | `http://127.0.0.1:3000` | Steel Browser API (path prefix not stripped — use direct IP for API calls) |
|
||||||
|
|
||||||
Two corrections to the pre-2026-10-25 table:
|
Two corrections to the pre-2026-10-25 table:
|
||||||
|
|||||||
@@ -1264,3 +1264,121 @@ No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe st
|
|||||||
still **2** (`0e753e_e7f8194a7d06`, `c0a9fa_764558b5ae83`) — identical to the page's 2026-10-25
|
still **2** (`0e753e_e7f8194a7d06`, `c0a9fa_764558b5ae83`) — identical to the page's 2026-10-25
|
||||||
entry. No drift, nothing to change.
|
entry. No drift, nothing to change.
|
||||||
- **No secrets written.** Job IDs, incident IDs, IPs, counts, paths and timestamps only.
|
- **No secrets written.** Job IDs, incident IDs, IPs, counts, paths and timestamps only.
|
||||||
|
|
||||||
|
## [2026-10-26] audit | Nightly wiki audit — window 2026-10-25 00:00 → 2026-10-26 00:5x AWST: a runaway digest and a dead Nous credential
|
||||||
|
|
||||||
|
- **Fleet was quiet but one job was not.** Live `executions.db`: 2026-10-25 settled **172 `completed` / 2 `failed`**;
|
||||||
|
**both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times** in the day (08:31:14 →
|
||||||
|
10:26:37 runaway at **1 h 55 m** vs 91 s / 104 s the two prior days; then 17:19:36, 18:40:06, 19:18:41, 19:30:31,
|
||||||
|
19:38:53). New signature on the 19:18:41 fire: **`Cron admission missing: the owner never admitted this fire`**.
|
||||||
|
All six fires — failures included — carry `delivery_outcome: delivered`. **Neither failure created a `cron_incidents`
|
||||||
|
row**; the page's open-incident count is unchanged at 2. → new `incidents/2026-10-25-gmail-digest-runaway-and-refire.md`
|
||||||
|
- **Root cause of the missing context compression found: the Nous Portal credential is dead.**
|
||||||
|
`Nous Portal runtime credentials unavailable: Invalid refresh token` — **1** line 2026-10-22, **3** on 10-24,
|
||||||
|
**20** on 10-25 (last 14:53:04, `agent.context_compressor: Failed to generate context summary`), **0** since the
|
||||||
|
19:27:13 restart. → new `incidents/2026-10-22-nous-portal-refresh-token-invalid.md`
|
||||||
|
- **⚠ The global model default moved BACK to OmniRoute — and the last three audits recorded the stale value.**
|
||||||
|
A 2026-10-24 19:06:10 switch note reads *"model was just switched from meituan/longcat-2.5-preview:free to
|
||||||
|
`smart-route` via omni-101"*; live config confirms `custom:omni-101` / `smart-route` (`api_mode:
|
||||||
|
chat_completions`), `config.yaml` last written 2026-10-25 19:13. A **six-provider top-level `fallback_providers`
|
||||||
|
chain is live again** (headed by the dead `nous` credential), so the 2026-10-21 "no fallback chain" record is
|
||||||
|
stale too. Consequence: the 8 unpinned jobs are back on OmniRoute → **the fleet is a single upstream path again**.
|
||||||
|
Corrected on `systems/current-profile.md`, `systems/model-providers.md`, `concepts/model-routing.md`,
|
||||||
|
`comparisons/model-routers.md`, `systems/omniroute.md`.
|
||||||
|
- **Auxiliary routing captured.** At 2026-10-25 19:13 **all eight** `auxiliary:` slots (approval, compression,
|
||||||
|
curator, mcp, profile_describer, skills_hub, title_generation, vision) moved off `nous` to
|
||||||
|
**`no-think/freeapi/auto` on `custom:omni-101`** with `free_only: true` and a `freeapi/deepseek-v4-flash` →
|
||||||
|
`kg/kilo-auto/free` chain. ⚠ Masking, not fixing — `nous` remains first in `fallback_providers`. New section on
|
||||||
|
`systems/model-providers.md`.
|
||||||
|
- **`systems/scheduled-tasks.md` — new 2026-10-26 entry added**; the 2026-10-25 entry marked superseded. Fleet
|
||||||
|
unchanged (28 jobs / 25 enabled / 3 disabled, all 28 IDs present); non-clean `last_status` **4 → 5**
|
||||||
|
(`08eb86f02d37` joined as `delivery_queued` with an empty `last_error`); pins unchanged.
|
||||||
|
- **`systems/conduit.md` re-probed live.** Version **`0.69.2`** in `plugin.yaml` *and* `dashboard/manifest.json`
|
||||||
|
(both), HEAD **`942ddb8`** (2026-10-24 15:35:42, *"Pulse: list candidates, decide in code, journal them (#342)"*),
|
||||||
|
**0 behind `origin/main`**, 24 commits on since the `0.64.0` baseline, working tree still carrying the
|
||||||
|
uncommitted `dashboard/plugin_api.py` patch. **The dashboard gateway restarted 2026-10-25 18:42:33, so the
|
||||||
|
upstream commits are now live** — the previous "not live" note is superseded. `conduit-calls.json → history`
|
||||||
|
is still **empty**; `conduit_show` ran once on 10-25 14:37. Status on
|
||||||
|
`incidents/2026-10-24-conduit-call-request-refused.md` **stays open** (untested, not fixed).
|
||||||
|
- **`systems/messaging-integrations.md`** — CT255 still `stopped` (Mattermost offline **36 d**, `tx 312 / rx 0`);
|
||||||
|
268 timeouts on 10-25 vs 264 on 10-24; reconnect counter reset to 54 by the 19:27 restart (per-process, not
|
||||||
|
outage duration).
|
||||||
|
- **`systems/hermes-agent.md`** — gateway restart record added (three starts on 2026-10-25: 00:32:20, 18:30:04,
|
||||||
|
19:27:13; pid 2476414 up since 19:26:33) plus the credential fault and its restart-resets-counters caveat.
|
||||||
|
- **`index.md`** — header date, the two new incident links, and rewritten `conduit`, `messaging-integrations` and
|
||||||
|
`scheduled-tasks` summaries.
|
||||||
|
- **No secrets written.** Job IDs, incident IDs, base URLs, counts, paths and timestamps only.
|
||||||
|
- **Not changed:** `current-state.md` (its dated sections are historical records), and the dead `custom:omni`
|
||||||
|
provider entry (still named by 2 jobs — Anthony's call, not the wiki's).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2026-10-26 — Stocktake: 8 stale pages corrected against live state (0 stale remaining)
|
||||||
|
|
||||||
|
`wiki-maintain.sh` ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted, `hermes-wiki-serve.service`
|
||||||
|
restarted). The first freshness report listed **8 stale pages**; each was then **diffed against live state** and
|
||||||
|
corrected where reality could be verified. A second script run then rewrote the same-day report with
|
||||||
|
**0 stale / 103 fresh / 4 no-date**, so the committed report matches the tree it ships with.
|
||||||
|
|
||||||
|
- **`infrastructure/hosts.md`** — **CT600 is live, not retired.** Live `pct list` on both hypervisors shows
|
||||||
|
**CT600 running** on `pve` (100.88.81.19) and **CT101 running** on `m93p` (100.78.63.20); **CT601 does not
|
||||||
|
exist on either host** (`pct config 601` → no such config), so the old "retired → CT601" note was wrong on both
|
||||||
|
counts. **CT333** (`clawtest`) moved from "unverified" to verified running. Both hypervisors now carry a
|
||||||
|
verified row (`pve` 100.96.100.82 / 17 LXCs; `m93p` 100.104.51.98 / 19 LXCs). Config version **33 → 51**;
|
||||||
|
skills **66 → 302**; cron **"25 active" → 28 (25 enabled / 3 disabled)**; Xvfb display **:102 → :99**
|
||||||
|
(`Xvfb :99 -screen 0 1600x1200x24`). The `omni` provider entry was replaced by the two real ones —
|
||||||
|
`custom:Omniroute:20128` → `http://omniroute:20128/v1` (6,480 models) and **`custom:omni-101`** →
|
||||||
|
`http://100.78.63.20:20128` (**the live default provider**). The authoring tree vs the vault mirror are now
|
||||||
|
distinguished by path.
|
||||||
|
- **`agents/hermes-production.md`** — **Tailscale IP corrected: `100.96.100.82` → `100.118.5.51`.** The old value
|
||||||
|
is the **PVE hypervisor**, not CT460. Version **v0.21.5 → v0.22.3** (`5b147ab036d6`); stable **v0.22.5** is
|
||||||
|
available (`hermes update --check`). The **dead 8082** port claim was removed (no listener; live ports are
|
||||||
|
9119 gateway + 8085 webhook). Default provider is **`custom:omni-101`**, and the **fallback chain was rewritten
|
||||||
|
from live config** — its **first entry is the dead `nous` credential**, so a fallover tries a broken provider
|
||||||
|
first. All **eight `auxiliary:` slots** are `no-think/freeapi/auto` on `custom:omni-101` (moved 2026-10-25
|
||||||
|
19:13). Cron **30 (27/3) → 28 (25/3)** with the 5 non-clean `last_status` named. Mattermost still unreachable —
|
||||||
|
CT255 `stopped`, peer offline **36 d**.
|
||||||
|
- **`agents/hermes-chaos.md`** — version **v0.22.3**, and **both CT450 and CT460 now run the same build**
|
||||||
|
(`5b147ab036d6`); the "**3 commits behind**" claim is superseded (**0 behind `origin/main`**). Default model
|
||||||
|
is **`smart-route`** on `custom` → `http://100.78.63.20:20128`, not the old `deepseek-flash-mix` on
|
||||||
|
`custom:omniroute`. Cron **"11 paused" → 25 jobs (13 enabled / 12 disabled)**, with **9 names shared with
|
||||||
|
CT460** (8 disabled here, `Important-mail monitor` the reverse).
|
||||||
|
- **`infrastructure/tailscale.md`** — node table rebuilt from `tailscale status --json` (keyed on `DNSName`,
|
||||||
|
IPv4-filtered) and container identity re-confirmed from **inside** each LXC via
|
||||||
|
`pct exec <vmid> -- tailscale ip -4` on both hypervisors. **`octoagent` (100.117.10.68) and
|
||||||
|
`omarchymac-1` (100.126.173.115) were missing entirely** and are now listed — **52 → 54 non-funnel nodes**
|
||||||
|
(53 live IPv4 peers + `hermes`). `m93p` now reads **online** (it was a stale peer record). **Every offline age
|
||||||
|
was refreshed** (`mattermost` 35d → **36d**, `omni` 68d → **69d**, `elementsynapse` 95d → **96d**,
|
||||||
|
`ubuntu-server-xfce` 64d → **66d**, `omarchymac` 7d → **8d**, `omarchybook` 3d → **4d**,
|
||||||
|
`openmausbot` 3d → **4d**). The `omni` row now reads "**CT601 removed from both hypervisors**" rather than
|
||||||
|
"CT601 stopped". Serve tables re-probed per host: **CT460 matched live exactly** (12 paths on :443 plus the
|
||||||
|
:8443 Funnel and :9443 route); **CT201 still 502s on `/`** (asset proxy → dead 8082) while `/mgr` and `/steel`
|
||||||
|
return **200** — the 502 claim was re-confirmed, not merely carried over.
|
||||||
|
- **`systems/memory-backend.md`** — re-verified against live `config.yaml`: added `reflect.disabled_for_cron`,
|
||||||
|
`sleep_threshold: 50`, `nudge_interval: 10`, and live totals (**2,350 memories — 842 working / 1,090 episodic /
|
||||||
|
418 `memoria_facts`**). Embedding coverage re-measured: **1,899 / 2,350 (80.8 %)** overall, **working 815 / 842
|
||||||
|
(96.8 %)** and **episodic 1,084 / 1,090 (99.4 %)** — both tiers **above the 95 % reindex threshold**, so no
|
||||||
|
reindex was triggered.
|
||||||
|
- **`runbooks/provider-health.md`** — the step-3 OmniRoute snippet was **broken and raised `IndexError`**: the
|
||||||
|
provider was renamed `omniroute` → **`Omniroute:20128`** and the key moved from an inline `api_key` to a
|
||||||
|
**`key_env`** indirection. The snippet now resolves the key from `key_env`. A **new step 3b** documents the
|
||||||
|
**`omni-101` quirk** — its `/v1/models` returns `Invalid API key` even with the correct key while
|
||||||
|
`/v1/chat/completions` **works** (a `smart-route` call returned `minimaxai/minimax-m3.1-flash`), so a models
|
||||||
|
check on that node is a **false failure**. Also corrected the typo "config slocated" → "located".
|
||||||
|
Re-tested 2026-10-26: LiteLLM 401 unauthed (expected), CT600 OmniRoute 6,480 models, WebUI `/health` 200,
|
||||||
|
opencode/nous/ollama fallbacks all 200.
|
||||||
|
- **`runbooks/update-hermes-safely.md`** — added the **managed-runtime dependency re-assert step**
|
||||||
|
(`hermes-runtime-deps-guard.sh`) after an update, since the updater can leave a runtime without
|
||||||
|
`fastembed`/`dotenv`/`mnemosyne`. Clarified `--branch main` vs the **stable** channel (currently **v0.22.5**,
|
||||||
|
local **v0.22.3**). Version and `last_tested` brought to 2026-10-26.
|
||||||
|
- **`incidents/2026-10-03-ct450-hermes-update.md`** — added a **Current state** section: both instances now run
|
||||||
|
**v0.22.3** (`5b147ab036d6`), CT450 is **0 behind `origin/main`**, and the CT450 Mattermost timeout is still an
|
||||||
|
infrastructure condition (CT255 `stopped`), not an update regression.
|
||||||
|
- **`incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md`** — added a **Current state** section: the
|
||||||
|
follow-up is **satisfied** — `fastembed` 0.8.1 imports in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`,
|
||||||
|
`d41b16c8b6674454a4a819bd4acced1f`) and 0.8.0 in the legacy venv, `FASTEMBED_CACHE_PATH` pinned. Current
|
||||||
|
coverage figures recorded; both tiers above the 95 % threshold, so monitor job `b63ed5817212`
|
||||||
|
(`last_status: ok`) has nothing to fire on.
|
||||||
|
- **All 9 corrected pages verified at `http://127.0.0.1:9120/<path>` — every one returns 200**, and the served
|
||||||
|
`infrastructure/tailscale` contains `octoagent`, `omarchymac-1` and `54 non-funnel`.
|
||||||
|
- **No secrets written.** Job IDs, incident IDs, base URLs, key *env var names*, counts, paths and timestamps only.
|
||||||
|
|||||||
@@ -0,0 +1,120 @@
|
|||||||
|
# Wiki freshness audit — 2026-10-26
|
||||||
|
Threshold: >21 days since `updated`
|
||||||
|
|
||||||
|
## STALE (0)
|
||||||
|
|
||||||
|
|
||||||
|
## FRESH (103)
|
||||||
|
|
||||||
|
- [ 6d] CONTRIBUTING.md
|
||||||
|
- [ 0d] agents/hermes-chaos.md
|
||||||
|
- [ 0d] agents/hermes-production.md
|
||||||
|
- [ 6d] agents/index.md
|
||||||
|
- [ 15d] agents/nanobot.md
|
||||||
|
- [ 6d] agents/open-claw.md
|
||||||
|
- [ 15d] agents/qwenpaw.md
|
||||||
|
- [ 6d] agents/zeroclaw.md
|
||||||
|
- [ 6d] comparisons/camoufox-vs-chrome.md
|
||||||
|
- [ 6d] comparisons/hermes-vs-nanobot.md
|
||||||
|
- [ 6d] comparisons/memory-backends.md
|
||||||
|
- [ 6d] comparisons/model-routers.md
|
||||||
|
- [ 13d] comparisons/steel-vs-camoufox.md
|
||||||
|
- [ 6d] concepts/agent-memory.md
|
||||||
|
- [ 13d] concepts/browser-automation.md
|
||||||
|
- [ 6d] concepts/model-routing.md
|
||||||
|
- [ 6d] concepts/persistent-shells.md
|
||||||
|
- [ 6d] concepts/tool-calling.md
|
||||||
|
- [ 1d] current-state.md
|
||||||
|
- [ 6d] decisions/index.md
|
||||||
|
- [ 6d] decisions/keep-openrouter-as-fallback.md
|
||||||
|
- [ 6d] decisions/persistent-shell-enabled.md
|
||||||
|
- [ 6d] decisions/rollback-steel-to-camofox.md
|
||||||
|
- [ 6d] experiments/index.md
|
||||||
|
- [ 2d] incidents/2026-07-29-mnemosyne-relay.md
|
||||||
|
- [ 6d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||||
|
- [ 2d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||||
|
- [ 4d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||||
|
- [ 3d] incidents/2026-08-15-skills-index-drift.md
|
||||||
|
- [ 14d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||||
|
- [ 1d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||||
|
- [ 6d] incidents/2026-08-29-gmail-read-state.md
|
||||||
|
- [ 5d] incidents/2026-09-02-sftpgo-outage.md
|
||||||
|
- [ 15d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||||
|
- [ 20d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||||
|
- [ 0d] incidents/2026-10-03-ct450-hermes-update.md
|
||||||
|
- [ 0d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||||
|
- [ 20d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||||
|
- [ 2d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||||
|
- [ 17d] incidents/2026-10-09-memory-recall-relevance.md
|
||||||
|
- [ 10d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||||
|
- [ 4d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||||
|
- [ 7d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||||
|
- [ 5d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||||
|
- [ 4d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||||
|
- [ 1d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||||
|
- [ 0d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||||
|
- [ 0d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||||
|
- [ 0d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||||
|
- [ 4d] infrastructure/backups.md
|
||||||
|
- [ 6d] infrastructure/credentials-map.md
|
||||||
|
- [ 15d] infrastructure/docker-services.md
|
||||||
|
- [ 6d] infrastructure/domains-and-tunnels.md
|
||||||
|
- [ 0d] infrastructure/hosts.md
|
||||||
|
- [ 12d] infrastructure/network-map.md
|
||||||
|
- [ 12d] infrastructure/proxmox.md
|
||||||
|
- [ 0d] infrastructure/tailscale.md
|
||||||
|
- [ 14d] queries/retained-research-answers.md
|
||||||
|
- [ 1d] reports/communities-wa-social-media-audit-2026.md
|
||||||
|
- [ 5d] reports/uber-eats/2026-07.md
|
||||||
|
- [ 5d] reports/uber-eats/2026-08.md
|
||||||
|
- [ 5d] reports/uber-eats/2026-09.md
|
||||||
|
- [ 5d] reports/uber-eats/index.md
|
||||||
|
- [ 6d] runbooks/backup-wiki.md
|
||||||
|
- [ 6d] runbooks/diagnose-docker.md
|
||||||
|
- [ 6d] runbooks/gateway-resource-resilience.md
|
||||||
|
- [ 13d] runbooks/job-radar.md
|
||||||
|
- [ 6d] runbooks/multiplexer-setup.md
|
||||||
|
- [ 0d] runbooks/provider-health.md
|
||||||
|
- [ 6d] runbooks/recover-docker-service.md
|
||||||
|
- [ 6d] runbooks/restart-browser.md
|
||||||
|
- [ 6d] runbooks/restart-hermes.md
|
||||||
|
- [ 6d] runbooks/restore-openrouter.md
|
||||||
|
- [ 6d] runbooks/test-openrouter.md
|
||||||
|
- [ 6d] runbooks/troubleshoot-tool-loop.md
|
||||||
|
- [ 0d] runbooks/update-hermes-safely.md
|
||||||
|
- [ 13d] systems/browser-backend.md
|
||||||
|
- [ 6d] systems/cloudflare-tunnel.md
|
||||||
|
- [ 0d] systems/conduit.md
|
||||||
|
- [ 0d] systems/current-profile.md
|
||||||
|
- [ 6d] systems/guanaco.md
|
||||||
|
- [ 6d] systems/headless-server.md
|
||||||
|
- [ 0d] systems/hermes-agent.md
|
||||||
|
- [ 6d] systems/hermes-ui.md
|
||||||
|
- [ 14d] systems/home-assistant.md
|
||||||
|
- [ 1d] systems/homepage-dashboard.md
|
||||||
|
- [ 15d] systems/ignis.md
|
||||||
|
- [ 0d] systems/memory-backend.md
|
||||||
|
- [ 0d] systems/messaging-integrations.md
|
||||||
|
- [ 2d] systems/mnemosyne.md
|
||||||
|
- [ 0d] systems/model-providers.md
|
||||||
|
- [ 14d] systems/nanobot.md
|
||||||
|
- [ 6d] systems/nextcloud.md
|
||||||
|
- [ 6d] systems/obsidian.md
|
||||||
|
- [ 4d] systems/omniroute.md
|
||||||
|
- [ 6d] systems/opencode-go.md
|
||||||
|
- [ 6d] systems/openrouter.md
|
||||||
|
- [ 15d] systems/paseo.md
|
||||||
|
- [ 7d] systems/push-notifications.md
|
||||||
|
- [ 0d] systems/scheduled-tasks.md
|
||||||
|
- [ 0d] systems/skills-index.md
|
||||||
|
- [ 6d] systems/terminal-backend.md
|
||||||
|
- [ 6d] systems/tool-search.md
|
||||||
|
|
||||||
|
## NO updated DATE (4)
|
||||||
|
|
||||||
|
- SCHEMA.md
|
||||||
|
- experiments/active/example.md
|
||||||
|
- index.md
|
||||||
|
- log.md
|
||||||
|
|
||||||
|
TOTAL: 0 stale, 103 fresh, 4 no-date of 107 pages
|
||||||
+30
-11
@@ -3,9 +3,9 @@ title: Provider Health Check
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-04
|
updated: 2026-10-26
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-26
|
||||||
last_tested: 2026-07-22
|
last_tested: 2026-10-26
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [runbook, providers, health, routing]
|
tags: [runbook, providers, health, routing]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -23,7 +23,7 @@ Verify Hermes can reach LiteLLM, OmniRoute, OpenRouter-style fallbacks, and the
|
|||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
- Network access from this host to `litellm:4000`, `omniroute:20128`, `opencode.ai`, `api.telegram.org`, Telegram fallback IPs
|
- Network access from this host to `litellm:4000`, `omniroute:20128`, `opencode.ai`, `api.telegram.org`, Telegram fallback IPs
|
||||||
- Hermes config slocated at `~/.hermes/config.yaml`
|
- Hermes config located at `~/.hermes/config.yaml`
|
||||||
|
|
||||||
## Procedure
|
## Procedure
|
||||||
|
|
||||||
@@ -45,14 +45,32 @@ python3 -c "
|
|||||||
import urllib.request, json, yaml
|
import urllib.request, json, yaml
|
||||||
with open('/home/hermes/.hermes/config.yaml') as f:
|
with open('/home/hermes/.hermes/config.yaml') as f:
|
||||||
cfg = yaml.safe_load(f)
|
cfg = yaml.safe_load(f)
|
||||||
key = [p['api_key'] for p in cfg.get('custom_providers',[]) if p.get('name')=='omniroute'][0]
|
import os
|
||||||
req = urllib.request.Request('http://omniroute:20128/v1/models', headers={'Authorization': f'Bearer {key}'})
|
# NOTE: the provider was renamed 'omniroute' -> 'Omniroute:20128' and the key moved
|
||||||
|
# from an inline 'api_key' to a 'key_env' indirection. The old snippet raised IndexError.
|
||||||
|
p = [x for x in cfg.get('custom_providers',[]) if x.get('name')=='Omniroute:20128'][0]
|
||||||
|
key = os.environ[p['key_env']]
|
||||||
|
req = urllib.request.Request(p['base_url'] + '/models', headers={'Authorization': f'Bearer {key}'})
|
||||||
with urllib.request.urlopen(req, timeout=5) as r:
|
with urllib.request.urlopen(req, timeout=5) as r:
|
||||||
data = json.loads(r.read())
|
data = json.loads(r.read())
|
||||||
print('OmniRoute OK, models:', len(data.get('data', [])))
|
print('OmniRoute OK, models:', len(data.get('data', [])))
|
||||||
"
|
"
|
||||||
```
|
```
|
||||||
Current verified result: reachable and returns a non-empty model catalog.
|
Current verified result (2026-10-26): reachable and returns a non-empty model catalog — **6,480 models** on `omniroute:20128` (CT600, 100.88.81.19).
|
||||||
|
|
||||||
|
### 3b. Check the default `omni-101` provider (CT101, 100.78.63.20)
|
||||||
|
The **default provider** is `custom:omni-101` → `http://100.78.63.20:20128`, keyed via
|
||||||
|
`HERMES_CUSTOM_100_78_63_20_20128_API_KEY`. Probe it with its **own** key — the CT600 key is rejected here:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s --max-time 8 -H "Authorization: Bearer $HERMES_CUSTOM_100_78_63_20_20128_API_KEY" \
|
||||||
|
http://100.78.63.20:20128/v1/models | head -c 200
|
||||||
|
```
|
||||||
|
|
||||||
|
⚠ **Known quirk (verified 2026-10-26):** `/v1/models` on this node returns
|
||||||
|
`Invalid API key` even with the correct key, while `/v1/chat/completions` **works**
|
||||||
|
(a `smart-route` call returned `minimaxai/minimax-m3.1-flash`). So a `/v1/models`
|
||||||
|
check on `omni-101` will read as a false failure — **smoke-test the chat endpoint instead**.
|
||||||
|
|
||||||
### 4. Check fallback provider endpoints
|
### 4. Check fallback provider endpoints
|
||||||
```bash
|
```bash
|
||||||
@@ -84,10 +102,11 @@ Look for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or
|
|||||||
- Gateway journal shows no red alarm pattern.
|
- Gateway journal shows no red alarm pattern.
|
||||||
|
|
||||||
## Current install references
|
## Current install references
|
||||||
- Hermes: v0.21.5+6169.gab86a20 (2026.9.24), git install: `/home/hermes/.hermes/hermes-agent`
|
- Hermes: **v0.22.3** (2026.9.24), local `5b147ab036d6`, git install: `/home/hermes/.hermes/hermes-agent` (`hermes update --check` reports stable **v0.22.5** available)
|
||||||
- LiteLLM base URL: `http://litellm:4000/v1`
|
- LiteLLM base URL: `http://litellm:4000/v1`
|
||||||
- OmniRoute base URL: `http://omniroute:20128/v1`
|
- OmniRoute base URL: `http://omniroute:20128/v1` (provider name `Omniroute:20128`, key env `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`)
|
||||||
- OmniRoute direct IP: `100.88.81.19:20128`
|
- OmniRoute direct IP (CT600): `100.88.81.19:20128`
|
||||||
|
- Default provider `omni-101` (CT101): `http://100.78.63.20:20128` (key env `HERMES_CUSTOM_100_78_63_20_20128_API_KEY`) — ⚠ `/v1/models` returns a false `Invalid API key`; chat works
|
||||||
- WebUI: `http://0.0.0.0:8787/health`
|
- WebUI: `http://0.0.0.0:8787/health`
|
||||||
|
|
||||||
## Rollback
|
## Rollback
|
||||||
@@ -95,7 +114,7 @@ Look for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or
|
|||||||
- If you change fallback providers during remediation, record and revert via `hermes fallback list` + previous wiki state
|
- If you change fallback providers during remediation, record and revert via `hermes fallback list` + previous wiki state
|
||||||
|
|
||||||
## Last tested
|
## Last tested
|
||||||
2026-07-22
|
2026-10-26 — LiteLLM 401 unauthed (expected), OmniRoute/CT600 6,480 models, WebUI `/health` 200, opencode/nous/ollama fallbacks all 200
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[model-providers]]
|
- [[model-providers]]
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Update Hermes safely
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-04
|
updated: 2026-10-26
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-26
|
||||||
last_tested: 2026-07-22
|
last_tested: 2026-10-26
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [runbook, hermes, update]
|
tags: [runbook, hermes, update]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -24,6 +24,7 @@ Update Hermes to a newer release without losing configuration, skills, or sessio
|
|||||||
## Prerequisites
|
## Prerequisites
|
||||||
- Hermes install method: git-based install at `/home/hermes/.hermes/hermes-agent`
|
- Hermes install method: git-based install at `/home/hermes/.hermes/hermes-agent`
|
||||||
- Working Python virtualenv at `/home/hermes/.hermes/hermes-agent/venv`
|
- Working Python virtualenv at `/home/hermes/.hermes/hermes-agent/venv`
|
||||||
|
- **Managed runtimes live under `~/.hermes/installs/<install>/environments/<hash>/venv`** — these are what the gateway actually imports from; a missing dep there breaks cron jobs with no obvious trace (see [[incidents/2026-09-28-hermes-managed-runtime-deps]])
|
||||||
- Available disk space in Hermes install and backup paths
|
- Available disk space in Hermes install and backup paths
|
||||||
- No active Hermes chat/webui shell issues before starting
|
- No active Hermes chat/webui shell issues before starting
|
||||||
|
|
||||||
@@ -53,7 +54,17 @@ ls -la ~/.hermes/config.yaml.bak*
|
|||||||
```bash
|
```bash
|
||||||
cd /home/hermes/.hermes/hermes-agent
|
cd /home/hermes/.hermes/hermes-agent
|
||||||
git status --short
|
git status --short
|
||||||
hermes update --yes --branch main
|
hermes update --yes
|
||||||
|
```
|
||||||
|
Use `--branch main` only when deliberately tracking a branch other than the configured
|
||||||
|
channel. The **stable** channel is currently the default and offers **v0.22.5**
|
||||||
|
(`hermes update --check`, verified 2026-10-26; local install is v0.22.3 / `5b147ab036d6`).
|
||||||
|
`--force-venv` rebuilds the virtualenv — avoid unless the venv itself is broken.
|
||||||
|
|
||||||
|
**After any update, re-assert the managed-runtime dependencies** (the updater can
|
||||||
|
leave a runtime without `fastembed`/`dotenv`/`mnemosyne`):
|
||||||
|
```bash
|
||||||
|
~/.hermes/scripts/hermes-runtime-deps-guard.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
### 5. Verify the new version
|
### 5. Verify the new version
|
||||||
@@ -93,7 +104,7 @@ hermes dashboard restart
|
|||||||
- If update warns about config migration, revert the backup and re-run manually
|
- If update warns about config migration, revert the backup and re-run manually
|
||||||
|
|
||||||
## Notes
|
## Notes
|
||||||
- Hermes version on this install: v0.21.5+6169.gab86a20 (2026.9.24); install tree rebuilt 2026-10-03 10:11 AWST
|
- Hermes version on this install: **v0.22.3** (2026.9.24), local `5b147ab036d6`; install tree rebuilt 2026-10-21 22:47→22:57 AWST. Stable **v0.22.5** available as of 2026-10-26
|
||||||
- Install directory: `/home/hermes/.hermes/hermes-agent`
|
- Install directory: `/home/hermes/.hermes/hermes-agent`
|
||||||
- Update preserves venv by default unless `--force-venv` is used
|
- Update preserves venv by default unless `--force-venv` is used
|
||||||
|
|
||||||
@@ -118,7 +129,7 @@ pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"
|
|||||||
**After:** `pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"`, confirm CT460 production is untouched (separate config/skills/cron/memory). Full record: [[incidents/2026-10-03-ct450-hermes-update]].
|
**After:** `pct exec 450 -- su - hermes -c "/home/hermes/.local/bin/hermes --version"`, confirm CT460 production is untouched (separate config/skills/cron/memory). Full record: [[incidents/2026-10-03-ct450-hermes-update]].
|
||||||
|
|
||||||
## Last tested
|
## Last tested
|
||||||
2026-07-22 on debian 13, git-based Hermes install
|
2026-10-26 on debian 13, git-based Hermes install — `hermes update --check` (stable v0.22.5 available), `hermes backup --help` and `hermes update --help` flags confirmed
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[restart-hermes]]
|
- [[restart-hermes]]
|
||||||
|
|||||||
+14
-5
@@ -3,8 +3,8 @@ title: Conduit
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-10-24
|
created: 2026-10-24
|
||||||
updated: 2026-10-25
|
updated: 2026-10-26
|
||||||
verified_on: 2026-10-25
|
verified_on: 2026-10-26
|
||||||
review_after: 2026-11-08
|
review_after: 2026-11-08
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling]
|
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling]
|
||||||
@@ -21,9 +21,17 @@ be started from the phone where the agent can speak and listen (`conduit-voice`
|
|||||||
Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[systems/hermes-ui]]
|
Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[systems/hermes-ui]]
|
||||||
(the browser desktop), [[systems/push-notifications]] (the older `hermes-push` bridge).
|
(the browser desktop), [[systems/push-notifications]] (the older `hermes-push` bridge).
|
||||||
|
|
||||||
|
## Runtime checks (2026-10-26 00:1x AWST)
|
||||||
|
- **Working tree:** M dashboard/plugin_api.py — the uncommitted memory-timeout patch is **still present**.
|
||||||
|
- **`conduit-calls.json` → `history` entries:** **0** — the call-request path remains unused
|
||||||
|
→ [[incidents/2026-10-24-conduit-call-request-refused]].
|
||||||
|
- **Tool use in-window:** `conduit_show` ×1 — the plugin works for cards; the open incident concerns *placing calls*.
|
||||||
|
|
||||||
## Where it lives
|
## Where it lives
|
||||||
- **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push`
|
- **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push`
|
||||||
- **Version:** `0.69.0` (`plugin.yaml` and `dashboard/manifest.json`), git HEAD `f000bf0` —
|
- **Version (re-verified 2026-10-26):** `0.69.2` (plugin.yaml) / `0.69.2` (dashboard manifest), git HEAD `942ddb8` — *Plugin and relay fixes left over from earlier reviews (0.69.2) (#342)*, committed 2026-10-24 15:35:42 +0000. **0 commits behind `origin/main`**; **24 commits on from the `0.64.0` / `eaa14be` baseline** of the 2026-10-24 audit (upstream `#324`–`#342`: Pulse cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway restart, `channels.yaml`, App Store builds following the archived plugin version).
|
||||||
|
- **⚠ Superseded provenance:** the 2026-10-24 audit read `0.69.0` / `f000bf0`; two further commits landed on
|
||||||
|
2026-10-24 15:35. Git HEAD was previously recorded as: `- **Version:** `0.69.0` (`plugin.yaml` and `dashboard/manifest.json`), git HEAD `f000bf0` —
|
||||||
*“Pulse: list candidates, decide in code, journal them (#337)”*. **22 commits on from the
|
*“Pulse: list candidates, decide in code, journal them (#337)”*. **22 commits on from the
|
||||||
`0.64.0` / `eaa14be` recorded at the 2026-10-24 audit** (upstream `#324`–`#338`: Pulse
|
`0.64.0` / `eaa14be` recorded at the 2026-10-24 audit** (upstream `#324`–`#338`: Pulse
|
||||||
cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway
|
cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway
|
||||||
@@ -41,8 +49,9 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste
|
|||||||
`conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`.
|
`conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`.
|
||||||
|
|
||||||
## Current state (verified 2026-10-25 00:2x–00:5x AWST)
|
## Current state (verified 2026-10-25 00:2x–00:5x AWST)
|
||||||
- **Dashboard gateway:** listening on `0.0.0.0:9119`; process started **2026-10-23 18:54:43** and
|
- **Dashboard gateway (re-verified 2026-10-26):** listening on `0.0.0.0:9119`; process started **2026-10-25
|
||||||
**not restarted since** — so it is still the process that picked up the memory-timeout patch, and
|
18:42:33** — **restarted since the 2026-10-24 entry, so the upstream commits are now live** (the recorded
|
||||||
|
*"not restarted since / not live"* note is superseded, not wrong at the time) — so it is still the process that picked up the memory-timeout patch, and
|
||||||
the `0.64.0` → `0.69.0` upstream commits are **not** live.
|
the `0.64.0` → `0.69.0` upstream commits are **not** live.
|
||||||
- **Device:** one registered — **iPhone** (`id 70aa63f3…`, `kid c24e8215…`), app version `2.0.0`,
|
- **Device:** one registered — **iPhone** (`id 70aa63f3…`, `kid c24e8215…`), app version `2.0.0`,
|
||||||
features `chat-cards`, `info-cards`, `live-cards`, `maps-navigate`, `media-cards`, `show-cards`;
|
features `chat-cards`, `info-cards`, `live-cards`, `maps-navigate`, `media-cards`, `show-cards`;
|
||||||
|
|||||||
+32
-22
@@ -3,9 +3,9 @@ title: Current Hermes Profile
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-21
|
updated: 2026-10-26
|
||||||
verified_on: 2026-10-21
|
verified_on: 2026-10-26
|
||||||
review_after: 2026-10-28
|
review_after: 2026-11-09
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system, hermes, profile]
|
tags: [system, hermes, profile]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026-10-21]
|
sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026-10-21]
|
||||||
@@ -18,27 +18,37 @@ sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026
|
|||||||
- **Location:** implicit — the top-level `~/.hermes/config.yaml`. There is **no** `~/.hermes/profiles/default/` directory (verified 2026-10-02); `~/.hermes/profiles/` holds only named sub-profiles.
|
- **Location:** implicit — the top-level `~/.hermes/config.yaml`. There is **no** `~/.hermes/profiles/default/` directory (verified 2026-10-02); `~/.hermes/profiles/` holds only named sub-profiles.
|
||||||
- **Other profiles present:** `ops` only (`~/.hermes/profiles/ops/`) — the Tusk elephant persona, with its own SOUL.md, skills, plugins and memories (last modified 2026-09-30). Other profiles do **not** affect this session; see the profile-isolation note below.
|
- **Other profiles present:** `ops` only (`~/.hermes/profiles/ops/`) — the Tusk elephant persona, with its own SOUL.md, skills, plugins and memories (last modified 2026-09-30). Other profiles do **not** affect this session; see the profile-isolation note below.
|
||||||
|
|
||||||
## Key Settings (model routing re-verified against live config 2026-10-21; remaining items as recorded 2026-10-02)
|
## Key Settings (model routing + auxiliary slots re-verified against live config 2026-10-26; remaining items as recorded 2026-10-02)
|
||||||
|
|
||||||
### ⚠ Model default moved OFF OmniRoute (2026-10-20)
|
### ⚠ Model default moved BACK ONTO OmniRoute (reverted 2026-10-24 19:06)
|
||||||
- **Provider:** **`nous`** — `model.base_url: https://inference-api.nousresearch.com/v1`, `api_mode: chat_completions`.
|
- **Provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`),
|
||||||
- **Model default:** **`meituan/longcat-2.5-preview:free`**
|
`api_mode: chat_completions`.
|
||||||
- **When:** `config.yaml` last written **2026-10-20 22:01:38 AWST**. The move was **not** in place at the
|
- **Model default:** **`smart-route`** (paid alias — not a free combo).
|
||||||
2026-10-09 19:42 snapshot and **was** in place by 2026-10-20 11:00. Previous values (2026-10-02 record):
|
- **When:** a session switch note at **2026-10-24 19:06:10 AWST** reads *"model was just switched from
|
||||||
provider `custom` → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`, provider id `custom:omni-101`),
|
meituan/longcat-2.5-preview:free to smart-route via omni-101."* The 2026-10-21 config snapshot still read
|
||||||
default `smart-route`.
|
`nous` / `meituan/longcat-2.5-preview:free`, so the revert landed between **2026-10-20 22:01** and
|
||||||
- **The `model` block now carries only** `base_url`, `default`, `provider`, `api_mode`, `persist_switch_by_default`.
|
**2026-10-24 19:06**; `config.yaml` was last written **2026-10-25 19:13**. ⚠ The 2026-10-22, 10-24 and
|
||||||
**The 6-provider fallback chain recorded at the 2026-10-02 snapshot is gone** — there is no `fallbacks` key under
|
10-25 audits all recorded the 10-20 move as current — this section corrects them. The revert coincides with
|
||||||
`model`, so an unreachable `nous` is no longer covered by the old `custom` / `or-inclusionai_ling-2_6-1t` →
|
the **invalid Nous Portal refresh token** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||||
`custom:litellm` → `litellm:4000` chain.
|
- **The `model` block carries** `base_url`, `default`, `provider`, `api_mode`, `persist_switch_by_default` —
|
||||||
- **⚠ Cron pins did NOT move with it.** **20 of the 28 scheduled jobs still pin OmniRoute explicitly**
|
still **no `fallbacks` key under `model`**.
|
||||||
(`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). **The 8 jobs with no pin are
|
- **⚠ A top-level six-provider `fallback_providers` chain is live again** (re-verified 2026-10-26):
|
||||||
the ones whose upstream changed** — they now call `nous` direct instead of the router. The fleet therefore
|
`nous` / `meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` →
|
||||||
runs **two upstream paths for the first time**. → [[systems/scheduled-tasks]]
|
`custom:guacano` / `deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` →
|
||||||
- **Routing posture:** no fallback chain under `model`; per-job pins are the only routing control that remains.
|
`kilocode` / `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6-1t`.
|
||||||
|
The 2026-10-21 record that *no* fallback chain exists is **stale**. ⚠ The first entry is the **dead `nous`
|
||||||
|
credential** — a fallover would try it first.
|
||||||
|
- **⚠ Cron pins did NOT need to move.** **18 of the 28 scheduled jobs pin OmniRoute explicitly**
|
||||||
|
(`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). The 8 unpinned jobs follow
|
||||||
|
the global default, which is OmniRoute again — so the fleet is back to a **single upstream path**, not the
|
||||||
|
two-path split recorded on 2026-10-21. → [[systems/scheduled-tasks]]
|
||||||
|
- **Routing posture:** top-level `fallback_providers` chain + per-job pins are the routing controls;
|
||||||
|
`model.fallbacks` does not exist.
|
||||||
|
- ⚠ The `custom:omni` provider key is **no longer configured** (only 2 jobs still name it) — see
|
||||||
|
[[systems/model-providers]].
|
||||||
|
|
||||||
### Other settings (recorded 2026-10-02 — not re-verified 2026-10-21)
|
### Other settings (recorded 2026-10-02 — auxiliary line re-verified 2026-10-26)
|
||||||
- **Auxiliary vision model:** `stepfun/step-3.7-flash:free` on `nous`
|
- **Auxiliary slots (all 8 — `approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`, `title_generation`, `vision`):** `no-think/freeapi/auto` on `custom:omni-101`; per-slot fallbacks `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`; `free_only: true`; compression timeout 90 s, vision timeout 180 s. Re-placed 2026-10-25 19:13 — previously vision was `stepfun/step-3.7-flash:free` on `nous`. → [[systems/model-providers]]
|
||||||
- **Switch persistence:** `persist_switch_by_default: false`
|
- **Switch persistence:** `persist_switch_by_default: false`
|
||||||
- **Timezone:** `Australia/Perth`
|
- **Timezone:** `Australia/Perth`
|
||||||
- **Approvals mode:** `off` (`destructive_slash_confirm: false`, `mcp_reload_confirm: false`)
|
- **Approvals mode:** `off` (`destructive_slash_confirm: false`, `mcp_reload_confirm: false`)
|
||||||
|
|||||||
+19
-2
@@ -3,8 +3,8 @@ title: Hermes Agent
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-22
|
updated: 2026-10-26
|
||||||
verified_on: 2026-10-22
|
verified_on: 2026-10-26
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system, hermes, agent, llm]
|
tags: [system, hermes, agent, llm]
|
||||||
sources: [live-hermes-version, live-config, live-skill-inventory]
|
sources: [live-hermes-version, live-config, live-skill-inventory]
|
||||||
@@ -169,3 +169,20 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
|
|||||||
- **Mattermost platform unreachable** since 2026-10-10; the gateway keeps retrying on a 5-minute
|
- **Mattermost platform unreachable** since 2026-10-10; the gateway keeps retrying on a 5-minute
|
||||||
cadence - [[systems/messaging-integrations]].
|
cadence - [[systems/messaging-integrations]].
|
||||||
- **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]].
|
- **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]].
|
||||||
|
|
||||||
|
## 2026-10-26 00:1x AWST — gateway restart record and the dead Nous credential
|
||||||
|
|
||||||
|
- **Three gateway starts on 2026-10-25**: `00:32:20`, `18:30:04`, `19:27:13` AWST (after `2026-10-24 12:24:39`).
|
||||||
|
The `18:29:22` shutdown carried `signal=SIGTERM under_systemd=yes`, `loadavg_1m=3.79` — a planned stop, not a
|
||||||
|
crash; `19:26:16` logged *"Stopping gateway for restart..."* (the plugin self-update path). Current gateway
|
||||||
|
pid **2476414**, up since **19:26:33** (≈4 h 45 m at audit time).
|
||||||
|
- **Consequences of the 19:27 start:** the Conduit dashboard gateway restarted at 18:42:33 and so picked up the
|
||||||
|
plugin's upstream commits → [[systems/conduit]]; the Mattermost reconnect attempt counter reset to 54
|
||||||
|
→ [[systems/messaging-integrations]]; the `invalid_grant` spam stopped.
|
||||||
|
- ⚠ **The Nous Portal credential is dead** — `Nous Portal runtime credentials unavailable: Invalid refresh token`.
|
||||||
|
Gateway-log counts: **1 line 2026-10-22, 3 on 10-24, 20 on 10-25** (last 14:53:04,
|
||||||
|
`agent.context_compressor: Failed to generate context summary`), **0 since the 19:27:13 start**. Context
|
||||||
|
compression was unavailable for most of 10-25 and contributed to the digest runaway.
|
||||||
|
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||||
|
- **Restart hygiene:** a gateway restart resets in-process counters (Mattermost retries, cron admission
|
||||||
|
bookkeeping). Never read a counter as an outage duration.
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Memory Backend
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-04
|
updated: 2026-10-26
|
||||||
verified_on: 2026-10-04
|
verified_on: 2026-10-26
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [memory, mnemosyne, agent]
|
tags: [memory, mnemosyne, agent]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -30,7 +30,11 @@ Persist and recall agent memory across sessions.
|
|||||||
- **Shared surface read:** true
|
- **Shared surface read:** true
|
||||||
- **Sync roles:** user
|
- **Sync roles:** user
|
||||||
- **Skip contexts:** cron, flush, subagent, background, skill_loop
|
- **Skip contexts:** cron, flush, subagent, background, skill_loop
|
||||||
- **Reflect max calls per session:** 3
|
- **Reflect max calls per session:** 3 (`reflect.disabled_for_cron: true`)
|
||||||
|
- **Sleep threshold:** 50 turns
|
||||||
|
- **Nudge interval:** 10
|
||||||
|
- **Ignore patterns:** none set
|
||||||
|
- **Live totals (2026-10-26):** 2,350 memories — 842 working / 1,090 episodic / 418 `memoria_facts`
|
||||||
|
|
||||||
## Storage Location
|
## Storage Location
|
||||||
- Mnemosyne home: `~/.hermes/mnemosyne/`
|
- Mnemosyne home: `~/.hermes/mnemosyne/`
|
||||||
@@ -44,6 +48,7 @@ Persist and recall agent memory across sessions.
|
|||||||
- Health monitoring currently treats capture as stale only after 24 hours; an activity-aware comparison is preferable to catch dry-intake failures sooner without false alarms during idle periods.
|
- Health monitoring currently treats capture as stale only after 24 hours; an activity-aware comparison is preferable to catch dry-intake failures sooner without false alarms during idle periods.
|
||||||
- The 2026-08-01 automatic-capture failure and repair are documented in [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
- The 2026-08-01 automatic-capture failure and repair are documented in [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
||||||
- **Vector recall degraded silently until 2026-10-03** — `fastembed` was missing from every uv-managed runtime, so `mnemosyne_recall` fell back to FTS5 keyword scoring with no error raised. Repaired 2026-10-03; the 15-min dependency guard now asserts `fastembed` and the model cache path. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
- **Vector recall degraded silently until 2026-10-03** — `fastembed` was missing from every uv-managed runtime, so `mnemosyne_recall` fell back to FTS5 keyword scoring with no error raised. Repaired 2026-10-03; the 15-min dependency guard now asserts `fastembed` and the model cache path. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
||||||
|
- **Embedding coverage re-verified 2026-10-26** — `check_embedding_coverage.py` reports 1,899 / 2,350 (80.8 %) embedded; **working 815 / 842 (96.8 %)** and **episodic 1,084 / 1,090 (99.4 %)**. Both tiers are **above the 95 % reindex threshold**, so no reindex was triggered. `fastembed` 0.8.1 is present in **both** managed runtimes and 0.8.0 in the legacy venv; `FASTEMBED_CACHE_PATH` is pinned. The 2026-10-03 follow-up (assert `fastembed` in *all* managed runtimes) is now satisfied.
|
||||||
|
|
||||||
**Correction (2026-07-22, per Anthony):** this page previously referenced Hindsight as a possibly-used backend on this box. That was a mix-up with a different Hermes instance — see [[comparisons/memory-backends]].
|
**Correction (2026-07-22, per Anthony):** this page previously referenced Hindsight as a possibly-used backend on this box. That was a mix-up with a different Hermes instance — see [[comparisons/memory-backends]].
|
||||||
|
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Messaging Integrations
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-19
|
updated: 2026-10-26
|
||||||
verified_on: 2026-10-19
|
verified_on: 2026-10-26
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [messaging, telegram, discord, email, integrations]
|
tags: [messaging, telegram, discord, email, integrations]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
|
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
|
||||||
@@ -113,3 +113,15 @@ This pass replaced the 10-11 observation's "cause not determined" with a definit
|
|||||||
|
|
||||||
**Open item:** start CT255, or retire the platform block. The 5-minute reconnect loop continues until
|
**Open item:** start CT255, or retire the platform block. The 5-minute reconnect loop continues until
|
||||||
one of those happens — restarting the adapter while the host is stopped achieves nothing.
|
one of those happens — restarting the adapter while the host is stopped achieves nothing.
|
||||||
|
|
||||||
|
## 2026-10-26 00:1x AWST — Mattermost still down; attempt counter reset by the 10-25 restart
|
||||||
|
|
||||||
|
- **Cause unchanged: CT255 (`mattermost`) is `stopped`.** `tailscale status` shows the peer **offline, last seen
|
||||||
|
36 d ago**, `tx 312 / rx 0` — `rx` has read 0 in every audit since 2026-10-11, i.e. no packet has come back.
|
||||||
|
- **Reconnect loop still running:** **268** `mattermost connect timed out` lines on 2026-10-25 vs **264** on
|
||||||
|
2026-10-24 (≈1 per 5 min, as designed); **2** so far on 2026-10-26.
|
||||||
|
- **The attempt counter reset to 54** with the 19:27:13 gateway restart (`Reconnecting mattermost (attempt 54)`
|
||||||
|
at 2026-10-26 00:07:06). The previous reset was the 2026-10-17 05:10 restart. ⚠ The counter measures **this
|
||||||
|
process's** attempts, not the outage's length — do not quote it as downtime.
|
||||||
|
- **Impact unchanged:** messaging path only; no cron job delivers via Mattermost.
|
||||||
|
- **Not attempted:** `pct start 255` — starting a service on the PVE host is Anthony's call.
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Model Providers
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-23
|
updated: 2026-10-26
|
||||||
verified_on: 2026-10-23
|
verified_on: 2026-10-26
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [model, provider, inference, routing]
|
tags: [model, provider, inference, routing]
|
||||||
sources: [live-probe, config, web-search]
|
sources: [live-probe, config, web-search]
|
||||||
@@ -20,27 +20,56 @@ Verified from live `~/.hermes/config.yaml` → `custom_providers`, 2026-10-01.
|
|||||||
|
|
||||||
| Provider | URL | Models | Role |
|
| Provider | URL | Models | Role |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| `custom:omni-101` | `http://100.78.63.20:20128` | 5,384 | **Primary.** OmniRoute on the estate (`omni-101`). 15 of 30 cron jobs reference it |
|
| `custom:omni-101` | `http://100.78.63.20:20128` | 5,384 | **Primary.** OmniRoute on the estate (`omni-101`). 18 of 28 cron jobs reference it (2026-10-26) |
|
||||||
| `custom:Omniroute:20128` | `http://omniroute:20128/v1` | 5,398 | Secondary OmniRoute host — legacy naming, kept alongside `omni-101` |
|
| `custom:Omniroute:20128` | `http://omniroute:20128/v1` | 5,398 | Secondary OmniRoute host — legacy naming, kept alongside `omni-101` |
|
||||||
| `custom:litellm` | `http://litellm:4000/v1` | 1,728 | Fallback inference (slower path, 6–26s/call) |
|
| `custom:litellm` | `http://litellm:4000/v1` | 1,728 | Fallback inference (slower path, 6–26s/call) |
|
||||||
| `custom:command-code` | `https://api.commandcode.ai/provider/v1` | 71 | DeepSeek Flash primary via the `deepseek-flash-mix` combo (added 2026-08-19); also the source of the `MODEL_NOT_IN_PLAN` 403 on `gpt-5.6-terra` |
|
| `custom:command-code` | `https://api.commandcode.ai/provider/v1` | 71 | DeepSeek Flash primary via the `deepseek-flash-mix` combo (added 2026-08-19); also the source of the `MODEL_NOT_IN_PLAN` 403 on `gpt-5.6-terra` |
|
||||||
| `custom:guacano` | `http://ollama:11000/v1` | 46 | Guanaco LLM proxy/router on CT205 (Ollama stack). See [[systems/guanaco]] |
|
| `custom:guacano` | `http://ollama:11000/v1` | 46 | Guanaco LLM proxy/router on CT205 (Ollama stack). See [[systems/guanaco]] |
|
||||||
| `custom:omni` | *(removed)* | — | **No longer configured.** It once served auxiliary roles (vision, skills, approval, mcp, web_extract, compression). Two cron jobs still name it as their provider key — see [[systems/scheduled-tasks]] |
|
| `custom:omni` | *(removed)* | — | **No longer configured.** It once served auxiliary roles (vision, skills, approval, mcp, web_extract, compression). Two cron jobs still name it as their provider key — see [[systems/scheduled-tasks]] |
|
||||||
|
|
||||||
**Built-in (non-`custom_providers`) routes still in use as fallbacks:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`. `charm-hyper` (Charmbracelet, `hyper.charm.land/v1`, free tier 100 credits/mo) is no longer in the live stack.
|
**Built-in (non-`custom_providers`) routes still in use as fallbacks:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`. `charm-hyper` (Charmbracelet, `hyper.charm.land/v1`, free tier 100 credits/mo) is no longer in the live stack. **The live `fallback_providers` chain (2026-10-26) is `nous` / `meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` → `custom:guacano` / `deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` → `kilocode` / `poolside/laguna-m.1:free` → `custom` / `or-inclusionai_ling-2_6-1t`.**
|
||||||
|
|
||||||
**Default provider:** **`nous`** → `https://inference-api.nousresearch.com/v1`. **Default model:** **`meituan/longcat-2.5-preview:free`**.
|
**Default provider:** **`custom:omni-101`** → `http://100.78.63.20:20128`. **Default model:** **`smart-route`**.
|
||||||
The default moved OFF OmniRoute on 2026-10-20 — see below. The `custom` provider key is no longer the default.
|
⚠ **The default moved BACK to OmniRoute on 2026-10-24 19:06:10 AWST** (session switch note: *"model was just
|
||||||
|
switched from meituan/longcat-2.5-preview:free to smart-route via omni-101"*). The `nous` direct window ran
|
||||||
|
2026-10-20 22:01 → 2026-10-24 19:06 and the 2026-10-22/10-24/10-25 audit entries recorded it as current —
|
||||||
|
**stale**. `config.yaml` last written 2026-10-25 19:13.
|
||||||
|
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||||
|
|
||||||
Provider names that appear in job configs but not in `custom_providers` do **not** necessarily fail: the default `custom` block resolves to the same base URL, so a stale provider key can go unnoticed. Prefer routing aliases (`smart-route`) over raw upstream model names in pins.
|
Provider names that appear in job configs but not in `custom_providers` do **not** necessarily fail: the default `custom` block resolves to the same base URL, so a stale provider key can go unnoticed. Prefer routing aliases (`smart-route`) over raw upstream model names in pins.
|
||||||
|
|
||||||
## Default Model
|
## Default Model
|
||||||
- **`meituan/longcat-2.5-preview:free`** on **`nous`** (`https://inference-api.nousresearch.com/v1`, `api_mode: chat_completions`) — the live default since 2026-10-20. Verified against live `~/.hermes/config.yaml` 2026-10-23.
|
- **`smart-route`** on **`custom:omni-101`** (`http://100.78.63.20:20128`, `api_mode: chat_completions`) — the
|
||||||
- **No fallback chain under `model`.** The 6-provider fallback recorded at the 2026-10-02 snapshot is gone; `model` now carries only `base_url`, `default`, `provider`, `api_mode`, `persist_switch_by_default`. An unreachable `nous` has no automatic failover — per-job pins are the only routing control that remains.
|
live default since **2026-10-24 19:06:10 AWST**, re-verified against live `~/.hermes/config.yaml`
|
||||||
|
**2026-10-26 00:1x**.
|
||||||
|
- **⚠ A six-provider top-level `fallback_providers` chain is live again** (re-verified 2026-10-26): `nous` /
|
||||||
|
`meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` → `custom:guacano` /
|
||||||
|
`deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` → `kilocode` /
|
||||||
|
`poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6-1t`. The
|
||||||
|
2026-10-21 claim that **no** fallback chain exists is **stale**; the `model` block itself still carries only
|
||||||
|
`base_url`, `default`, `provider`, `api_mode`, `persist_switch_by_default`.
|
||||||
|
- ⚠ **The first fallback entry is the dead `nous` credential** (`Invalid refresh token`, since 2026-10-22)
|
||||||
|
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||||
- Anthony has explicitly **rejected** `free-lkgp` as a default; stable defaults are `smart-route` / `deepseek-paid-smart` (paid, not free)
|
- Anthony has explicitly **rejected** `free-lkgp` as a default; stable defaults are `smart-route` / `deepseek-paid-smart` (paid, not free)
|
||||||
- `resilient-free` was the default after the 2026-08-14 OmniRoute factory reset, but that combo is now in the broken set
|
- `resilient-free` was the default after the 2026-08-14 OmniRoute factory reset, but that combo is now in the broken set
|
||||||
- Previous default `auto/best-free` was misconfigured — had no combo definition and routed to a paid `gpt-5.6-sol`
|
- Previous default `auto/best-free` was misconfigured — had no combo definition and routed to a paid `gpt-5.6-sol`
|
||||||
- **20 of 28 scheduled jobs still pin OmniRoute explicitly** (`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). The fleet now runs two upstream paths. → [[systems/scheduled-tasks]]
|
- **18 of 28 scheduled jobs pin OmniRoute explicitly** (`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). The remaining 8 follow the global default, which is **also** OmniRoute — so the fleet is back to a **single upstream path**. → [[systems/scheduled-tasks]]
|
||||||
|
|
||||||
|
## Auxiliary Role Routing (re-verified live 2026-10-26)
|
||||||
|
**All eight `auxiliary:` slots** — `approval`, `compression`, `curator`, `mcp`, `profile_describer`,
|
||||||
|
`skills_hub`, `title_generation`, `vision` — are **`no-think/freeapi/auto` on `custom:omni-101`**, each with
|
||||||
|
the chain `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` and `free_only: true`.
|
||||||
|
**Compression timeout 90 s; vision timeout 180 s.**
|
||||||
|
|
||||||
|
- Re-placed **2026-10-25 19:13 AWST**, replacing the `nous` / `stepfun/step-3.7-flash:free` and `custom:omni`
|
||||||
|
aux paths recorded on 2026-10-02 and 2026-10-25 (17:50).
|
||||||
|
- ⚠ The move was made **because the Nous Portal credential is dead** (`Invalid refresh token`), so it **masks
|
||||||
|
rather than fixes** that fault → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||||
|
- Probe results from that session (17:50–19:13, not re-run 2026-10-26): `freeapi/auto` healthy for text and
|
||||||
|
vision; `kg/kilo-auto/free` returns **empty content on image input**; `stepfun/step-3.7-flash:free` → 401
|
||||||
|
*no active credentials*; freeapi pinned models in cooldown (429). ⚠ A **vision fallback returning empty
|
||||||
|
content** is a silent failure mode — check `freeapi/auto` itself, not the fallback order, if image calls
|
||||||
|
go blank.
|
||||||
|
|
||||||
## Retired / Unavailable Models (2026-09-25 → 2026-10-01)
|
## Retired / Unavailable Models (2026-09-25 → 2026-10-01)
|
||||||
Pins that fail at request time, with the exact upstream error. Each is an open cron-repin in [[systems/scheduled-tasks]].
|
Pins that fail at request time, with the exact upstream error. Each is an open cron-repin in [[systems/scheduled-tasks]].
|
||||||
|
|||||||
@@ -47,7 +47,11 @@ sources: [raw/configs/hermes-config-sanitized.txt]
|
|||||||
burst, not afterwards.
|
burst, not afterwards.
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Secondary provider/router for Hermes auxiliary roles and some agent routing.
|
Provider/router for Hermes — ⚠ **updated 2026-10-26: the `omniroute-1` node is the DEFAULT chat path again.**
|
||||||
|
Hermes's default provider is `custom:omni-101` → `http://100.78.63.20:20128` with model `smart-route`
|
||||||
|
(reverted 2026-10-24 19:06; the 2026-10-20 → 10-24 window pointed the default at `nous`).
|
||||||
|
The CT600 provider key `custom:omni` (`http://omniroute:20128/v1`) remains auxiliary-only.
|
||||||
|
→ [[systems/current-profile]]
|
||||||
|
|
||||||
## Current Configuration
|
## Current Configuration
|
||||||
- **Hermes provider name:** `custom:omni` (maps to provider `omniroute:20128`)
|
- **Hermes provider name:** `custom:omni` (maps to provider `omniroute:20128`)
|
||||||
@@ -56,7 +60,7 @@ Secondary provider/router for Hermes auxiliary roles and some agent routing.
|
|||||||
- **Runs on:** Proxmox CT600 (`omniroute`, Tailscale `100.88.81.19`, onboot=1) — verified running 2026-10-11
|
- **Runs on:** Proxmox CT600 (`omniroute`, Tailscale `100.88.81.19`, onboot=1) — verified running 2026-10-11
|
||||||
- **DB path:** `/opt/omniroute/storage.sqlite` (was `/var/lib/omniroute/storage.sqlite`)
|
- **DB path:** `/opt/omniroute/storage.sqlite` (was `/var/lib/omniroute/storage.sqlite`)
|
||||||
- **Backup:** pre-migration config saved at `~/.hermes/config.yaml.bak.omniroute-removal`
|
- **Backup:** pre-migration config saved at `~/.hermes/config.yaml.bak.omniroute-removal`
|
||||||
- **Miss:** Hermes default provider path does not use OmniRoute for main chat; used for auxiliary roles instead
|
- **Correction (2026-10-26):** the earlier *"Hermes default provider path does not use OmniRoute for main chat"* note is **stale** — the default is OmniRoute again (`custom:omni-101`, model `smart-route`). The **CT600** key `custom:omni` is still auxiliary-only.
|
||||||
- **Auxiliary roles in Hermes config:** `skills_hub`, `approval`, `mcp`
|
- **Auxiliary roles in Hermes config:** `skills_hub`, `approval`, `mcp`
|
||||||
- **Key prefix visible in config:** `sk-5c67fbdc0...` — full value not stored in wiki
|
- **Key prefix visible in config:** `sk-5c67fbdc0...` — full value not stored in wiki
|
||||||
|
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-25
|
updated: 2026-10-26
|
||||||
verified_on: 2026-10-25
|
verified_on: 2026-10-26
|
||||||
review_after: 2026-10-31
|
review_after: 2026-10-31
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, scheduled, automation]
|
tags: [cron, scheduled, automation]
|
||||||
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
|||||||
# Scheduled Tasks
|
# Scheduled Tasks
|
||||||
|
|
||||||
## Count
|
## Count
|
||||||
- **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page.
|
- **Current as of 2026-10-26 00:1x AWST (live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`).
|
||||||
|
- **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page. *(Superseded by the 2026-10-26 entry above.)*
|
||||||
- **Current as of 2026-10-24 00:2x AWST *(superseded by the 2026-10-25 entry above)* (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]]
|
- **Current as of 2026-10-24 00:2x AWST *(superseded by the 2026-10-25 entry above)* (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]]
|
||||||
- **Current as of 2026-10-23 00:1x AWST *(superseded by the 2026-10-24 entry above)* (`jobs.json.updated_at 2026-10-23T00:07:25+08:00`) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. **A DB recount by `started_at` and by `claimed_at` puts 2026-10-21 at 168 / 1**, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Five incidents closed in-window and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (`0e519874d280`, closed 08:01:40), `06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so **unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`: `1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, `c0a9fa_764558b5ae83`, `0e753e_e7f8194a7d06`). **Non-clean `last_status` falls 9 → 5:** `1b23c042e17d` (`error`, streak 2 — **next slot Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` (`delivery_failed` — Mon 10-26 03:00), `066bb5191280` (`delivery_failed` — 11-01). All four cleared rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its first slot of 10-22. **Pins:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (`provider`/`model` null → global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). ⚠ **Stocktake anomaly:** `1a8e90ea2271` ran **03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, finished `completed` with delivery `suppressed` — and **committed nothing**. It left its regenerated `systems/skills-index.md` uncommitted (mtime 10-22 03:00, with a `## .Archive (41)` section reading **347**) and its diagnosed fixes unapplied → [[incidents/2026-08-15-skills-index-drift]]. **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop` (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models `mimo-v2.5`, `or-inclusionai_ling-2_6-1t`, `minimax/minimax-m3.1-flash`), 2 `acp` (0 messages), 1 `oneshot` memory-capture probe. → [[systems/skills-index]], [[incidents/2026-08-15-skills-index-drift]]
|
- **Current as of 2026-10-23 00:1x AWST *(superseded by the 2026-10-24 entry above)* (`jobs.json.updated_at 2026-10-23T00:07:25+08:00`) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. **A DB recount by `started_at` and by `claimed_at` puts 2026-10-21 at 168 / 1**, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Five incidents closed in-window and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (`0e519874d280`, closed 08:01:40), `06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so **unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`: `1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, `c0a9fa_764558b5ae83`, `0e753e_e7f8194a7d06`). **Non-clean `last_status` falls 9 → 5:** `1b23c042e17d` (`error`, streak 2 — **next slot Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` (`delivery_failed` — Mon 10-26 03:00), `066bb5191280` (`delivery_failed` — 11-01). All four cleared rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its first slot of 10-22. **Pins:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (`provider`/`model` null → global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). ⚠ **Stocktake anomaly:** `1a8e90ea2271` ran **03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, finished `completed` with delivery `suppressed` — and **committed nothing**. It left its regenerated `systems/skills-index.md` uncommitted (mtime 10-22 03:00, with a `## .Archive (41)` section reading **347**) and its diagnosed fixes unapplied → [[incidents/2026-08-15-skills-index-drift]]. **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop` (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models `mimo-v2.5`, `or-inclusionai_ling-2_6-1t`, `minimax/minimax-m3.1-flash`), 2 `acp` (0 messages), 1 `oneshot` memory-capture probe. → [[systems/skills-index]], [[incidents/2026-08-15-skills-index-drift]]
|
||||||
- **Current as of 2026-10-22 00:1x AWST *(superseded by the 2026-10-23 entry above)* (`jobs.json.updated_at 2026-10-22T00:11:31+08:00`) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen.** Window 2026-10-21: **171 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`), which died at 07:40:48 after **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string, which appears in no 2026-10-21 row → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` stays at 9** (7 `error` / 2 `delivery_failed`) but the composition moved: **`08eb86f02d37` cleared** (its 08:30 slot ran clean; incident `08eb86_472c7b609c80` **resolved 2026-10-21 08:37:21**) while **`dca8482e4f76` re-entered** (streak 1). Survivors, with next run: `dca8482e4f76` (1, **10-22 07:00**), `0e519874d280` (2, **10-22 08:00**), `06a0a1c8704f` (1, **10-22 10:00**), `c8d52904e1d6` (1, **10-22 18:00**), `1b23c042e17d` (2, Fri 10-23 08:00), `c0a9fa314542` (1, Mon 10-26 09:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 11-01 02:00). **Incident ledger: 1 opened / 1 closed, unclosed stays 9** (8 `alerted` / 1 `detected`). **Pins:** 17 of the 28 jobs pin `custom:omni-101` (10 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`), **7 carry no pin at all** (`provider`/`model` null → global default), 1 pins `custom:omni` / `smart-route`. **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. **12 sessions in the window, all `cron` — no interactive sessions.** ⚠ **Infrastructure changed in-window: Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]], [[infrastructure/backups]].
|
- **Current as of 2026-10-22 00:1x AWST *(superseded by the 2026-10-23 entry above)* (`jobs.json.updated_at 2026-10-22T00:11:31+08:00`) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen.** Window 2026-10-21: **171 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`), which died at 07:40:48 after **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string, which appears in no 2026-10-21 row → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` stays at 9** (7 `error` / 2 `delivery_failed`) but the composition moved: **`08eb86f02d37` cleared** (its 08:30 slot ran clean; incident `08eb86_472c7b609c80` **resolved 2026-10-21 08:37:21**) while **`dca8482e4f76` re-entered** (streak 1). Survivors, with next run: `dca8482e4f76` (1, **10-22 07:00**), `0e519874d280` (2, **10-22 08:00**), `06a0a1c8704f` (1, **10-22 10:00**), `c8d52904e1d6` (1, **10-22 18:00**), `1b23c042e17d` (2, Fri 10-23 08:00), `c0a9fa314542` (1, Mon 10-26 09:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 11-01 02:00). **Incident ledger: 1 opened / 1 closed, unclosed stays 9** (8 `alerted` / 1 `detected`). **Pins:** 17 of the 28 jobs pin `custom:omni-101` (10 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`), **7 carry no pin at all** (`provider`/`model` null → global default), 1 pins `custom:omni` / `smart-route`. **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. **12 sessions in the window, all `cron` — no interactive sessions.** ⚠ **Infrastructure changed in-window: Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]], [[infrastructure/backups]].
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-25
|
updated: 2026-10-26
|
||||||
verified_on: 2026-10-25
|
verified_on: 2026-10-26
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [skills, hermes]
|
tags: [skills, hermes]
|
||||||
sources: []
|
sources: []
|
||||||
|
|||||||
Reference in New Issue
Block a user