audit(2026-10-26): runaway Gmail digest + dead Nous credential; model default reverted to OmniRoute; conduit 0.69.2 live; freshness floor still 2026-10-08

This commit is contained in:
Tony0410
2026-10-26 00:19:37 +08:00
parent 2df1bce5ea
commit 123a74cb0b
16 changed files with 433 additions and 59 deletions
+14 -5
View File
@@ -3,8 +3,8 @@ title: Conduit
type: system
status: active
created: 2026-10-24
updated: 2026-10-25
verified_on: 2026-10-25
updated: 2026-10-26
verified_on: 2026-10-26
review_after: 2026-11-08
confidence: high
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling]
@@ -21,9 +21,17 @@ be started from the phone where the agent can speak and listen (`conduit-voice`
Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[systems/hermes-ui]]
(the browser desktop), [[systems/push-notifications]] (the older `hermes-push` bridge).
## Runtime checks (2026-10-26 00:1x AWST)
- **Working tree:** M dashboard/plugin_api.py — the uncommitted memory-timeout patch is **still present**.
- **`conduit-calls.json` → `history` entries:** **0** — the call-request path remains unused
→ [[incidents/2026-10-24-conduit-call-request-refused]].
- **Tool use in-window:** `conduit_show` ×1 — the plugin works for cards; the open incident concerns *placing calls*.
## Where it lives
- **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push`
- **Version:** `0.69.0` (`plugin.yaml` and `dashboard/manifest.json`), git HEAD `f000bf0` —
- **Version (re-verified 2026-10-26):** `0.69.2` (plugin.yaml) / `0.69.2` (dashboard manifest), git HEAD `942ddb8` — *Plugin and relay fixes left over from earlier reviews (0.69.2) (#342)*, committed 2026-10-24 15:35:42 +0000. **0 commits behind `origin/main`**; **24 commits on from the `0.64.0` / `eaa14be` baseline** of the 2026-10-24 audit (upstream `#324`–`#342`: Pulse cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway restart, `channels.yaml`, App Store builds following the archived plugin version).
- **⚠ Superseded provenance:** the 2026-10-24 audit read `0.69.0` / `f000bf0`; two further commits landed on
2026-10-24 15:35. Git HEAD was previously recorded as: `- **Version:** `0.69.0` (`plugin.yaml` and `dashboard/manifest.json`), git HEAD `f000bf0` —
*“Pulse: list candidates, decide in code, journal them (#337)”*. **22 commits on from the
`0.64.0` / `eaa14be` recorded at the 2026-10-24 audit** (upstream `#324`–`#338`: Pulse
cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway
@@ -41,8 +49,9 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste
`conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`.
## Current state (verified 2026-10-25 00:2x–00:5x AWST)
- **Dashboard gateway:** listening on `0.0.0.0:9119`; process started **2026-10-23 18:54:43** and
**not restarted since** — so it is still the process that picked up the memory-timeout patch, and
- **Dashboard gateway (re-verified 2026-10-26):** listening on `0.0.0.0:9119`; process started **2026-10-25
18:42:33** — **restarted since the 2026-10-24 entry, so the upstream commits are now live** (the recorded
*"not restarted since / not live"* note is superseded, not wrong at the time) — so it is still the process that picked up the memory-timeout patch, and
the `0.64.0` → `0.69.0` upstream commits are **not** live.
- **Device:** one registered — **iPhone** (`id 70aa63f3…`, `kid c24e8215…`), app version `2.0.0`,
features `chat-cards`, `info-cards`, `live-cards`, `maps-navigate`, `media-cards`, `show-cards`;
+32 -22
View File
@@ -3,9 +3,9 @@ title: Current Hermes Profile
type: system
status: active
created: 2026-07-22
updated: 2026-10-21
verified_on: 2026-10-21
review_after: 2026-10-28
updated: 2026-10-26
verified_on: 2026-10-26
review_after: 2026-11-09
confidence: high
tags: [system, hermes, profile]
sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026-10-21]
@@ -18,27 +18,37 @@ sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026
- **Location:** implicit — the top-level `~/.hermes/config.yaml`. There is **no** `~/.hermes/profiles/default/` directory (verified 2026-10-02); `~/.hermes/profiles/` holds only named sub-profiles.
- **Other profiles present:** `ops` only (`~/.hermes/profiles/ops/`) — the Tusk elephant persona, with its own SOUL.md, skills, plugins and memories (last modified 2026-09-30). Other profiles do **not** affect this session; see the profile-isolation note below.
## Key Settings (model routing re-verified against live config 2026-10-21; remaining items as recorded 2026-10-02)
## Key Settings (model routing + auxiliary slots re-verified against live config 2026-10-26; remaining items as recorded 2026-10-02)
### ⚠ Model default moved OFF OmniRoute (2026-10-20)
- **Provider:** **`nous`** — `model.base_url: https://inference-api.nousresearch.com/v1`, `api_mode: chat_completions`.
- **Model default:** **`meituan/longcat-2.5-preview:free`**
- **When:** `config.yaml` last written **2026-10-20 22:01:38 AWST**. The move was **not** in place at the
2026-10-09 19:42 snapshot and **was** in place by 2026-10-20 11:00. Previous values (2026-10-02 record):
provider `custom` → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`, provider id `custom:omni-101`),
default `smart-route`.
- **The `model` block now carries only** `base_url`, `default`, `provider`, `api_mode`, `persist_switch_by_default`.
**The 6-provider fallback chain recorded at the 2026-10-02 snapshot is gone** — there is no `fallbacks` key under
`model`, so an unreachable `nous` is no longer covered by the old `custom` / `or-inclusionai_ling-2_6-1t` →
`custom:litellm` → `litellm:4000` chain.
- **⚠ Cron pins did NOT move with it.** **20 of the 28 scheduled jobs still pin OmniRoute explicitly**
(`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). **The 8 jobs with no pin are
the ones whose upstream changed** — they now call `nous` direct instead of the router. The fleet therefore
runs **two upstream paths for the first time**. → [[systems/scheduled-tasks]]
- **Routing posture:** no fallback chain under `model`; per-job pins are the only routing control that remains.
### ⚠ Model default moved BACK ONTO OmniRoute (reverted 2026-10-24 19:06)
- **Provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`),
`api_mode: chat_completions`.
- **Model default:** **`smart-route`** (paid alias — not a free combo).
- **When:** a session switch note at **2026-10-24 19:06:10 AWST** reads *"model was just switched from
meituan/longcat-2.5-preview:free to smart-route via omni-101."* The 2026-10-21 config snapshot still read
`nous` / `meituan/longcat-2.5-preview:free`, so the revert landed between **2026-10-20 22:01** and
**2026-10-24 19:06**; `config.yaml` was last written **2026-10-25 19:13**. ⚠ The 2026-10-22, 10-24 and
10-25 audits all recorded the 10-20 move as current — this section corrects them. The revert coincides with
the **invalid Nous Portal refresh token** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
- **The `model` block carries** `base_url`, `default`, `provider`, `api_mode`, `persist_switch_by_default` —
still **no `fallbacks` key under `model`**.
- **⚠ A top-level six-provider `fallback_providers` chain is live again** (re-verified 2026-10-26):
`nous` / `meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` →
`custom:guacano` / `deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` →
`kilocode` / `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6-1t`.
The 2026-10-21 record that *no* fallback chain exists is **stale**. ⚠ The first entry is the **dead `nous`
credential** — a fallover would try it first.
- **⚠ Cron pins did NOT need to move.** **18 of the 28 scheduled jobs pin OmniRoute explicitly**
(`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). The 8 unpinned jobs follow
the global default, which is OmniRoute again — so the fleet is back to a **single upstream path**, not the
two-path split recorded on 2026-10-21. → [[systems/scheduled-tasks]]
- **Routing posture:** top-level `fallback_providers` chain + per-job pins are the routing controls;
`model.fallbacks` does not exist.
- ⚠ The `custom:omni` provider key is **no longer configured** (only 2 jobs still name it) — see
[[systems/model-providers]].
### Other settings (recorded 2026-10-02 — not re-verified 2026-10-21)
- **Auxiliary vision model:** `stepfun/step-3.7-flash:free` on `nous`
### Other settings (recorded 2026-10-02 — auxiliary line re-verified 2026-10-26)
- **Auxiliary slots (all 8 — `approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`, `title_generation`, `vision`):** `no-think/freeapi/auto` on `custom:omni-101`; per-slot fallbacks `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`; `free_only: true`; compression timeout 90 s, vision timeout 180 s. Re-placed 2026-10-25 19:13 — previously vision was `stepfun/step-3.7-flash:free` on `nous`. → [[systems/model-providers]]
- **Switch persistence:** `persist_switch_by_default: false`
- **Timezone:** `Australia/Perth`
- **Approvals mode:** `off` (`destructive_slash_confirm: false`, `mcp_reload_confirm: false`)
+19 -2
View File
@@ -3,8 +3,8 @@ title: Hermes Agent
type: system
status: active
created: 2026-07-22
updated: 2026-10-22
verified_on: 2026-10-22
updated: 2026-10-26
verified_on: 2026-10-26
confidence: high
tags: [system, hermes, agent, llm]
sources: [live-hermes-version, live-config, live-skill-inventory]
@@ -169,3 +169,20 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
- **Mattermost platform unreachable** since 2026-10-10; the gateway keeps retrying on a 5-minute
cadence - [[systems/messaging-integrations]].
- **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]].
## 2026-10-26 00:1x AWST — gateway restart record and the dead Nous credential
- **Three gateway starts on 2026-10-25**: `00:32:20`, `18:30:04`, `19:27:13` AWST (after `2026-10-24 12:24:39`).
The `18:29:22` shutdown carried `signal=SIGTERM under_systemd=yes`, `loadavg_1m=3.79` — a planned stop, not a
crash; `19:26:16` logged *"Stopping gateway for restart..."* (the plugin self-update path). Current gateway
pid **2476414**, up since **19:26:33** (≈4 h 45 m at audit time).
- **Consequences of the 19:27 start:** the Conduit dashboard gateway restarted at 18:42:33 and so picked up the
plugin's upstream commits → [[systems/conduit]]; the Mattermost reconnect attempt counter reset to 54
→ [[systems/messaging-integrations]]; the `invalid_grant` spam stopped.
- ⚠ **The Nous Portal credential is dead** — `Nous Portal runtime credentials unavailable: Invalid refresh token`.
Gateway-log counts: **1 line 2026-10-22, 3 on 10-24, 20 on 10-25** (last 14:53:04,
`agent.context_compressor: Failed to generate context summary`), **0 since the 19:27:13 start**. Context
compression was unavailable for most of 10-25 and contributed to the digest runaway.
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
- **Restart hygiene:** a gateway restart resets in-process counters (Mattermost retries, cron admission
bookkeeping). Never read a counter as an outage duration.
+14 -2
View File
@@ -3,8 +3,8 @@ title: Messaging Integrations
type: system
status: active
created: 2026-07-22
updated: 2026-10-19
verified_on: 2026-10-19
updated: 2026-10-26
verified_on: 2026-10-26
confidence: high
tags: [messaging, telegram, discord, email, integrations]
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
@@ -113,3 +113,15 @@ This pass replaced the 10-11 observation's "cause not determined" with a definit
**Open item:** start CT255, or retire the platform block. The 5-minute reconnect loop continues until
one of those happens — restarting the adapter while the host is stopped achieves nothing.
## 2026-10-26 00:1x AWST — Mattermost still down; attempt counter reset by the 10-25 restart
- **Cause unchanged: CT255 (`mattermost`) is `stopped`.** `tailscale status` shows the peer **offline, last seen
36 d ago**, `tx 312 / rx 0` — `rx` has read 0 in every audit since 2026-10-11, i.e. no packet has come back.
- **Reconnect loop still running:** **268** `mattermost connect timed out` lines on 2026-10-25 vs **264** on
2026-10-24 (≈1 per 5 min, as designed); **2** so far on 2026-10-26.
- **The attempt counter reset to 54** with the 19:27:13 gateway restart (`Reconnecting mattermost (attempt 54)`
at 2026-10-26 00:07:06). The previous reset was the 2026-10-17 05:10 restart. ⚠ The counter measures **this
process's** attempts, not the outage's length — do not quote it as downtime.
- **Impact unchanged:** messaging path only; no cron job delivers via Mattermost.
- **Not attempted:** `pct start 255` — starting a service on the PVE host is Anthony's call.
+38 -9
View File
@@ -3,8 +3,8 @@ title: Model Providers
type: system
status: active
created: 2026-07-22
updated: 2026-10-23
verified_on: 2026-10-23
updated: 2026-10-26
verified_on: 2026-10-26
confidence: high
tags: [model, provider, inference, routing]
sources: [live-probe, config, web-search]
@@ -20,27 +20,56 @@ Verified from live `~/.hermes/config.yaml` → `custom_providers`, 2026-10-01.
| Provider | URL | Models | Role |
|---|---|---|---|
| `custom:omni-101` | `http://100.78.63.20:20128` | 5,384 | **Primary.** OmniRoute on the estate (`omni-101`). 15 of 30 cron jobs reference it |
| `custom:omni-101` | `http://100.78.63.20:20128` | 5,384 | **Primary.** OmniRoute on the estate (`omni-101`). 18 of 28 cron jobs reference it (2026-10-26) |
| `custom:Omniroute:20128` | `http://omniroute:20128/v1` | 5,398 | Secondary OmniRoute host — legacy naming, kept alongside `omni-101` |
| `custom:litellm` | `http://litellm:4000/v1` | 1,728 | Fallback inference (slower path, 6–26s/call) |
| `custom:command-code` | `https://api.commandcode.ai/provider/v1` | 71 | DeepSeek Flash primary via the `deepseek-flash-mix` combo (added 2026-08-19); also the source of the `MODEL_NOT_IN_PLAN` 403 on `gpt-5.6-terra` |
| `custom:guacano` | `http://ollama:11000/v1` | 46 | Guanaco LLM proxy/router on CT205 (Ollama stack). See [[systems/guanaco]] |
| `custom:omni` | *(removed)* | — | **No longer configured.** It once served auxiliary roles (vision, skills, approval, mcp, web_extract, compression). Two cron jobs still name it as their provider key — see [[systems/scheduled-tasks]] |
**Built-in (non-`custom_providers`) routes still in use as fallbacks:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`. `charm-hyper` (Charmbracelet, `hyper.charm.land/v1`, free tier 100 credits/mo) is no longer in the live stack.
**Built-in (non-`custom_providers`) routes still in use as fallbacks:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`. `charm-hyper` (Charmbracelet, `hyper.charm.land/v1`, free tier 100 credits/mo) is no longer in the live stack. **The live `fallback_providers` chain (2026-10-26) is `nous` / `meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` → `custom:guacano` / `deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` → `kilocode` / `poolside/laguna-m.1:free` → `custom` / `or-inclusionai_ling-2_6-1t`.**
**Default provider:** **`nous`** → `https://inference-api.nousresearch.com/v1`. **Default model:** **`meituan/longcat-2.5-preview:free`**.
The default moved OFF OmniRoute on 2026-10-20 — see below. The `custom` provider key is no longer the default.
**Default provider:** **`custom:omni-101`** → `http://100.78.63.20:20128`. **Default model:** **`smart-route`**.
⚠ **The default moved BACK to OmniRoute on 2026-10-24 19:06:10 AWST** (session switch note: *"model was just
switched from meituan/longcat-2.5-preview:free to smart-route via omni-101"*). The `nous` direct window ran
2026-10-20 22:01 → 2026-10-24 19:06 and the 2026-10-22/10-24/10-25 audit entries recorded it as current —
**stale**. `config.yaml` last written 2026-10-25 19:13.
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
Provider names that appear in job configs but not in `custom_providers` do **not** necessarily fail: the default `custom` block resolves to the same base URL, so a stale provider key can go unnoticed. Prefer routing aliases (`smart-route`) over raw upstream model names in pins.
## Default Model
- **`meituan/longcat-2.5-preview:free`** on **`nous`** (`https://inference-api.nousresearch.com/v1`, `api_mode: chat_completions`) — the live default since 2026-10-20. Verified against live `~/.hermes/config.yaml` 2026-10-23.
- **No fallback chain under `model`.** The 6-provider fallback recorded at the 2026-10-02 snapshot is gone; `model` now carries only `base_url`, `default`, `provider`, `api_mode`, `persist_switch_by_default`. An unreachable `nous` has no automatic failover — per-job pins are the only routing control that remains.
- **`smart-route`** on **`custom:omni-101`** (`http://100.78.63.20:20128`, `api_mode: chat_completions`) — the
live default since **2026-10-24 19:06:10 AWST**, re-verified against live `~/.hermes/config.yaml`
**2026-10-26 00:1x**.
- **⚠ A six-provider top-level `fallback_providers` chain is live again** (re-verified 2026-10-26): `nous` /
`meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` → `custom:guacano` /
`deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` → `kilocode` /
`poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6-1t`. The
2026-10-21 claim that **no** fallback chain exists is **stale**; the `model` block itself still carries only
`base_url`, `default`, `provider`, `api_mode`, `persist_switch_by_default`.
- ⚠ **The first fallback entry is the dead `nous` credential** (`Invalid refresh token`, since 2026-10-22)
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
- Anthony has explicitly **rejected** `free-lkgp` as a default; stable defaults are `smart-route` / `deepseek-paid-smart` (paid, not free)
- `resilient-free` was the default after the 2026-08-14 OmniRoute factory reset, but that combo is now in the broken set
- Previous default `auto/best-free` was misconfigured — had no combo definition and routed to a paid `gpt-5.6-sol`
- **20 of 28 scheduled jobs still pin OmniRoute explicitly** (`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). The fleet now runs two upstream paths. → [[systems/scheduled-tasks]]
- **18 of 28 scheduled jobs pin OmniRoute explicitly** (`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). The remaining 8 follow the global default, which is **also** OmniRoute — so the fleet is back to a **single upstream path**. → [[systems/scheduled-tasks]]
## Auxiliary Role Routing (re-verified live 2026-10-26)
**All eight `auxiliary:` slots** — `approval`, `compression`, `curator`, `mcp`, `profile_describer`,
`skills_hub`, `title_generation`, `vision` — are **`no-think/freeapi/auto` on `custom:omni-101`**, each with
the chain `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` and `free_only: true`.
**Compression timeout 90 s; vision timeout 180 s.**
- Re-placed **2026-10-25 19:13 AWST**, replacing the `nous` / `stepfun/step-3.7-flash:free` and `custom:omni`
aux paths recorded on 2026-10-02 and 2026-10-25 (17:50).
- ⚠ The move was made **because the Nous Portal credential is dead** (`Invalid refresh token`), so it **masks
rather than fixes** that fault → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
- Probe results from that session (17:50–19:13, not re-run 2026-10-26): `freeapi/auto` healthy for text and
vision; `kg/kilo-auto/free` returns **empty content on image input**; `stepfun/step-3.7-flash:free` → 401
*no active credentials*; freeapi pinned models in cooldown (429). ⚠ A **vision fallback returning empty
content** is a silent failure mode — check `freeapi/auto` itself, not the fallback order, if image calls
go blank.
## Retired / Unavailable Models (2026-09-25 → 2026-10-01)
Pins that fail at request time, with the exact upstream error. Each is an open cron-repin in [[systems/scheduled-tasks]].
+6 -2
View File
@@ -47,7 +47,11 @@ sources: [raw/configs/hermes-config-sanitized.txt]
burst, not afterwards.
## Purpose
Secondary provider/router for Hermes auxiliary roles and some agent routing.
Provider/router for Hermes — ⚠ **updated 2026-10-26: the `omniroute-1` node is the DEFAULT chat path again.**
Hermes's default provider is `custom:omni-101` → `http://100.78.63.20:20128` with model `smart-route`
(reverted 2026-10-24 19:06; the 2026-10-20 → 10-24 window pointed the default at `nous`).
The CT600 provider key `custom:omni` (`http://omniroute:20128/v1`) remains auxiliary-only.
→ [[systems/current-profile]]
## Current Configuration
- **Hermes provider name:** `custom:omni` (maps to provider `omniroute:20128`)
@@ -56,7 +60,7 @@ Secondary provider/router for Hermes auxiliary roles and some agent routing.
- **Runs on:** Proxmox CT600 (`omniroute`, Tailscale `100.88.81.19`, onboot=1) — verified running 2026-10-11
- **DB path:** `/opt/omniroute/storage.sqlite` (was `/var/lib/omniroute/storage.sqlite`)
- **Backup:** pre-migration config saved at `~/.hermes/config.yaml.bak.omniroute-removal`
- **Miss:** Hermes default provider path does not use OmniRoute for main chat; used for auxiliary roles instead
- **Correction (2026-10-26):** the earlier *"Hermes default provider path does not use OmniRoute for main chat"* note is **stale** — the default is OmniRoute again (`custom:omni-101`, model `smart-route`). The **CT600** key `custom:omni` is still auxiliary-only.
- **Auxiliary roles in Hermes config:** `skills_hub`, `approval`, `mcp`
- **Key prefix visible in config:** `sk-5c67fbdc0...` — full value not stored in wiki
+4 -3
View File
@@ -3,8 +3,8 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-25
verified_on: 2026-10-25
updated: 2026-10-26
verified_on: 2026-10-26
review_after: 2026-10-31
confidence: high
tags: [cron, scheduled, automation]
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks
## Count
- **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page.
- **Current as of 2026-10-26 00:1x AWST (live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`).
- **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page. *(Superseded by the 2026-10-26 entry above.)*
- **Current as of 2026-10-24 00:2x AWST *(superseded by the 2026-10-25 entry above)* (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]]
- **Current as of 2026-10-23 00:1x AWST *(superseded by the 2026-10-24 entry above)* (`jobs.json.updated_at 2026-10-23T00:07:25+08:00`) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. **A DB recount by `started_at` and by `claimed_at` puts 2026-10-21 at 168 / 1**, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Five incidents closed in-window and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (`0e519874d280`, closed 08:01:40), `06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so **unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`: `1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, `c0a9fa_764558b5ae83`, `0e753e_e7f8194a7d06`). **Non-clean `last_status` falls 9 → 5:** `1b23c042e17d` (`error`, streak 2 — **next slot Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` (`delivery_failed` — Mon 10-26 03:00), `066bb5191280` (`delivery_failed` — 11-01). All four cleared rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its first slot of 10-22. **Pins:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (`provider`/`model` null → global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). ⚠ **Stocktake anomaly:** `1a8e90ea2271` ran **03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, finished `completed` with delivery `suppressed` — and **committed nothing**. It left its regenerated `systems/skills-index.md` uncommitted (mtime 10-22 03:00, with a `## .Archive (41)` section reading **347**) and its diagnosed fixes unapplied → [[incidents/2026-08-15-skills-index-drift]]. **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop` (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models `mimo-v2.5`, `or-inclusionai_ling-2_6-1t`, `minimax/minimax-m3.1-flash`), 2 `acp` (0 messages), 1 `oneshot` memory-capture probe. → [[systems/skills-index]], [[incidents/2026-08-15-skills-index-drift]]
- **Current as of 2026-10-22 00:1x AWST *(superseded by the 2026-10-23 entry above)* (`jobs.json.updated_at 2026-10-22T00:11:31+08:00`) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen.** Window 2026-10-21: **171 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`), which died at 07:40:48 after **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string, which appears in no 2026-10-21 row → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` stays at 9** (7 `error` / 2 `delivery_failed`) but the composition moved: **`08eb86f02d37` cleared** (its 08:30 slot ran clean; incident `08eb86_472c7b609c80` **resolved 2026-10-21 08:37:21**) while **`dca8482e4f76` re-entered** (streak 1). Survivors, with next run: `dca8482e4f76` (1, **10-22 07:00**), `0e519874d280` (2, **10-22 08:00**), `06a0a1c8704f` (1, **10-22 10:00**), `c8d52904e1d6` (1, **10-22 18:00**), `1b23c042e17d` (2, Fri 10-23 08:00), `c0a9fa314542` (1, Mon 10-26 09:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 11-01 02:00). **Incident ledger: 1 opened / 1 closed, unclosed stays 9** (8 `alerted` / 1 `detected`). **Pins:** 17 of the 28 jobs pin `custom:omni-101` (10 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`), **7 carry no pin at all** (`provider`/`model` null → global default), 1 pins `custom:omni` / `smart-route`. **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. **12 sessions in the window, all `cron` — no interactive sessions.** ⚠ **Infrastructure changed in-window: Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]], [[infrastructure/backups]].
+2 -2
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system
status: active
created: 2026-07-22
updated: 2026-10-25
verified_on: 2026-10-25
updated: 2026-10-26
verified_on: 2026-10-26
confidence: high
tags: [skills, hermes]
sources: []