wiki audit 2026-10-02: 2 new cron failures, cliprox/terra connection auth incident, delivery-failure semantics corrected, profile rewrite, broken wikilinks repaired

- new: incidents/2026-10-01-cliproxy-connection-auth-expired.md (HTTP 401, credential fault, cannot self-recover)
- systems/scheduled-tasks.md: 30 jobs / 27 enabled; failing 3 -> 6; streaks advanced; wiki-tree drift recorded
- systems/messaging-integrations.md: two distinct thread-failure variants (lost vs thread-less fallback)
- systems/current-profile.md: full rewrite from live config (no profiles/default/, smart-route, OmniRoute)
- current-state.md / index.md / log.md: 2026-10-02 entry
- repair: copied 5 vault-only pages referenced by index.md (ignis, nextcloud, 3 incidents) so wikilinks resolve
- include pending stocktake output: skills-index.md 2026-09-27, raw/stocktake/freshness-2026-09-2{6,7}.txt
This commit is contained in:
Tony0410
2026-10-02 00:17:15 +08:00
parent a32475bd54
commit 0c4ed3350b
15 changed files with 684 additions and 96 deletions
@@ -0,0 +1,43 @@
---
title: "Skills Index Drift — 4x Undercount"
type: incident
status: resolved
created: 2026-08-15
updated: 2026-08-15
resolved_on: 2026-08-15
confidence: high
tags: [skills, wiki, drift, auto-regeneration]
sources: [session-investigation, agent-estate-wiki]
---
# Skills Index Drift — 4x Undercount
## Summary
The `systems/skills-index.md` page in both wikis claimed **66 skills** when **330 were actually installed** — a 4x undercount. The page had been hand-written once on 2026-07-22/25 and never regenerated as the estate grew.
## Root Cause
Static documentation for a mutable inventory. The skills tree at `~/.hermes/skills/` grew from ~66 to 330 skills over three weeks, but the wiki page was never updated because there was no automation to keep it in sync.
## Impact
- Wiki page was 4x out of date
- Any inventory-based decisions (skill loading, capacity planning) would have been based on wrong numbers
- No functional impact — skills still loaded correctly via Tool Search
## Resolution
1. Regenerated `systems/skills-index.md` from the live skill tree (all 330 skills, 26 categories)
2. Created `regenerate-skills-index.sh` script for automated regeneration
3. Set up cron job `1a8e90ea2271` to run daily at 03:00 AWST
4. Script auto-commits and pushes to Gitea (`Anthony/agent-estate-wiki`)
5. Quartz site at `http://100.118.5.51:9120/systems/skills-index` rebuilt and redeployed
## Lesson
Static wiki pages that reflect **mutable inventory** (skills, services, cron jobs) **MUST be auto-regenerated on a schedule**, not hand-maintained. The moment the estate grows, a hand-written page becomes stale.
## Prevention
- Daily cron job regenerates skills-index.md from live tree
- Page includes "auto-regenerated, do not hand-edit" warning
- Same script pattern is reusable for other inventory pages (cron jobs, services, agents)
## Related
- [[systems/skills-index]] — The updated page
- [[systems/scheduled-tasks]] — The new cron job
@@ -0,0 +1,70 @@
---
title: "Mnemosyne Capture Pipeline Broken (Aug 18)"
type: incident
status: active
created: 2026-08-18
updated: 2026-08-18
verified_on: 2026-08-18
confidence: high
tags: [memory, mnemosyne, hermes, incident, capture]
sources: [mnemosyne-db, health-monitor, run_agent.py]
---
# Mnemosyne Capture Pipeline Broken — 2026-08-18
## Impact
Working memory capture (`sync_turn`) stopped functioning after **2026-08-16 23:15:39**. As of verification on 2026-08-18, the pipeline had been broken for ~48 hours. Multiple active conversation turns during this period were **not captured** into working memory. The plugin loads, initializes, and tools work — but the post-turn sync path (`_sync_external_memory_for_turn` → `sync_all` → `sync_turn`) is not being triggered.
**What remains healthy:**
- DB: 27M, 326 WM / 693 EP / 352 facts
- FTS indexes: all aligned (OK)
- Embedding coverage: 100% (WM), 201.6% total
- Last consolidation: within threshold
- Plugin: loads and registers correctly; config types are proper YAML
## Root Cause
One of the silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()` (line ~4333–4343) is blocking every turn. Three candidates:
1. `if interrupted: return` — an interrupted flag may be stuck
2. `if not (self._memory_manager and final_response and original_user_message): return` — one of these is falsy
3. `if not (user_text and response_text): return` — text extraction failing
Unlike the Aug 1 incident (stringified YAML `sync_roles`), the config is now clean — native YAML types verified. The failure is in the runtime guard logic, not configuration.
## Diagnostic Steps
Add temporary diagnostic logging before each guard in `run_agent.py:_sync_external_memory_for_turn()` (around line 4333):
```python
if interrupted:
logger.warning("MEMDEBUG: skip - interrupted=True")
if not (self._memory_manager and final_response and original_user_message):
logger.warning("MEMDEBUG: skip - manager=%s final=%s orig=%s",
bool(self._memory_manager), bool(final_response), bool(original_user_message))
if not (user_text and response_text):
logger.warning("MEMDEBUG: skip - user_text=%s response_text=%s",
bool(user_text), bool(response_text))
```
Then restart gateway (`hermes gateway restart` from external shell) and `grep MEMDEBUG ~/.hermes/logs/agent.log` after one conversation turn to identify which guard fires.
## Verification
```bash
python3 -c "
import sqlite3
db = '/home/hermes/.hermes/mnemosyne/data/mnemosyne.db'
conn = sqlite3.connect(db)
cur = conn.cursor()
cur.execute('SELECT MAX(created_at) FROM working_memory')
print('Last WM capture:', cur.fetchone()[0]) # Should be recent
"
```
## Prevention
- Never use `hermes config set memory.mnemosyne.<field>` for list/dict/null values
- After any config change, verify YAML types with `yaml.safe_load` + `type()` on each field
- After any gateway restart, verify capture within 2 hours via health monitor
## Related
- [[systems/mnemosyne]]
- [[incidents/2026-08-01-mnemosyne-capture-failure]] — Previous capture failure (YAML stringification)
- [[incidents/2026-07-29-mnemosyne-relay]] — Earlier relay scope corruption
+77
View File
@@ -0,0 +1,77 @@
---
title: "Gmail Read-State Incident (2026-08-29)"
type: incident
status: active
created: 2026-08-29
updated: 2026-08-29
verified_on: 2026-08-29
confidence: high
tags: [email, gmail, cron, himalaya, incident]
sources: [session:default/20260829_204103_9e1087]
---
# Gmail Read-State Incident
**Date discovered:** 2026-08-29 20:41 AWST
**Status:** Active — mitigation in place, root cause identified but not fully fixed
## Summary
Anthony's Gmail inbox messages were being automatically marked as read without his consent. The issue was traced to Hermes cron jobs using Himalaya's `message read` command, which applies the "seen" flag even with `--preview`.
## Timeline
- **2026-08-29 20:41** — Anthony reports: "What is marking my Gmail email as read it is not meant to be"
- **Investigation** — Loaded skills: `gmail-imap-email-monitor`, `hermes-cron-operations`, `email-inbox-triage`
- **Root cause identified** — `fetch_newsletters.py` (line 49) runs:
```bash
himalaya message read --preview --folder INBOX <message-id>
```
Himalaya's `message read` command **automatically applies the "seen" flag** unless `--preview` is used for preview-only access. However, even `--preview` may still mark messages read in some configurations.
- **Other scripts checked** — `gmail_readonly_imap.py` uses `SELECT readonly` and `BODY.PEEK[]` (safe, does not mark read)
- **Mitigation applied:**
- Stopped `goimapnotify.service` (IMAP watcher)
- Paused "Important-mail monitor" cron job (if present)
- Newsletter digest already patched to use `--preview`
## Root Cause
The `fetch_newsletters.py` script at `/home/hermes/.hermes/scripts/fetch_newsletters.py` line 49:
```python
r2 = subprocess.run(
["himalaya", "message", "read", "--preview", "--folder", "INBOX", m["id"]],
capture_output=True, text=True, timeout=60
)
```
Despite using `--preview`, Himalaya may still mark messages as read depending on configuration. The proper fix is to use readonly IMAP with `BODY.PEEK[]` instead of the `himalaya message read` command.
## Impact
- Newsletter emails processed by the Tech & AI Newsletter Digest cron job were marked as read
- Potential for other Gmail automation scripts to have similar behavior
- User trust issue — unexpected state changes in personal email
## Resolution Status
**Mitigation:** Partially complete
- `goimapnotify` stopped
- Important-mail monitor paused (if it existed)
**Fix required:**
- Rewrite `fetch_newsletters.py` to use readonly IMAP (`BODY.PEEK[]`) instead of `himalaya message read`
- Audit all Gmail-related scripts for similar patterns
- Verify `gmail_action_digest_v3.py` and other email scripts use readonly access
## Related Pages
- [[systems/scheduled-tasks]] — Tech & AI Newsletter Digest cron job
- [[systems/messaging-integrations]] — Email integration patterns
- [[runbooks/job-radar]] — Email monitoring patterns (for reference)
## Prevention
- All Gmail IMAP scripts should use `SELECT readonly` and `BODY.PEEK[]` patterns
- Add a pre-commit check or linting rule for `himalaya message read` usage
- Document readonly email access pattern in a runbook
@@ -0,0 +1,74 @@
---
title: OmniRoute cliprox/terra Connection Auth Expiry
type: incident
status: open
created: 2026-10-02
updated: 2026-10-02
verified_on: 2026-10-02
confidence: high
tags: [cron, omniroute, provider, auth, incident]
sources: [live-jobs-json, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040]
---
# OmniRoute `cliprox/terra` Connection Auth Expiry
**Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); still open
**Status:** Open — not self-recovering
**Impact:** 1 job failed outright; 2 more enabled jobs share the same model pin and are expected to fail on their next runs
## Symptoms
Job `06a0a1c8704f` (**AI Newsletter Curation for Patrick**, Thu 10:00) aborted with:
```
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
— please reconnect in the dashboard
```
Provider `custom:Omniroute:20128`, model `cliprox/terra`. The run's failure notice was
delivered to forum topic `346`, but no newsletter digest was produced.
## Root Cause
A **connection credential at the provider**, not a missing or plan-gated model and not a
routing error. The error names a single connection (`All 1 connection(s)`) whose
authorisation lapsed, so the model pin itself is still valid — the *credential behind it*
is not.
This matters because it is the one failure class from the 2026-10-01 sweep that **cannot
self-recover**. A retired or plan-gated model stays broken until repinned; an expired
connection stays broken until a human re-authenticates it in the dashboard.
## Blast Radius
Three enabled jobs run `cliprox/terra`:
| Job | Name | Schedule | Delivery |
|---|---|---|---|
| `06a0a1c8704f` | AI Newsletter Curation for Patrick | `0 10 * * 4` | `telegram:-1004321904721:346` |
| `1b23c042e17d` | Enhanced Weekend Adventure Planner | `0 8 * * 5` | `telegram:-1004321904721:102` |
| `79f9bb41ecf5` | Weekly Review | `0 18 * * 0` | `origin` |
Only `06a0a1c8704f` had run by 2026-10-02; the other two are exposed on their next
scheduled runs.
## Fix
Re-authenticate the OmniRoute connection for `cliprox/terra`. Do **not** repin these jobs
to another raw model as a first move — that treats a credential fault as a routing fault
and will produce a second, different outage. If the connection cannot be restored, repin
all three jobs together to a routing alias (`smart-route`).
## Distinguish From
- **Plan-gated model** (403 `MODEL_NOT_IN_PLAN`, job `dca8482e4f76`) — the upstream rejects
the model by plan; no credential involved.
- **Retired model** (HTTP 410, job `1a8e90ea2271`) — the model no longer exists upstream.
- **Catalog-missing model** (HTTP 400, jobs `fad5a86979d4` / `0e753e9720fd`) — the model
name is absent from the provider's live catalog.
## Related
- [[systems/scheduled-tasks]]
- [[systems/model-providers]]
- [[systems/omniroute]]