wiki audit 2026-10-02: 2 new cron failures, cliprox/terra connection auth incident, delivery-failure semantics corrected, profile rewrite, broken wikilinks repaired
- new: incidents/2026-10-01-cliproxy-connection-auth-expired.md (HTTP 401, credential fault, cannot self-recover)
- systems/scheduled-tasks.md: 30 jobs / 27 enabled; failing 3 -> 6; streaks advanced; wiki-tree drift recorded
- systems/messaging-integrations.md: two distinct thread-failure variants (lost vs thread-less fallback)
- systems/current-profile.md: full rewrite from live config (no profiles/default/, smart-route, OmniRoute)
- current-state.md / index.md / log.md: 2026-10-02 entry
- repair: copied 5 vault-only pages referenced by index.md (ignis, nextcloud, 3 incidents) so wikilinks resolve
- include pending stocktake output: skills-index.md 2026-09-27, raw/stocktake/freshness-2026-09-2{6,7}.txt
This commit is contained in:
@@ -0,0 +1,43 @@
|
||||
---
|
||||
title: "Skills Index Drift — 4x Undercount"
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-08-15
|
||||
updated: 2026-08-15
|
||||
resolved_on: 2026-08-15
|
||||
confidence: high
|
||||
tags: [skills, wiki, drift, auto-regeneration]
|
||||
sources: [session-investigation, agent-estate-wiki]
|
||||
---
|
||||
|
||||
# Skills Index Drift — 4x Undercount
|
||||
|
||||
## Summary
|
||||
The `systems/skills-index.md` page in both wikis claimed **66 skills** when **330 were actually installed** — a 4x undercount. The page had been hand-written once on 2026-07-22/25 and never regenerated as the estate grew.
|
||||
|
||||
## Root Cause
|
||||
Static documentation for a mutable inventory. The skills tree at `~/.hermes/skills/` grew from ~66 to 330 skills over three weeks, but the wiki page was never updated because there was no automation to keep it in sync.
|
||||
|
||||
## Impact
|
||||
- Wiki page was 4x out of date
|
||||
- Any inventory-based decisions (skill loading, capacity planning) would have been based on wrong numbers
|
||||
- No functional impact — skills still loaded correctly via Tool Search
|
||||
|
||||
## Resolution
|
||||
1. Regenerated `systems/skills-index.md` from the live skill tree (all 330 skills, 26 categories)
|
||||
2. Created `regenerate-skills-index.sh` script for automated regeneration
|
||||
3. Set up cron job `1a8e90ea2271` to run daily at 03:00 AWST
|
||||
4. Script auto-commits and pushes to Gitea (`Anthony/agent-estate-wiki`)
|
||||
5. Quartz site at `http://100.118.5.51:9120/systems/skills-index` rebuilt and redeployed
|
||||
|
||||
## Lesson
|
||||
Static wiki pages that reflect **mutable inventory** (skills, services, cron jobs) **MUST be auto-regenerated on a schedule**, not hand-maintained. The moment the estate grows, a hand-written page becomes stale.
|
||||
|
||||
## Prevention
|
||||
- Daily cron job regenerates skills-index.md from live tree
|
||||
- Page includes "auto-regenerated, do not hand-edit" warning
|
||||
- Same script pattern is reusable for other inventory pages (cron jobs, services, agents)
|
||||
|
||||
## Related
|
||||
- [[systems/skills-index]] — The updated page
|
||||
- [[systems/scheduled-tasks]] — The new cron job
|
||||
@@ -0,0 +1,70 @@
|
||||
---
|
||||
title: "Mnemosyne Capture Pipeline Broken (Aug 18)"
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-08-18
|
||||
updated: 2026-08-18
|
||||
verified_on: 2026-08-18
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, hermes, incident, capture]
|
||||
sources: [mnemosyne-db, health-monitor, run_agent.py]
|
||||
---
|
||||
|
||||
# Mnemosyne Capture Pipeline Broken — 2026-08-18
|
||||
|
||||
## Impact
|
||||
Working memory capture (`sync_turn`) stopped functioning after **2026-08-16 23:15:39**. As of verification on 2026-08-18, the pipeline had been broken for ~48 hours. Multiple active conversation turns during this period were **not captured** into working memory. The plugin loads, initializes, and tools work — but the post-turn sync path (`_sync_external_memory_for_turn` → `sync_all` → `sync_turn`) is not being triggered.
|
||||
|
||||
**What remains healthy:**
|
||||
- DB: 27M, 326 WM / 693 EP / 352 facts
|
||||
- FTS indexes: all aligned (OK)
|
||||
- Embedding coverage: 100% (WM), 201.6% total
|
||||
- Last consolidation: within threshold
|
||||
- Plugin: loads and registers correctly; config types are proper YAML
|
||||
|
||||
## Root Cause
|
||||
One of the silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()` (line ~4333–4343) is blocking every turn. Three candidates:
|
||||
|
||||
1. `if interrupted: return` — an interrupted flag may be stuck
|
||||
2. `if not (self._memory_manager and final_response and original_user_message): return` — one of these is falsy
|
||||
3. `if not (user_text and response_text): return` — text extraction failing
|
||||
|
||||
Unlike the Aug 1 incident (stringified YAML `sync_roles`), the config is now clean — native YAML types verified. The failure is in the runtime guard logic, not configuration.
|
||||
|
||||
## Diagnostic Steps
|
||||
Add temporary diagnostic logging before each guard in `run_agent.py:_sync_external_memory_for_turn()` (around line 4333):
|
||||
|
||||
```python
|
||||
if interrupted:
|
||||
logger.warning("MEMDEBUG: skip - interrupted=True")
|
||||
if not (self._memory_manager and final_response and original_user_message):
|
||||
logger.warning("MEMDEBUG: skip - manager=%s final=%s orig=%s",
|
||||
bool(self._memory_manager), bool(final_response), bool(original_user_message))
|
||||
if not (user_text and response_text):
|
||||
logger.warning("MEMDEBUG: skip - user_text=%s response_text=%s",
|
||||
bool(user_text), bool(response_text))
|
||||
```
|
||||
|
||||
Then restart gateway (`hermes gateway restart` from external shell) and `grep MEMDEBUG ~/.hermes/logs/agent.log` after one conversation turn to identify which guard fires.
|
||||
|
||||
## Verification
|
||||
```bash
|
||||
python3 -c "
|
||||
import sqlite3
|
||||
db = '/home/hermes/.hermes/mnemosyne/data/mnemosyne.db'
|
||||
conn = sqlite3.connect(db)
|
||||
cur = conn.cursor()
|
||||
cur.execute('SELECT MAX(created_at) FROM working_memory')
|
||||
print('Last WM capture:', cur.fetchone()[0]) # Should be recent
|
||||
"
|
||||
```
|
||||
|
||||
## Prevention
|
||||
- Never use `hermes config set memory.mnemosyne.<field>` for list/dict/null values
|
||||
- After any config change, verify YAML types with `yaml.safe_load` + `type()` on each field
|
||||
- After any gateway restart, verify capture within 2 hours via health monitor
|
||||
|
||||
## Related
|
||||
- [[systems/mnemosyne]]
|
||||
- [[incidents/2026-08-01-mnemosyne-capture-failure]] — Previous capture failure (YAML stringification)
|
||||
- [[incidents/2026-07-29-mnemosyne-relay]] — Earlier relay scope corruption
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
title: "Gmail Read-State Incident (2026-08-29)"
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-08-29
|
||||
updated: 2026-08-29
|
||||
verified_on: 2026-08-29
|
||||
confidence: high
|
||||
tags: [email, gmail, cron, himalaya, incident]
|
||||
sources: [session:default/20260829_204103_9e1087]
|
||||
---
|
||||
|
||||
# Gmail Read-State Incident
|
||||
|
||||
**Date discovered:** 2026-08-29 20:41 AWST
|
||||
**Status:** Active — mitigation in place, root cause identified but not fully fixed
|
||||
|
||||
## Summary
|
||||
|
||||
Anthony's Gmail inbox messages were being automatically marked as read without his consent. The issue was traced to Hermes cron jobs using Himalaya's `message read` command, which applies the "seen" flag even with `--preview`.
|
||||
|
||||
## Timeline
|
||||
|
||||
- **2026-08-29 20:41** — Anthony reports: "What is marking my Gmail email as read it is not meant to be"
|
||||
- **Investigation** — Loaded skills: `gmail-imap-email-monitor`, `hermes-cron-operations`, `email-inbox-triage`
|
||||
- **Root cause identified** — `fetch_newsletters.py` (line 49) runs:
|
||||
```bash
|
||||
himalaya message read --preview --folder INBOX <message-id>
|
||||
```
|
||||
Himalaya's `message read` command **automatically applies the "seen" flag** unless `--preview` is used for preview-only access. However, even `--preview` may still mark messages read in some configurations.
|
||||
- **Other scripts checked** — `gmail_readonly_imap.py` uses `SELECT readonly` and `BODY.PEEK[]` (safe, does not mark read)
|
||||
- **Mitigation applied:**
|
||||
- Stopped `goimapnotify.service` (IMAP watcher)
|
||||
- Paused "Important-mail monitor" cron job (if present)
|
||||
- Newsletter digest already patched to use `--preview`
|
||||
|
||||
## Root Cause
|
||||
|
||||
The `fetch_newsletters.py` script at `/home/hermes/.hermes/scripts/fetch_newsletters.py` line 49:
|
||||
|
||||
```python
|
||||
r2 = subprocess.run(
|
||||
["himalaya", "message", "read", "--preview", "--folder", "INBOX", m["id"]],
|
||||
capture_output=True, text=True, timeout=60
|
||||
)
|
||||
```
|
||||
|
||||
Despite using `--preview`, Himalaya may still mark messages as read depending on configuration. The proper fix is to use readonly IMAP with `BODY.PEEK[]` instead of the `himalaya message read` command.
|
||||
|
||||
## Impact
|
||||
|
||||
- Newsletter emails processed by the Tech & AI Newsletter Digest cron job were marked as read
|
||||
- Potential for other Gmail automation scripts to have similar behavior
|
||||
- User trust issue — unexpected state changes in personal email
|
||||
|
||||
## Resolution Status
|
||||
|
||||
**Mitigation:** Partially complete
|
||||
- `goimapnotify` stopped
|
||||
- Important-mail monitor paused (if it existed)
|
||||
|
||||
**Fix required:**
|
||||
- Rewrite `fetch_newsletters.py` to use readonly IMAP (`BODY.PEEK[]`) instead of `himalaya message read`
|
||||
- Audit all Gmail-related scripts for similar patterns
|
||||
- Verify `gmail_action_digest_v3.py` and other email scripts use readonly access
|
||||
|
||||
## Related Pages
|
||||
|
||||
- [[systems/scheduled-tasks]] — Tech & AI Newsletter Digest cron job
|
||||
- [[systems/messaging-integrations]] — Email integration patterns
|
||||
- [[runbooks/job-radar]] — Email monitoring patterns (for reference)
|
||||
|
||||
## Prevention
|
||||
|
||||
- All Gmail IMAP scripts should use `SELECT readonly` and `BODY.PEEK[]` patterns
|
||||
- Add a pre-commit check or linting rule for `himalaya message read` usage
|
||||
- Document readonly email access pattern in a runbook
|
||||
@@ -0,0 +1,74 @@
|
||||
---
|
||||
title: OmniRoute cliprox/terra Connection Auth Expiry
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-10-02
|
||||
updated: 2026-10-02
|
||||
verified_on: 2026-10-02
|
||||
confidence: high
|
||||
tags: [cron, omniroute, provider, auth, incident]
|
||||
sources: [live-jobs-json, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040]
|
||||
---
|
||||
|
||||
# OmniRoute `cliprox/terra` Connection Auth Expiry
|
||||
|
||||
**Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); still open
|
||||
**Status:** Open — not self-recovering
|
||||
**Impact:** 1 job failed outright; 2 more enabled jobs share the same model pin and are expected to fail on their next runs
|
||||
|
||||
## Symptoms
|
||||
|
||||
Job `06a0a1c8704f` (**AI Newsletter Curation for Patrick**, Thu 10:00) aborted with:
|
||||
|
||||
```
|
||||
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
|
||||
— please reconnect in the dashboard
|
||||
```
|
||||
|
||||
Provider `custom:Omniroute:20128`, model `cliprox/terra`. The run's failure notice was
|
||||
delivered to forum topic `346`, but no newsletter digest was produced.
|
||||
|
||||
## Root Cause
|
||||
|
||||
A **connection credential at the provider**, not a missing or plan-gated model and not a
|
||||
routing error. The error names a single connection (`All 1 connection(s)`) whose
|
||||
authorisation lapsed, so the model pin itself is still valid — the *credential behind it*
|
||||
is not.
|
||||
|
||||
This matters because it is the one failure class from the 2026-10-01 sweep that **cannot
|
||||
self-recover**. A retired or plan-gated model stays broken until repinned; an expired
|
||||
connection stays broken until a human re-authenticates it in the dashboard.
|
||||
|
||||
## Blast Radius
|
||||
|
||||
Three enabled jobs run `cliprox/terra`:
|
||||
|
||||
| Job | Name | Schedule | Delivery |
|
||||
|---|---|---|---|
|
||||
| `06a0a1c8704f` | AI Newsletter Curation for Patrick | `0 10 * * 4` | `telegram:-1004321904721:346` |
|
||||
| `1b23c042e17d` | Enhanced Weekend Adventure Planner | `0 8 * * 5` | `telegram:-1004321904721:102` |
|
||||
| `79f9bb41ecf5` | Weekly Review | `0 18 * * 0` | `origin` |
|
||||
|
||||
Only `06a0a1c8704f` had run by 2026-10-02; the other two are exposed on their next
|
||||
scheduled runs.
|
||||
|
||||
## Fix
|
||||
|
||||
Re-authenticate the OmniRoute connection for `cliprox/terra`. Do **not** repin these jobs
|
||||
to another raw model as a first move — that treats a credential fault as a routing fault
|
||||
and will produce a second, different outage. If the connection cannot be restored, repin
|
||||
all three jobs together to a routing alias (`smart-route`).
|
||||
|
||||
## Distinguish From
|
||||
|
||||
- **Plan-gated model** (403 `MODEL_NOT_IN_PLAN`, job `dca8482e4f76`) — the upstream rejects
|
||||
the model by plan; no credential involved.
|
||||
- **Retired model** (HTTP 410, job `1a8e90ea2271`) — the model no longer exists upstream.
|
||||
- **Catalog-missing model** (HTTP 400, jobs `fad5a86979d4` / `0e753e9720fd`) — the model
|
||||
name is absent from the provider's live catalog.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/model-providers]]
|
||||
- [[systems/omniroute]]
|
||||
Reference in New Issue
Block a user