snapshot: preserve central wiki state 2026-08-15 (7 modified + 1 new file)

2026-08-15 22:11:30 +08:00
commit 92e8028bb8
71 changed files with 9897 additions and 0 deletions
@@ -0,0 +1,44 @@
+# [2026-08-04] OmniRoute free-DeepSeek silent stops (empty-after-tool)
+
+## Status
+Mitigated — combos reworked; upstream still degraded.
+
+## Symptom
+On **both** Hermes instances (ops/Tusk + the other instance), turns using `deepseek-flash-free` (and `smart-route` via `good-free`) did one tool call, then went **silently** — no text, no error. "continue" → one more tool call → silent stop again.
+
+## Root cause
+The free-DeepSeek upstreams behind the combos started returning **valid-but-empty HTTP 200s right after a tool_calls turn** (no content, no further tool call). Hermes can't distinguish "empty" from "model chose to stop" → ends turn silently. Began **2026-08-01 ~09:14 UTC**, still active. OmniRoute log markers:
+
+```
+[STREAM] Empty assistant response after tool_calls completion (<provider>:<model>)
+[ProxyEgress] <provider>/<conn-hash> ... status=error
+```
+
+Per-model evidence (counts in `storage.sqlite` era log):
+- `238d5978/deepseek-v4-flash` — 63× empty-after-tool (conn `d4b9fee0`, 189 proxy errors)
+- `238d5978/deepseek-v4-flash-0731` — 70× empty-after-tool
+- `eb5af0da/deepseek-v4-flash-thinking:free` (conn `f8169018`) — 504 proxy errors (combo #1 slot)
+- `nvidia` deepseek + `opencode-zen` deepseek-free — noisy, errors
+
+Strategy trap: both combos used `priority`, which retries the **same** broken connection and never fails over on an empty 200.
+
+## Change applied (2026-08-04, ~03:30 AWST)
+Edited `combos` in `/var/lib/omniroute/storage.sqlite` on CT600 (live DB, hot-reloads — no restart):
+
+- **`deepseek-flash-free`** — strategy `priority` → `fill-first`; 8 → 7 models: healthy free first (`mimo-v2.5-free`, `laguna-s-2.1:free` ×2 providers, `step-3.7-flash`, `glm-5.2-cheap:free`), one deepseek (`deepseek-v4-flash-thinking:free`) demoted to last resort.
+- **`good-free`** — strategy `priority` → `fill-first`; 15 → 12 models. **Dropped** the proven culprits: `238d5978/deepseek-v4-flash`, `nvidia/deepseek-ai/deepseek-v4-pro`, `nvidia/deepseek-ai/deepseek-v4-flash`. `opencode/deepseek-v4-flash-free` kept as last resort.
+
+Backup of both pre-change rows: `/tmp/combo_backup_20260804.json` on the Hermes gateway LXC.
+
+## Verification
+- Live API (`/v1/combos`) reflects both changes immediately (hot reload).
+- 2-step tool-call round-trip through `deepseek-flash-free` returns proper final content.
+- Direct round-trips (stream + non-stream) through the OLD combo were also healthy — the failure needs the degraded upstream + real session, hence counts not repro.
+
+## Follow-ups
+- Upstream free-DeepSeek tier still degraded; re-check counts in a few days and restore deepseek-first ordering if it recovers.
+- Consider `auto`+explorationRate for self-tuning if free tier stays flaky.
+- Hermes-side: empty-post-tool turns are nudge/retry/fallback handled, but still end silently once exhausted — possible future improvement: surface "(empty after tool calls)" more visibly.
+
+## Refs
+- [[systems/omniroute]]