Files
agent-estate-wiki/incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
T
Hermes ab3652aa20 Stocktake 2026-11-12: tailscale live-vs-page drift corrected; sftpgo incident re-verified; pre-dispatch-filter signature found recurring on a second job/pin
- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json.
  mailflow/qwenpaw/springroll read Online:false in the peer record but all three ping
  (CT1262 + CT465 running; springroll via DERP(sin)) - stale peer records, rows stay online.
  omarchymac genuinely stale in the other direction (offline 2d/11-08 -> offline 4d, ping timeout).
  9 named offline ages + 6 codex ephemeral peers refreshed. New guest CT1007 homebridge
  (running, tailscaled inactive). DEAD Serve targets 8769/9123 re-confirmed dead; CT201 table
  unchanged; container counts 17 LXC+4 VM (pve) / 20 LXC+2 VM (m93p) confirmed.
- incidents/2026-09-02-sftpgo-outage.md: 22d stale, re-probed live (pct list running, peer
  Online:true, pong, healthz 401; swap now 512 not 43Gi). Cleared.
- incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md: signature recurred 2026-11-10 on
  cbfe434a3b5e (custom:omni-101/free-stack, HTTP 429 credits 100%); incident row resolved
  2026-11-12T00:25. Original subject clean 4 slots. Retention-floor caveat recorded.
- Threshold re-verifications (19d/18d): mnemosyne-relay (tools:None gone, 0 log hits),
  work-outlook (himalaya still login not supported), omniroute-free-deepseek
  (good-free/free-stack both HTTP 200, non-deepseek resolution).
- Correction recorded: previous entry mis-described the 2026-11-12 stale page as log.md.
- wiki-maintain.sh re-run after all edits: freshness 0 stale / 130 fresh / 3 no-date of 133.
2026-11-12 03:19:12 +08:00

75 lines
4.6 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: OmniRoute free-DeepSeek Silent Stops (Empty-After-Tool) — 2026-08-04
type: incident
status: monitoring
created: 2026-08-04
updated: 2026-11-12
verified_on: 2026-11-12
confidence: high
tags: [omniroute, deepseek, free-tier, silent-failure, incident, mitigated]
sources: [live-probe, omniroute-log, session-20260804]
---
# [2026-08-04] OmniRoute free-DeepSeek silent stops (empty-after-tool)
## Status
Mitigated — combos reworked; upstream still degraded.
## Symptom
On **both** Hermes instances (ops/Tusk + the other instance), turns using `deepseek-flash-free` (and `smart-route` via `good-free`) did one tool call, then went **silently** — no text, no error. "continue" → one more tool call → silent stop again.
## Root cause
The free-DeepSeek upstreams behind the combos started returning **valid-but-empty HTTP 200s right after a tool_calls turn** (no content, no further tool call). Hermes can't distinguish "empty" from "model chose to stop" → ends turn silently. Began **2026-08-01 ~09:14 UTC**, still active. OmniRoute log markers:
```
[STREAM] Empty assistant response after tool_calls completion (<provider>:<model>)
[ProxyEgress] <provider>/<conn-hash> ... status=error
```
Per-model evidence (counts in `storage.sqlite` era log):
- `238d5978/deepseek-v4-flash` — 63× empty-after-tool (conn `d4b9fee0`, 189 proxy errors)
- `238d5978/deepseek-v4-flash-0731` — 70× empty-after-tool
- `eb5af0da/deepseek-v4-flash-thinking:free` (conn `f8169018`) — 504 proxy errors (combo #1 slot)
- `nvidia` deepseek + `opencode-zen` deepseek-free — noisy, errors
Strategy trap: both combos used `priority`, which retries the **same** broken connection and never fails over on an empty 200.
## Change applied (2026-08-04, ~03:30 AWST)
Edited `combos` in `/var/lib/omniroute/storage.sqlite` on CT600 (live DB, hot-reloads — no restart):
- **`deepseek-flash-free`** — strategy `priority` → `fill-first`; 8 → 7 models: healthy free first (`mimo-v2.5-free`, `laguna-s-2.1:free` ×2 providers, `step-3.7-flash`, `glm-5.2-cheap:free`), one deepseek (`deepseek-v4-flash-thinking:free`) demoted to last resort.
- **`good-free`** — strategy `priority` → `fill-first`; 15 → 12 models. **Dropped** the proven culprits: `238d5978/deepseek-v4-flash`, `nvidia/deepseek-ai/deepseek-v4-pro`, `nvidia/deepseek-ai/deepseek-v4-flash`. `opencode/deepseek-v4-flash-free` kept as last resort.
Backup of both pre-change rows: `/tmp/combo_backup_20260804.json` on the Hermes gateway LXC.
## Verification
- Live API (`/v1/combos`) reflects both changes immediately (hot reload).
- 2-step tool-call round-trip through `deepseek-flash-free` returns proper final content.
- Direct round-trips (stream + non-stream) through the OLD combo were also healthy — the failure needs the degraded upstream + real session, hence counts not repro.
## Follow-ups
- Upstream free-DeepSeek tier still degraded; re-check counts in a few days and restore deepseek-first ordering if it recovers.
- Consider `auto`+explorationRate for self-tuning if free tier stays flaky.
- Hermes-side: empty-post-tool turns are nudge/retry/fallback handled, but still end silently once exhausted — possible future improvement: surface "(empty after tool calls)" more visibly.
## Re-verified 2026-11-12 03:4x AWST (stocktake — page was 19d old, inside the staleness window)
Both reworked combos probed live on **`omniroute-1`** (`100.78.63.20:20128`, `custom:omni-101`) with the
correct `key_env` (`HERMES_CUSTOM_100_78_63_20_20128_API_KEY`) and the exact aliases used:
| Probe | Result |
|---|---|
| `good-free` (the 12-model rework) | **HTTP 200**, resolving to **`glm-5.2-cheap:free`** with a valid non-empty completion |
| `free-stack` | **HTTP 200**, resolving to **`stepfun-ai/step-5-preview`** with a valid completion |
**The mitigation is holding, and the 2026-08-04 follow-up can be partially closed:** both combos answer with
real content, and **neither resolves to a deepseek model** — which is exactly the intent of dropping the
proven culprits from `good-free`. ⚠ **What is *not* re-verified:** I did not individually probe the
free-DeepSeek upstreams (`238d5978/deepseek-v4-flash`, `nvidia/deepseek-ai/deepseek-v4-pro`,
`deepseek-v4-flash-thinking:free`), so the original claim *"upstream free-DeepSeek tier still degraded"*
stands **neither confirmed nor refuted** — the combos serve healthy non-deepseek free models, which may mean
the upstream recovered *or* simply that it is no longer reached. **Restoring deepseek-first ordering is
therefore still not justified** and should wait on a direct probe of those three upstreams.
## Refs
- [[systems/omniroute]]