- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json. mailflow/qwenpaw/springroll read Online:false in the peer record but all three ping (CT1262 + CT465 running; springroll via DERP(sin)) - stale peer records, rows stay online. omarchymac genuinely stale in the other direction (offline 2d/11-08 -> offline 4d, ping timeout). 9 named offline ages + 6 codex ephemeral peers refreshed. New guest CT1007 homebridge (running, tailscaled inactive). DEAD Serve targets 8769/9123 re-confirmed dead; CT201 table unchanged; container counts 17 LXC+4 VM (pve) / 20 LXC+2 VM (m93p) confirmed. - incidents/2026-09-02-sftpgo-outage.md: 22d stale, re-probed live (pct list running, peer Online:true, pong, healthz 401; swap now 512 not 43Gi). Cleared. - incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md: signature recurred 2026-11-10 on cbfe434a3b5e (custom:omni-101/free-stack, HTTP 429 credits 100%); incident row resolved 2026-11-12T00:25. Original subject clean 4 slots. Retention-floor caveat recorded. - Threshold re-verifications (19d/18d): mnemosyne-relay (tools:None gone, 0 log hits), work-outlook (himalaya still login not supported), omniroute-free-deepseek (good-free/free-stack both HTTP 200, non-deepseek resolution). - Correction recorded: previous entry mis-described the 2026-11-12 stale page as log.md. - wiki-maintain.sh re-run after all edits: freshness 0 stale / 130 fresh / 3 no-date of 133.
75 lines
4.6 KiB
Markdown
75 lines
4.6 KiB
Markdown
---
|
||
title: OmniRoute free-DeepSeek Silent Stops (Empty-After-Tool) — 2026-08-04
|
||
type: incident
|
||
status: monitoring
|
||
created: 2026-08-04
|
||
updated: 2026-11-12
|
||
verified_on: 2026-11-12
|
||
confidence: high
|
||
tags: [omniroute, deepseek, free-tier, silent-failure, incident, mitigated]
|
||
sources: [live-probe, omniroute-log, session-20260804]
|
||
---
|
||
|
||
# [2026-08-04] OmniRoute free-DeepSeek silent stops (empty-after-tool)
|
||
|
||
## Status
|
||
Mitigated — combos reworked; upstream still degraded.
|
||
|
||
## Symptom
|
||
On **both** Hermes instances (ops/Tusk + the other instance), turns using `deepseek-flash-free` (and `smart-route` via `good-free`) did one tool call, then went **silently** — no text, no error. "continue" → one more tool call → silent stop again.
|
||
|
||
## Root cause
|
||
The free-DeepSeek upstreams behind the combos started returning **valid-but-empty HTTP 200s right after a tool_calls turn** (no content, no further tool call). Hermes can't distinguish "empty" from "model chose to stop" → ends turn silently. Began **2026-08-01 ~09:14 UTC**, still active. OmniRoute log markers:
|
||
|
||
```
|
||
[STREAM] Empty assistant response after tool_calls completion (<provider>:<model>)
|
||
[ProxyEgress] <provider>/<conn-hash> ... status=error
|
||
```
|
||
|
||
Per-model evidence (counts in `storage.sqlite` era log):
|
||
- `238d5978/deepseek-v4-flash` — 63× empty-after-tool (conn `d4b9fee0`, 189 proxy errors)
|
||
- `238d5978/deepseek-v4-flash-0731` — 70× empty-after-tool
|
||
- `eb5af0da/deepseek-v4-flash-thinking:free` (conn `f8169018`) — 504 proxy errors (combo #1 slot)
|
||
- `nvidia` deepseek + `opencode-zen` deepseek-free — noisy, errors
|
||
|
||
Strategy trap: both combos used `priority`, which retries the **same** broken connection and never fails over on an empty 200.
|
||
|
||
## Change applied (2026-08-04, ~03:30 AWST)
|
||
Edited `combos` in `/var/lib/omniroute/storage.sqlite` on CT600 (live DB, hot-reloads — no restart):
|
||
|
||
- **`deepseek-flash-free`** — strategy `priority` → `fill-first`; 8 → 7 models: healthy free first (`mimo-v2.5-free`, `laguna-s-2.1:free` ×2 providers, `step-3.7-flash`, `glm-5.2-cheap:free`), one deepseek (`deepseek-v4-flash-thinking:free`) demoted to last resort.
|
||
- **`good-free`** — strategy `priority` → `fill-first`; 15 → 12 models. **Dropped** the proven culprits: `238d5978/deepseek-v4-flash`, `nvidia/deepseek-ai/deepseek-v4-pro`, `nvidia/deepseek-ai/deepseek-v4-flash`. `opencode/deepseek-v4-flash-free` kept as last resort.
|
||
|
||
Backup of both pre-change rows: `/tmp/combo_backup_20260804.json` on the Hermes gateway LXC.
|
||
|
||
## Verification
|
||
- Live API (`/v1/combos`) reflects both changes immediately (hot reload).
|
||
- 2-step tool-call round-trip through `deepseek-flash-free` returns proper final content.
|
||
- Direct round-trips (stream + non-stream) through the OLD combo were also healthy — the failure needs the degraded upstream + real session, hence counts not repro.
|
||
|
||
## Follow-ups
|
||
- Upstream free-DeepSeek tier still degraded; re-check counts in a few days and restore deepseek-first ordering if it recovers.
|
||
- Consider `auto`+explorationRate for self-tuning if free tier stays flaky.
|
||
- Hermes-side: empty-post-tool turns are nudge/retry/fallback handled, but still end silently once exhausted — possible future improvement: surface "(empty after tool calls)" more visibly.
|
||
|
||
## Re-verified 2026-11-12 03:4x AWST (stocktake — page was 19d old, inside the staleness window)
|
||
Both reworked combos probed live on **`omniroute-1`** (`100.78.63.20:20128`, `custom:omni-101`) with the
|
||
correct `key_env` (`HERMES_CUSTOM_100_78_63_20_20128_API_KEY`) and the exact aliases used:
|
||
|
||
| Probe | Result |
|
||
|---|---|
|
||
| `good-free` (the 12-model rework) | **HTTP 200**, resolving to **`glm-5.2-cheap:free`** with a valid non-empty completion |
|
||
| `free-stack` | **HTTP 200**, resolving to **`stepfun-ai/step-5-preview`** with a valid completion |
|
||
|
||
**The mitigation is holding, and the 2026-08-04 follow-up can be partially closed:** both combos answer with
|
||
real content, and **neither resolves to a deepseek model** — which is exactly the intent of dropping the
|
||
proven culprits from `good-free`. ⚠ **What is *not* re-verified:** I did not individually probe the
|
||
free-DeepSeek upstreams (`238d5978/deepseek-v4-flash`, `nvidia/deepseek-ai/deepseek-v4-pro`,
|
||
`deepseek-v4-flash-thinking:free`), so the original claim *"upstream free-DeepSeek tier still degraded"*
|
||
stands **neither confirmed nor refuted** — the combos serve healthy non-deepseek free models, which may mean
|
||
the upstream recovered *or* simply that it is no longer reached. **Restoring deepseek-first ordering is
|
||
therefore still not justified** and should wait on a direct probe of those three upstreams.
|
||
|
||
## Refs
|
||
- [[systems/omniroute]]
|