Files
agent-estate-wiki/comparisons/model-routers.md
T

87 lines
5.5 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Model Routers
type: comparison
status: active
created: 2026-07-22
updated: 2026-10-20
verified_on: 2026-10-20
confidence: high
tags: [comparison, routing]
sources: [raw/configs/hermes-config-sanitized.txt]
---
# Model Routers
## Purpose
Compare LiteLLM, OmniRoute, and OpenRouter as model-routing layers for Hermes.
## At a glance
| | LiteLLM | OmniRoute | OpenRouter |
|---|---|---|---|
| Role in Hermes | Fallback + a *custom_providers* entry | **Auxiliary + the pin on essentially every model-calling cron job** | Not configured as a standalone provider |
| Endpoint | `http://litellm:4000/v1` | `http://omniroute:20128/v1` (public: `omniroute.martinwa.org`) **and** `http://100.78.63.20:20128` (`omni-101`) | n/a |
| Model count (live, 2026-10-20) | **1869** (`/v1/models` with the configured key) | **6457** at `omniroute:20128/v1/models` | n/a |
| Model count (declared in `config.yaml`) | 1728 | 5398 (`Omniroute:20128`) / 5384 (`omni-101`) | n/a |
| Routing style | Straight proxy | Combo strategy, LKGP scoring, session affinity | n/a |
| Aux roles used for | — | `skills_hub`, `approval`, `mcp` | — |
| Config presence | `custom_providers` in `~/.hermes/config.yaml` | same (two separate entries) | absent from `custom_providers`; only referenced via a kept [[decisions/keep-openrouter-as-fallback]] decision |
> **The "LiteLLM is the default/primary" row was wrong by 2026-10-20 (verified and corrected).**
> The live `model:` block in `~/.hermes/config.yaml` is:
> ```yaml
> model:
> provider: nous
> default: meituan/longcat-2.5-preview:free
> base_url: https://inference-api.nousresearch.com/v1
> api_mode: chat_completions
> persist_switch_by_default: false
> ```
> So the **primary model path is Nous Research direct**, not LiteLLM. LiteLLM is the *last* entry
> in the fallback chain (`or-inclusionai_ling-2_6-1t` via `custom` → `http://litellm:4000/v1`).
>
> **The two OmniRoute endpoints matter.** `Omniroute:20128` is a DNS name (m93p CT101);
> `omni-101` is the raw IP `100.78.63.20:20128` (node `omniroute-1`) and is the alias pinned on
> effectively every model-calling cron job. The two carry **different catalogs** (5398 vs 5384
> declared models) and the `omni-101` entry's own `model:` is `smart-route` while
> `Omniroute:20128`'s is `go-cheap`.
>
> **Also: both OmniRoute endpoints and LiteLLM now require authentication.** `curl` without a key
> returns **401 `invalid_api_key`** on all three `/v1/models` endpoints. The keys live in
> `~/.hermes/.env` via each provider's `key_env` (e.g. `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`).
> Any runbook that probes these endpoints with a bare `curl` will see 401 and may wrongly conclude
> the provider is down — it is not.
## Detail
### LiteLLM — fallback tail + a `custom_providers` entry
LiteLLM hosts the `freellmapi-auto` model alias and is reachable at `http://litellm:4000/v1`, but
it is **no longer the default chat path** — that is Nous direct
(`meituan/longcat-2.5-preview:free`, per the live `model:` block). LiteLLM appears in the
fallback chain as `or-inclusionai_ling-2_6-1t` via the `custom` provider.
See [[systems/model-providers]] and [[concepts/model-routing]].
### FreeLLMAPI — the router behind `freellmapi-auto`
Unified LLM router at `100.82.40.23:3001` (also `freellmapi.martinwa.org`). Verified 2026-08-12:
- **3-axis weighted scoring:** Reliability + Speed + Intelligence, with user-configurable weights.
- **Dynamic guardrail multiplier:** `Final score = weighted avg × (free-quota headroom × live rate-limit penalty)`. Below 1.0 means the model is held back.
- **Penalty Inspector ("Router Pressure"):** granular penalty factors (e.g. `score × 0.5`) from recent 4xx/5xx, cooldowns, RPM/RPD limits, monthly token caps.
- **Capability-aware routing:** filters candidates on whether the request has images (vision) or tool calls.
- Exposes `/api/fallback/routing`, `/api/fallback/penalty-inspector`, `/api/fallback/token-usage`.
This is the router that resolves Hermes's **default model** `freellmapi-auto` — distinct from OmniRoute's LKGP (below).
### OmniRoute — ⚠ main chat path again since 2026-10-24 (was "fallback / auxiliary")
**Re-verified 2026-10-26: this section is out of date.** Hermes's **default** is `custom:omni-101` /
`smart-route` on the `omniroute-1` node (`http://100.78.63.20:20128`) — i.e. OmniRoute **is** the main chat
path, as it was before the 2026-10-20 → 10-24 `nous` detour → [[systems/current-profile]].
It also still handles auxiliary roles and appears in the `fallback_providers` chain.` Has a much larger advertised catalog (~3430 vs ~1512) but that catalog can diverge from what's actually resolvable — combo discovery vs. resolution can differ (documented known issue). See [[systems/omniroute]].
### OpenRouter — kept only as a fallback decision, not wired in
**Verified:** absent from `custom_providers` in the live config. A decision page ([[decisions/keep-openrouter-as-fallback]]) records intent to retain it as a fallback option, but there's no live routing through it today — see [[systems/openrouter]] for the "previously used, largely replaced" history.
## Verdict
Two routers are live (**OmniRoute primary** for the default path and auxiliary slots since 2026-10-24 19:06; **LiteLLM** as the deeper fallback), OpenRouter is a retained *intent*, not a working path — re-verified 2026-10-26. If [[decisions/keep-openrouter-as-fallback]] is still desired, it needs an actual `custom_providers` entry — currently it's fallback-in-name-only.
## Related
- [[systems/model-providers]]
- [[concepts/model-routing]]