Virtual models & fallback

A virtual model is a named, client-callable model id that resolves to an ordered list of real targets — each a (provider/model, account, weight) — that merido tries in turn, with fallback. It's the LiteLLM "model group" concept: callers name one stable model, and merido decides which real target serves it.

Why use one

• A stable name across providers. Your tools call smart-coder; you control which real models back it without touching the tools.
• Fallback. If a target errors or is rate-limited, merido advances to the next one.
• Cost / latency / health awareness. Targets can be ordered by live cost or latency scores, and unhealthy or cooled-down targets are skipped.

Creating one

You can create virtual models three ways:

Dashboard

Use the Virtual Models page: name it, pick a strategy, and add targets (the target picker suggests real provider/model options per connected account, so you select rather than type).

CLI / env (MERIDO_ROUTES)

Define them declaratively as a JSON array in the MERIDO_ROUTES environment variable. Each entry is a virtual-model definition (name, strategy, targets). This is parsed specially at startup — see Configuration.

API

POST /api/virtual-models (and GET / PUT / DELETE per id; POST /api/virtual-models/{id}/toggle, POST /api/virtual-models/reorder). See API endpoints.

Strategies

A virtual model declares one strategy (validated up front):

Strategy	Behavior
failover	Try targets strictly in declared order, advancing only on failure.
load_balance	Rotate the lead target across invocations (sticky window), then fall back through the rotated order.
weighted	Load-balance variant that biases the lead target by each target's weight.
cost_optimized	Order targets by a live cost score (lower is better), then fall back through that order. Targets with no score sort last, and ties keep your declared order.
latency_based	Order targets by a live latency score (lower is better), then fall back through that order. Targets with no score sort last, and ties keep your declared order.

For load_balance, a sticky limit pins consecutive requests to the chosen lead target before the rotation cursor advances. For the score-driven strategies (cost_optimized / latency_based), merido falls back to your declared target order whenever a score is unavailable — when no score map exists yet, for any target that has no score, and to break ties between equal scores. Declared order is always the stable backstop, so a freshly-created virtual model behaves like failover until cost/latency signals accumulate.

How a request is resolved

When a caller names a virtual model:

1. merido expands it to its ordered list of targets.
2. Live feedback (health, quota, per-(account, model) locks, ranking) filters and orders candidates.
3. The router picks a target and an account for it (cost-, latency-, health-, circuit-breaker-aware).
4. On failure it walks the fallback chain: next account → next target → next tier.

Context-window fallback

When a target rejects a prompt as too long for its context window, merido does not just fail down the chain — it reorders the untried targets largest-context-window first, so the next attempt lands on a model that can actually hold the prompt. Context windows come from the model metadata catalog (/v1/model_group/info), synced from the upstream rate card with a built-in fallback table. This mirrors LiteLLM's context_window_fallbacks, but uses the virtual model's own targets — no separate fallback list to configure.

Capability-aware routing

merido inspects what each request needs — image, audio or PDF/document input, tools, or extended reasoning — and matches it against what each target model is known to support (from the same model-capability metadata synced from the rate card, plus a curated table). For a virtual model this has two effects:

• Prefer a capable target. When a request carries an image, audio, or PDF part, targets that can serve that modality are floated to the front of the chain, so a capable model leads instead of being reached only after a wasted fallback hop. Plain text requests are unaffected — the strategy order stands.
• Skip an incapable one. A target positively known not to accept the modality is skipped (fallback moves to the next target) rather than sent a request it will reject. Unknown models are assumed capable (fail-open), so a missing metadata entry never blocks a request.

Video input is displayed in the Models capability view but is not used for routing — chat requests don't carry video.

Circuit breaker & cooldowns

When a target/account repeatedly fails, the circuit breaker records the failure and the account+model is put on a cooldown so the router stops hammering it; healthy alternatives are tried instead. The breaker recovers automatically once the cooldown expires. In a multi-instance deployment these cooldowns are shared across instances via Redis.

The threshold is fixed: the breaker opens after 5 consecutive failures and the cooldown is 30 seconds. These are not configurable knobs today.

What counts as a failure

Only upstream-health failures count toward that threshold — a 5xx, a transport/timeout error, or an unrecognized status. A 4xx — including 401 / 403 / 404 and a rate limit — no longer trips the breaker: it still falls back to the next candidate (or surfaces to the caller), it just isn't treated as evidence the account itself is unhealthy.

429 handling is scoped

A 429 is classified before it's acted on:

• Plain rate limit — locks only that (account, model) pair; every other model on the same account stays fully routable. No account-wide cooldown, no breaker count.
• Quota / credit exhaustion — detected from the error body (quota/credit/billing/limit language) or a Retry-After longer than 5 minutes — cools the whole account (honoring the upstream's Retry-After, capped at 30 minutes) in addition to the model lock, since an account that's out of quota won't recover for any model on it either.

Retry-After is parsed as integer seconds, an HTTP-date, or a relative shorthand (90s / 5m / 2h).

Recovery is a single probe

Once a cooldown expires, the breaker admits exactly one trial request to test the account; concurrent requests in the meantime skip it and fall back as usual. That trial succeeding closes the breaker; failing re-opens it for another cooldown.

Operator rules keep precedence

An operator-supplied MERIDO_ERROR_RULES classification always overrides this built-in logic — see Environment variables.

Related

• Add providers & keys — the accounts virtual models route across.
• Payload rules — the reactive field-strip retry that deliberately does not touch the circuit breaker above.
• Usage & the Advisor — the advisor can suggest cheaper targets.