Send easy questions to a cheap model with AnythingLLM's Model Router
Pick a model per AnythingLLM workspace, then add a Model Router that answers on Haiku by default and switches to Sonnet only for messages that need it.
· ranx.cloud
This is for someone who uses AnythingLLM every day across a few workspaces (documents, research, maybe a website widget) and has one strong model set for everything because it was the safe choice. You end up with each workspace on the model it needs, and a Model Router on the busy workspaces that answers routine messages with a cheap model and switches to a stronger one when the message calls for it: a long conversation, an image, or a request for analysis. Every answer shows which model handled it and why.
Settings below were checked against AnythingLLM v1.19.0 (Docker). The examples use Anthropic models, but the router works across any providers you've configured.
Start with a model per workspace
Before adding a router, check whether a fixed model per workspace already solves it. A workspace's gear icon → Chat Settings → Workspace LLM Provider lets you pick a provider and a Workspace Chat model for that workspace alone; leave it on the default and the workspace uses the system model from Settings → AI Providers → LLM.
| Workspace | Model | Why |
|---|---|---|
| Manuals and how-to docs | Claude Haiku 5.5 | Lookups: find the passage, restate it. |
| Contracts | Claude Sonnet 5.5 | Exceptions, conditions, careful reading. |
| Website support widget | Claude Haiku 5.5 | High volume, short answers from docs. |
| Research (agent) | Sonnet 5.5 as the agent model | Multi-step tool use. Set under Agent Configuration. |
Each workspace also has its own System Prompt in the same tab, so a contracts workspace can insist on quoting clauses while a how-to workspace keeps answers to five lines. Prompt and model are per workspace; documents are added per workspace too. That's often enough.
A router earns its place in a workspace where the questions are mixed: most are simple, a few are hard, and you can't tell in advance which workspace a question belongs in.
How the router decides
A router has a primary model and an ordered list of rules. For each message:
- Calculated rules are checked first, top to bottom. They look at the message and conversation (prompt text, conversation token count, message count, current hour, image attached) and cost nothing to evaluate.
- If none matched, LLM Classified rules are checked as a batch: the primary model reads the message and your plain-English descriptions and decides whether one applies. That's one extra model call.
- If nothing matched, the primary model answers.
A matched route is sticky. Follow-up messages in the same conversation stay on that model for the cooldown period (300 seconds by default), and each message on it renews the timer. Calculated rules are still checked on every message. A "no match" result from the LLM classifier is remembered for 30 seconds, so a burst of messages doesn't trigger a classifier call each time.
Step 1: Create the router
Settings → AI Providers → Model Router → New Router.
| Field | Value |
|---|---|
| Name | Haiku first |
| Description | Haiku by default, Sonnet for long, visual or analytical requests |
| Primary Provider & Model | Anthropic, Claude Haiku 5.5 |
| Cache Cooldown (seconds) | 120 |
The primary model answers everything no rule catches and also runs the LLM-classified rules, so it should be fast and cheap. If a provider in the picker shows "(setup required)", it asks for that provider's credentials right there.
The cooldown is a trade-off. Longer keeps a hard conversation on the strong model for its follow-ups ("and what about clause 9?"), which is usually what you want. Shorter returns to the cheap model sooner. Two minutes suits a chat where you send a message every minute or so. 0 turns caching off, which also makes LLM-classified rules run on every message.
Step 2: Add rules
Open the router and use New Rule. Each rule has a Title, a Rule Type and Route to Provider & Model.
Rule 1, Calculated: "Images"
Property: Has Image Attachment
Comparator: equals
Value: True
Route to: Anthropic, Claude Sonnet 5.5
Rule 2, Calculated: "Long conversation"
Property: Conversation Token Count
Comparator: greater than
Value: 30000
Route to: Anthropic, Claude Sonnet 5.5
The token count is everything that would be sent: system prompt, document excerpts, pinned documents, history and the new message. Past 30,000 tokens you're paying mostly for input anyway, and a long, context-heavy conversation is where the cheap model loses track. Note that this rule raises the price of exactly the most expensive messages, so set the number with the cost section below in mind.
Rule 3, Calculated: "Explicit analysis words"
Property: Prompt Content
Comparator: contains
Value: compare, analyze, analyse, trade-off, draft, rewrite, review this
Route to: Anthropic, Claude Sonnet 5.5
contains takes a comma-separated list, matches any of them, and ignores case. matches (regex) is there if you need word boundaries, e.g. /\b(compare|analy[sz]e)\b/i.
Rule 4, LLM Classified: "Needs careful reasoning"
Match Description:
The user asks for interpretation of contract terms or obligations, a
multi-step calculation, a recommendation between options, or anything
where a wrong answer has financial or legal consequences.
Route to: Anthropic, Claude Sonnet 5.5
This catches what keyword rules miss, at the price of a small Haiku call when no cached decision exists. Write the description as a test the classifier can apply to one message, not as a topic list.
Rules can have several conditions joined with ANY or ALL, and you can drag rules to reorder them. Calculated rules always run before LLM rules regardless of order.
Step 3: Attach it to a workspace
In the workspace's Chat Settings, open Workspace LLM Provider, choose Model Router, and select Haiku first. Each answer then shows a small badge, "Routed to … via …", naming the model and the rule. Ask a few questions of each kind and check the badges.
If you leave Agent Configuration empty in that workspace, @agent sessions go through the router too. For research-heavy workspaces, set the agent model explicitly instead.
A workspace that backs the website widget can use a router as well; the widget follows the workspace's model choice unless you enabled model overrides on the embed. For a support widget, a single cheap model is usually right, and the router is more useful in your own workspaces.
Costs
Prices are Anthropic's list prices as of November 2026: Claude Haiku 5.5 is $0.10 / $0.50 per million input / output tokens for prompts up to 100,000 tokens, Claude Sonnet 5.5 is $2 / $10.
Assume 3,000 messages a month, 3,000 input and 400 output tokens each on average.
Everything on Sonnet 5.5:
3,000 × (3,000 × $2 + 400 × $10) / 1,000,000 = 3,000 × $0.010 = $30.00
Router, 80% Haiku / 20% Sonnet:
2,400 × (3,000 × $0.10 + 400 × $0.50) / 1,000,000 = 2,400 × $0.0005 = $1.20
600 × $0.010 = $6.00
classifier: ~1,500 calls × (600 × $0.10 + 20 × $0.50) / 1,000,000 ≈ $0.11
total ≈ $7.30
Two caveats on the split. Routed messages are usually the longer ones, so 20% of messages can be well over 20% of tokens; and stickiness keeps follow-ups on Sonnet. Watch the badges for a week and adjust the rules if Sonnet is answering "thanks" and "ok".
The instance plan doesn't change any of this: model calls go to your provider, and a router adds no load worth mentioning on the instance.
Prompt caching
Under Show advanced settings in the Anthropic LLM settings there's a Prompt Caching option. It caches the system message. In a documents workspace that message includes the retrieved excerpts, which change with every question, so caching rarely hits; it helps more in workspaces without documents and a long, fixed system prompt. It doesn't change the routing math above.
What goes wrong
An overnight hour rule never matches. A Current Hour (0-23) rule with between 22,6 saves fine but never matches in v1.19.0, because the check requires the hour to be both at least 22 and at most 6 (#7034, fixed after v1.19.0). Use two rules, 22,23 and 0,6, until you're on a release with the fix.
Hour rules fire at the wrong time. The current hour is the server's clock, not your browser's. Check the hour the instance reports (send a message at a known time with a temporary rule) before relying on time-of-day routing.
Sonnet keeps answering after the hard question is over. Stickiness. Lower the cooldown, or start a new thread when the topic changes.
Everything goes to the primary model. Calculated rules are fine but the LLM-classified one never matches: the description is too vague, or the primary model is too weak to judge it. Make the description concrete. On Azure OpenAI, LLM-classified rules silently fell back to the primary model in earlier releases (#6686).
A rule can't be saved. "Condition 1 is incomplete" means a property, comparator or value is empty. A target provider showing "(setup required)" needs its credentials entered first.
The widget sends too much history. On older versions, an embed whose workspace used the router ignored the embed's Message History Limit (#6564). Update if you see answers that refer to messages from far back in a long widget session.
Routing decisions reset after a restart. Sticky routes and cached classifications live in memory, so after the instance restarts (for an update, say) the next message is routed from scratch. At worst, one conversation goes back to the primary model early.
