The bugs behind a cheap model router

A router that sends easy questions to cheap models and hard ones to frontier models sounds like a classification problem. Building Promptimizer, I found the classifier was the easy part. The bugs that took real time were the ones where the system looked like it worked.

Promptimizer is an OpenAI-compatible middleware. You bring your own provider keys (OpenAI, Groq, Baseten, OpenRouter and others), it discovers the models and sorts them into economy, standard and frontier tiers, then runs each request through the same path:

classify → pick tier → cache lookup → complete → quality gate → escalate if needed → ledger

The ledger records actual cost against an always-frontier baseline, and negative savings are allowed, because a router that hides its misses is not worth trusting. Anand Velpuri and I took it to GENESIS 2026 at SRM University-AP, where it was runner-up in the AI/ML track out of 300+ teams.

Streaming that wasn't

The chat in the CLI and web console streamed text smoothly. It was fake. routeChatStream awaited the full non-streaming completion and then replayed the finished text in 48-character chunks. On a long answer you stared at a spinner for the whole generation, then watched a typewriter animation of text that already existed.

Real streaming collided with the quality gate. If a cheap model's answer fails the gate, the router escalates to a stronger model, but by then the user has already seen half of the first answer. The fix (cdc14f9) wires the provider's SSE stream end to end and sends pipeline events alongside the tokens:

{ type: routing, model, tier, routing_policy, ... }
{ type: escalation, from_model, to_model, reason }

On an escalation event the client throws away the partial text, and the console shows that the quality gate failed while the stronger model re-streams. Failover to another provider is only allowed before the first token has been sent. After that, switching models silently would splice two answers together.

The provider that came back

After a user disconnected one provider and connected another, easy questions kept working but hard ones failed with a raw "Model is not configured" error from the provider they had just removed. That split was the clue. The resurrected provider's stale models happened to be the cheapest frontier ones, so only frontier-tier requests were routed to them.

There were two independent holes (50af669):

1. Resurrection. Through one auth path, the disconnect handler resolved no user and skipped deleting the provider's database row. The next session rebuild read that row and brought the host back. Cache invalidation was also keyed two different ways (acct_<userId> in one place, the bare user id in another), so it missed the real caches.
2. No tolerance. When a model was missing, the engine silently fell back to the first connection, sending a model id to a host that had never heard of it, and "model not configured" errors were surfaced as-is instead of failing over.

The fix derives the user from the session id in one function and uses it everywhere, makes the database mirror the session's fleet, awaits the Redis write on disconnect so a cold start cannot read a stale snapshot, prunes models whose host is gone, and treats "not configured" errors as a reason to fail over to a different host. The regression test is the exact repro: the cheapest frontier model on a host that rejects it, and the request still completes on the other host.

The same day a different stale path showed up: a session cached in Redis from before I filtered Baseten deployment UUIDs still carried them, and a chat surfaced "Function e598bfc1-…: Not found for account". Fresh connects already filtered these. The persisted-load path didn't (6ce59da). Every path that builds a fleet now filters the same way, and the failover stays as a second line of defense.

Answers that stopped mid-sentence

Groq's gpt-oss reasoning models default to 1,024 max_completion_tokens and spend them on reasoning first. On harder prompts the visible answer was cut off, and the response still looked like a success. The fix (f3f785a) sends an explicit token budget and treats finish_reason: "length" as a truncation, which triggers a retry or escalation. The same week I filtered Groq's prompt-guard and text-to-speech models out of chat fleets, because they were being picked for design prompts and returning probability scores instead of answers.

What I still don't claim

The README has no savings percentage, on purpose. Cost saved and quality depend on your providers and prompts, and the console reports both per request. Upstream LLMs and embeddings are faked in the API tests, so the live embedding client and the Redis cache backends are not exercised there. That is written down under known issues.

Takeaways

• Fake streaming is easy to ship and hard to notice. Test time to first token, not just the final text.
• Streaming and retries conflict. Decide up front what the user sees when you change your mind mid-answer.
• When one class of request fails and another doesn't, look at what is different about how they are routed, not at the requests.
• Stale state usually enters through the path you forgot. Every code path that builds the same object should share one constructor.
• A truncated response with a 200 status is still a failure. Check finish_reason.