Case Study — LLM FinOps
A benchmark-driven routing framework that sends the right task to the right model — automatically, at production scale.
01 — Problem
Most LLM-powered products start with a single model choice and never revisit it. Sonnet (or GPT-4o) becomes the default because it feels safe — and the cost difference stays invisible until the bill arrives.
At 100,000 requests/month across a product with five LLM features, routing every call to Sonnet costs roughly $216/month. The question I wanted to answer empirically: how much of that is for quality we're actually using?
| Dimension | Why it matters in production |
|---|---|
| Quality | Relevance score (LLM-as-judge) + hallucination rate — the two things routing can degrade |
| Cost per request | Input vs output token pricing differs 4–5×; task type changes the ratio significantly |
| Latency per step | Routing adds a classify step; must stay sub-ms to be invisible to end users |
| Token usage | Output-heavy tasks (code) cost much more than input-heavy ones (classification) |
| Error taxonomy | Timeout / context_length / quality_fail / hallucination — needed for SLA monitoring |
02 — Approach
Task type alone isn't enough. The same task can span a 10× cost range depending on how hard the instance is. Adding a complexity dimension makes routing precise.
| Task type | Production analog | % of volume | Default model |
|---|---|---|---|
| Classification | Intent routing, ticket triage, sentiment | ~30% | Haiku |
| Simple Extraction | Form parsing, entity extraction, data normalization | ~20% | Haiku |
| Summary | Call transcripts, support threads, news digests | ~25% | Haiku standard |
| Code Generation | IDE assistants, code review, test generation | ~10% | Haiku simple |
| Complex Reasoning | Root cause analysis, strategy recommendations, audits | ~15% | Sonnet |
The core logic is a simple two-parameter function. The classifier runs in <1ms (pure regex) and adds no perceptible latency.
def select_model(task_type: str, complexity: str) -> str: """Select the most cost-effective model for the task.""" # Classification, extraction — Haiku always (no reasoning chain needed) if task_type in ["classification", "simple_extraction"]: return "claude-haiku-4-5-20251001" # $0.80/1M input # Summarization — Haiku for standard docs; Sonnet only when complexity == "high" if task_type == "summary" and complexity != "high": return "claude-haiku-4-5-20251001" # meeting notes, reviews, news # Code generation — start cheap, upgrade on complexity if task_type == "code_generation": if complexity == "simple": return "claude-haiku-4-5-20251001" # boilerplate, one-liners return "claude-sonnet-4-6" # refactoring, API design # Complex reasoning — always Sonnet (hallucination risk too high on Haiku) if task_type == "complex_reasoning": return "claude-sonnet-4-6" return "claude-haiku-4-5-20251001" # default: start cheap
Captures every API call: token usage, latency, cost, and custom metadata
(task_type, complexity) per call.
Query back with GET /api/public/observations, groupby
metadata.task_type → monthly cost by feature automatically.
Reads Langfuse aggregated counts as input. Answers: which tasks route where, how much routing saves at actual production volume, what the quality threshold should be. Built for PM and engineering leadership — not ops.
03 — Findings
Benchmark: Haiku 4.5 vs Sonnet 4.6 · 5 task types · 2 difficulty levels · 3 runs each. Quality gap = Sonnet score − Haiku score.
| Task type | Difficulty | Haiku | Sonnet | Gap | Route to |
|---|---|---|---|---|---|
| Classification | Easy | 1.00 | 1.00 | 0.00 | Haiku |
| Classification | Hard | 0.90 | 0.90 | 0.00 | Haiku |
| Simple Extraction | Easy | 1.00 | 1.00 | 0.00 | Haiku |
| Simple Extraction | Hard | 0.85 | 0.90 | 0.05 | Haiku |
| Summary | Easy | 0.93 | 0.93 | 0.00 | Haiku |
| Summary | Hard | 0.83 | 0.93 | 0.10 | Haiku at default threshold |
| Code Generation | Easy | 1.00 | 1.00 | 0.00 | Haiku simple |
| Code Generation | Hard | 0.75 | 0.93 | 0.18 | Sonnet |
| Complex Reasoning | Easy | 0.75 | 0.75 | 0.00 | Haiku |
| Complex Reasoning | Hard | 0.60 | 0.75 | 0.15 | Sonnet |
Zero quality gap at both difficulty levels. These are the 50% of call volume where routing to Haiku costs nothing in quality — just $0.80/1M instead of $3.00/1M.
Hard summary (gap = 0.10) sits exactly at the default threshold boundary.
For customer-facing legal summaries, pass complexity="high" to force
Sonnet regardless of the global threshold.
Haiku's hallucination rate is too high on multi-step analysis. The quality gap holds at both difficulty levels — this is not a threshold decision, it's a capability boundary.
Latency advantage is consistent and compounds at scale. At 100k requests/month, routing to Haiku where appropriate also means faster p95 response times across the product.
04 — Impact
Routing is the biggest lever, but two others compound it. All three work independently and can be applied in sequence.
max_tokens per task type. Output costs
4–5× more than input — code generation and reasoning are the highest-impact targets.| Scenario | Monthly cost | vs. baseline | Notes |
|---|---|---|---|
| All Sonnet (baseline) | ~$216 | — | Default for most teams; no routing logic required |
| Smart routing (threshold = 0.10) | ~$65 | −70% | 80% traffic → Haiku; code/reasoning stay on Sonnet |
| All Haiku | ~$43 | −80% | Maximum savings; quality risk on hard code & reasoning |
05 — Monitoring
A routing table set from benchmark data is only as good as the quality signal feeding it back. Three metrics trigger a routing review.
| Metric | Watch for | Action |
|---|---|---|
| Relevance score by task × model |
Haiku avg drops below 0.75 on any task × complexity cell | Promote that cell to Sonnet; re-tag metadata.complexity in Langfuse |
| Hallucination rate by task type |
Rises above 5% on any task routed to Haiku | Investigate system prompt first; if persistent after prompt fix, upgrade model |
| Error breakdown quality_fail / hallucination |
Either error type trending up week-over-week | Check if task distribution shifted (new feature, new prompt template) |
# Pull quality scores by task_type × complexity from Langfuse # Run monthly — flag any cell where Haiku avg drops below threshold GET /api/public/observations?type=GENERATION &fromStartTime=2025-01-01T00:00:00Z &toStartTime=2025-01-31T23:59:59Z # Group client-side: df.groupby(["metadata.task_type", "metadata.complexity", "metadata.model"])[ "metadata.relevance_score" ].mean() # Promote to Sonnet where: # model == haiku AND avg_relevance < 0.75 AND task_type != "classification"
06 — Limitations
3 runs per task × difficulty cell is directional, not statistically significant. Shadow-test routing decisions on real production prompts before fully committing.
Real production prompts may have different distributions — especially for complex reasoning and legal summaries. The benchmark establishes the framework; production data validates it.
Keyword signals and prompt length are imperfect proxies. Passing complexity explicitly from the calling service (e.g., user uploaded a PDF flag) is more reliable than inferring it.
All benchmark calls are single-turn. Multi-turn conversations with accumulated context may behave differently — particularly for reasoning and code tasks where prior context matters.
Code & Documentation
Full benchmark pipeline, synthetic data generation, routing engine, and Streamlit dashboard available on GitHub. The README includes a quickstart to run the full analytics system locally.