Model Selection Framework

id: model-selection-2026-03-03 type: framework.evaluation domain: llm.production_selection status: [DESIGN] — framework designed, execution pending author: A.I. version: 1.0 audience: agent | human-operator scope: 4 models | 4 dimensions | ~40K calls/model

gap: public benchmarks miss production realities benchmarks: MMLU, HumanEval, Arena Elo — standardized but insufficient missing: ● latency_variance — matters more than mean at parallel scale ● retry_rates — multiply nominal costs 1.5-3x ● context_degradation — 4K vs 64K performance gap ● output_consistency — pipeline trust without human verification goal: measure operational variables, not headline scores

model provider context $/1M_in $/1M_out ────────────────── ─────────── ─────── ─────── ──────── Claude Opus 4.6 Anthropic 200K $15.00 $75.00 Claude Sonnet 4.6 Anthropic 200K $3.00 $15.00 Kimi 2.5 Fireworks 256K $0.80 $0.80 MiniMax 2.5 Fireworks 1M $0.26 $0.26 price_spread: input: Opus = 28x MiniMax | 58x MiniMax output: Opus = 288x MiniMax // whether premium translates to operational value = unknown

▸ dim.01 quality_distribution sample: 200 tasks x 10 samples x 4 models = 8,000 outputs tasks: code review, doc gen, data extraction, reasoning chains method: human rating 1-5 rubric, inter-rater reliability measures: mean, std_dev, percentile distributions observation: Kimi 2.5 shows more consistent formatting // anecdotal needs: rating rubric, ~40h human time, statistical analysis plan timeline: 4 weeks ▸ dim.02 latency_under_load concurrency: 1, 10, 50, 100+ measures: TTFT p50/p95/p99, TPS contexts: 1K, 4K, 16K, 32K, 64K input tokens origins: us-east-1, us-west-2, eu-west-1 includes: retry/backoff behavior, 429 responses, retry-after headers needs: load testing infra, consistent prompt templates, monitoring timeline: 2 weeks ▸ dim.03 retry_failure_analysis sample: 5,000 requests per model (automated pipeline) categories: first_pass_success | retry_success | max_retry_fail | timeout formula: true_cost = nominal_price x (1 + retry_rate) hypothesis: cheaper models may have higher retry rates, negating cost advantage observation: MiniMax produces structurally correct but semantically incomplete // anecdotal failure_modes: hallucination, formatting error, refusal, timeout needs: automated pipeline, structured logging, failure categorization timeline: 3 weeks ▸ dim.04 context_window_reality test_a: needle-in-haystack (500 tests) — fact at begin/mid/end positions test_b: reasoning complexity (200 tasks) at 4K/16K/32K/64K measures: retrieval accuracy, latency degradation question: do advertised specs match real usage? needs: synthetic long-context test sets, automated accuracy scoring timeline: 2 weeks

model role rationale ────────────────── ────────────────────── ───────────────────────────────── Kimi 2.5 default_prototyping lowest cost, acceptable quality Claude Sonnet 4.6 customer_facing safety training, edge-case risk Claude Opus 4.6 complex_reasoning failure cost exceeds price premium MiniMax 2.5 batch_processing price enables volume experimentation △ provisional: insufficient data to validate these assignments // framework above designed to replace assumptions with measurements

? Does Kimi's variance (if lower) translate to operational savings? ? At what volume does Opus premium become indefensible? ? Does MiniMax 1M context work reliably at full length? ? Fireworks latency penalty vs direct access? ? Anthropic direct reliability vs Fireworks-hosted for our geo?

experiment sample timeline ─────────────────── ────────────────── ──────── quality_distribution 8,000 outputs 4 weeks latency_under_load 10K req/model 2 weeks retry_analysis 5K req/model 3 weeks context_window 700 tests 2 weeks ─────────────────── ────────────────── ──────── total_per_model: ~40,000 API calls human_rating: ~40 hours full_timeline: 8-11 weeks (sequential) current_kimi_preference_basis: ● pricing transparency: $0.80/1M ● Fireworks hosting: fast TTFT, 99.9%+ uptime (brief monitoring) ● anecdotal consistency: less output variance in spot-checking △ these are hypotheses, not findings