FrugalRouter

FrugalRouter inverts the routing problem: instead of predicting the cheapest cloud model, it makes cloud calls unnecessary. Qwen3-1.7B and Qwen2.5-Coder-1.5B (Q4 GGUF, llama.cpp) are baked into the image and run CPU-only inside the grading container on 2 vCPUs — local inference costs zero leaderboard tokens by the rules. Small models fail raw prompting, so every category gets verification instead of trust: - Math: the model writes a Python program; we execute it and majority-vote the computed number. - Code gen: two independently sampled implementations run on model-generated inputs; agreement on observed behavior is required. - Code debug: the fix is behaviorally cross-checked against a from-scratch reference; the bug line is derived from the actual before/after diff. - Logic: entities/constraints extracted to JSON, all assignments brute-forced, answers shipped only when a unique solution is proven. - Summaries: programmatic sentence/word-count and truncation enforcement. Sentiment: grammar-constrained decoding (GBNF). NER: two-sample agreement, verbatim-presence filtering, deterministic span re-expansion. - Factual always escalates when budget allows (agreement can't verify facts; it's also the cheapest category to escalate). Every answer carries a confidence score from its verification outcome. The weakest escalate, cheapest-first, to the most token-frugal Fireworks model measured live (gpt-oss-120b, reasoning_effort=low: 137 tokens for a full word problem) under a hard 900-token global budget. Budget 0 = pure ZERO_API_CALLS mode. Robustness: startup tok/s probe with a speed governor, atomic incremental writes, per-task exception isolation, watchdog flush at 8.5 min — the container cannot TIMEOUT, crash, or emit invalid JSON. Measured on native x86 at 2 vCPU/4GB: 19-task run in 232s; 90% zero-escalation accuracy on an unseen 40-task suite, ~95% with escalation. Image 2.24 GB.