🤓 Latest Submissions
SmartRouter — cut your LLM bill
SmartRouter is an OpenAI-compatible proxy gateway that routes requests to the cheapest tier that can prove its correctness, escalating to paid remote models (like GLM-5.1 on Fireworks) only on failure. It resolves queries through a 5-stage verification-gated pipeline: 1. Tier 0 (0 tokens): 30+ deterministic solver families covering arithmetic, percentages, unit conversions, Roman numerals, statistics, equations, logic puzzles, sentiment, rule-based NER, and chitchat. 2. Tiers 0.5 and 0.75 (0 tokens): Code katas from templates and Python bugs fixed via AST patches, verified by executing locally in a subprocess against tests. 3. Tier 1 (0 tokens): Local Gemma 3 via Ollama drafts factual lookups, summaries, NER, and sentiment. It uses Chain-of-Thought prompting to boost reasoning. 4. Local Self-Repair: If local code fails tests, SmartRouter runs up to 3 turns of free conversational debugging on Gemma 3 to correct it, avoiding remote escalations. 5. Tier 2 (Paid): Fireworks models ranked cheapest-to-strongest. Token caps shrink as the budget depletes, degrading to local tiers instead of failing. Key Features: - Multi-Turn Logs and Caching: Feeds full history to models, enabling prompt caching to cut multi-turn token costs to near 0. - Multi-Language Heuristics: Skips Python compiler checks for Go, C++, JS, and Rust, using structural checks to keep them at the zero-token local tier. - Factual Freshness: Injects 2026 factual context (current leaders, year) directly into prompts. Common lookups are solved at Tier 0. - Semantic Caching: Caches paraphrases with a numeric guard blocking math query hits unless numbers match exactly. Results: Matches all-remote baseline's 100% accuracy using 0 Fireworks tokens (vs. 3,137 baseline tokens). Backed by 305 offline unit tests.
13 Jul 2026
