OptiRoute is a hybrid, token-efficient AI routing agent built for the AMD Developer Hackathon: ACT II (Track 1). The project tackles the trade-off between the high costs/latency of remote APIs and the zero-token cost of local models. It implements an intelligent, 3-tier cascading routing engine that ensures simple tasks are solved locally for free, while complex queries are escalated to remote models using a speculative correction strategy. Tier 0 uses deterministic python rules (like SymPy) to solve algebraic and arithmetic equations in under 1ms with zero tokens. Tier 1 classifies incoming prompts using a pre-baked, low-latency TF-IDF classifier to route tasks to specialized system prompts. Tier 2 executes local inference using a CPU-optimized Gemma-2B-Instruct model. All local outputs are passed through programmatic validators (checking Python syntax, JSON schema correctness, etc.). If validation passes, the result is returned instantly. Tier 3 executes a speculative remote correction fallback using Fireworks AI. If the local output fails validation, the draft and query are sent to the remote model, which only outputs the corrected portion, saving up to 70% of remote output token costs. Optimized to run within a 4.82GB Docker container (well under the 10GB limit), OptiRouter avoids heavy framework overhead (like PyTorch), ensuring container startup times under 10 seconds. OptiRouter maximizes routing accuracy while providing production-grade cost-efficiency.
Category tags: