Bifrost

Bifrost — Intelligent Hybrid AI Routing System Bifrost is a token-efficient routing agent built for AMD Developer Hackathon ACT II, Track 1, where the leaderboard rewards the fewest tokens spent above an accuracy gate. Rather than sending every prompt to an expensive remote LLM, Bifrost makes a cost-aware routing decision for each request through a multi-stage pipeline: 1.Zero-Token Router — deterministic prompts (simple math, NER, sentiment) are answered instantly with regex/heuristics, with zero LLM calls at all. 2.Semantic Cache — a thread-safe cache using both exact-match and NumPy cosine-similarity embedding search reuses prior answers, eliminating cost entirely on repeated or near-duplicate queries. 3.Adaptive Hybrid Classifier — for everything else, a routing engine weighs topic heuristics, prompt complexity, each model's historical success rate on that category, and the estimated cost of the remote call to decide: local model (gemma2:2b via Ollama) or remote model (llama-3.1-8b-instant via Fireworks AI)? 4.Two-Stage Quality Verification — a fast local heuristic first screens out weak/empty answers; if local model confidence is borderline, a lightweight remote evaluation gate double-checks for hallucination/correctness before the response is returned — protecting the accuracy floor that gates the whole leaderboard. 5.Observability Engine — structured JSON logs capture every routing decision, latency, complexity score, and token count, so the routing logic is auditable, not a black box. The system is built on FastAPI with asyncio, aiofiles, and asyncio.Lock for race-free concurrent request handling, ships with a Dockerfile/docker-compose for one-command deployment, and includes its own benchmarking suite (benchmark.py) that reports F1 accuracy, average/P95 latency, and remote token cost against an eval dataset — the exact axes the hackathon scores on.