.png&w=256&q=75)
🤝 Top Collaborators
.png&w=640&q=75)
🤓 Latest Submissions
Hybrid Token-Efficient Routing Agent
The Hybrid Token-Efficient Routing Agent minimizes LLM inference costs by routing every query through a three-tier pipeline: an in-memory semantic cache, a local Gemma 2B model (via llama-cpp-python, designed for ROCm on AMD Instinct GPUs), and a two-tier remote escalation path through the Fireworks AI API (Gemma 2 9B IT, then Qwen 3.7 Plus for harder cases). Rather than trusting the local model's self-reported confidence, the router computes a real confidence score from the mean log-probability across generated tokens, converting it to a 0–1 scale via exponentiation. Only when this score falls below a configurable threshold (default 0.75) does the query escalate to remote inference — keeping easy queries cheap and fast while reserving paid API calls for genuinely hard cases. In benchmark testing across 8 diverse tasks (factual QA, math, sentiment, summarization, NER, code debugging, logic, code generation), the agent resolved 37.5% of queries locally at zero API cost, escalating the remainder based on task difficulty and local confidence, completing the full batch in under 6 minutes — well within the hackathon's runtime constraints. A runtime deadline guard ensures the container never exceeds the time limit regardless of query mix, falling back gracefully rather than risking a timeout.
13 Jul 2026
