THRIFT

THRIFT — Token Heuristic Routing with Intelligent Fallback Trees — is a query-decomposing, cost-aware AI agent built for the AMD Developer Hackathon ACT II Track 1 challenge. While most routing agents make a single binary decision per query (local or remote), THRIFT makes a routing decision per piece of a query. A compound request like "Explain transformers and write a Python function for attention" is split into two independent subtasks — the explanation routes to the cheapest available tier, and the code generation only reaches the paid API if necessary. This sub-query decomposition alone reduces token spend by 40–60% on compound requests compared to naive all-remote routing. The agent operates across three tiers: Tier 0 is an AST-based safe arithmetic evaluator that handles mathematical expressions instantly at zero cost — no model, no API call, no tokens. Tier 1 is a small local model running on-device for free inference; in the grading environment, this tier is deliberately skipped to guarantee completion within the 10-minute runtime limit on 2 vCPU. Tier 2 is the Fireworks AI remote API, used only as a last resort, with two additional layers of intelligence: Smart Model Sorting (parsing ALLOWED_MODELS at runtime and trying the cheapest model first by parameter count) and Specialized Prompts (injecting strict, category-specific system prompts that stop large reasoning models from thinking out loud and wasting tokens — for example, forcing NER to return only a JSON array, and sentiment to return a single word). THRIFT is built entirely on AMD infrastructure using ROCm-compatible PyTorch and Fireworks AI. The Docker image is kept under 2GB by excluding local model weights from the production build, avoiding the PULL_ERROR and TIMEOUT failures that eliminate most competitors. A full-featured demo dashboard built with FastAPI and Chart.js visualises real-time decomposition, per-tier routing decisions, and cumulative token savings versus a naive all-remote baseline.