tokenfree

TokenFree is an intelligent, multi-tier AI routing agent built for the AMD Developer Hackathon: ACT II. Designed to maximize task accuracy while rigorously minimizing API token expenditure, it dynamically orchestrates workloads between a zero-cost local LLM and the powerful Fireworks AI platform. Core Architecture & Routing Strategy: The agent employs a fast, multi-stage decision pipeline to evaluate incoming prompts across eight diverse domains (including factual knowledge, coding, math, and logical reasoning). 1. Zero-Token NLP Classification: Every prompt is first evaluated by a regex-based classifier to determine its domain in microseconds, establishing a baseline confidence prior. 2. Logprob Confidence Scoring: The task is processed by a local `llama.cpp` instance. TokenFree extracts statistical features from the output logprobs (mean, variance, minimums) and passes them into a custom-trained CatBoost Machine Learning classifier. 3. The ML Gatekeeper: The CatBoost model grades the local LLM's response. - High Confidence: The local draft is returned immediately (0 Fireworks tokens). - Low Confidence: The task is instantly escalated to Fireworks AI. 4. Sandboxed Verification (Medium Confidence): For ambiguous cases in coding or math, the agent attempts to compile and run the local model's output in a secure, isolated Docker sandbox. If execution fails, the error is fed back to the local LLM for a self-correction attempt before defaulting to the API. Token Optimization When a task must be routed to Fireworks AI, TokenFree doesn't just pick the largest model. It uses the initial NLP classification to select the cheapest, most cost-effective model from the `ALLOWED_MODELS` list capable of handling that specific domain, ensuring peak efficiency without sacrificing output quality.