track1 Agent

This project is a routing agent for Track 1: it reads a batch of tasks, decides for each one whether a small model on our own machine can handle it or whether it's worth spending real tokens on a bigger model through Fireworks, answers accordingly, and writes the results back out. The interesting part isn't the answering, it's the deciding. Our first instinct was to hand-pick the split: math and code are hard, send those to the big model, let the small one handle the rest for free. That's a guess dressed up as a decision, so instead of hard-coding it, we built a way to measure it. We wrote adversarial queries across all eight task categories, each with a ground truth checkable by code instead of eyeballed: math computed in Python, logic puzzles brute-forced for a single unique solution, code tasks graded by running the generated function against real tests. Then we ran our local model against every one and graded it for real. A few things fell out that we wouldn't have guessed: the local model handles some "hard" categories fine and stumbles on a few "easy" ones. From that labeled data we trained a small DistilBERT classifier that decides local-or-Fireworks before anything else happens, at zero token cost, so only answers that actually need the expensive model spend any. If those weights aren't available, the agent falls back to the original hand-picked list rather than failing. Getting this submittable surfaced real problems: a stale dependency pin, a stray newline in a credential that silently broke the Fireworks path, and a training bug where a bad train/test split scored the router at 10% accuracy before we traced it to whole categories missing from training entirely. None of that got caught by reading code and assuming it was fine, it got caught by actually building the image, running it for real, and trusting the numbers over the intuition.