Kartik - Adaptive Inference Under Constraints

This project translates recent advances in adaptive model routing, confidence-aware prediction and cost-sensitive inference into a practical system for reliable AI under strict resource constraints. Adaptive Inference Under Constraints investigates how language-model workloads can be executed when compute, memory, time, and external inference budgets are simultaneously limited. Instead of applying one computational strategy to every request, it treats inference as a sequence of decisions made under uncertainty. Development produced practical findings on cascade routing, task-sensitive resource allocation and deadline-aware execution. Experiments showed that inexpensive early routing decisions can often approximate more computationally expensive procedures, while selective verification preserves reliability for uncertain tasks. They also demonstrated that aggressive input compression can reduce answer quality, universal generation limits are poorly suited to diverse workloads, and routing must account for execution time as well as predicted accuracy. The system supports reasoning, factual knowledge, code, information extraction, classification, and summarization. Its execution policy adapts to each request while considering expected quality, computational cost, uncertainty, and remaining runtime. Reliability is treated as part of the inference problem, not as an operational afterthought. Bounded execution, controlled escalation, recovery mechanisms, and deadline-aware completion prevent a single expensive request from compromising an entire workload. The broader contribution is a practical demonstration that efficient AI is not achieved simply by choosing a smaller model. It emerges from allocating limited computation intelligently: deciding when local processing is sufficient, when additional capability is justified, and when further computation is no longer worth its cost.