Hyper Efficient 0 token Router

This project delivers a production-ready, highly isolated container infrastructure designed to execute local LLM inference tasks under extreme resource constraints. The primary objective was to deploy the Qwen2.5-1.5B-Instruct model in a locked 2-vCPU environment with a strict 4GB RAM ceiling, while ensuring the entire 19-question evaluation pipeline securely runs in under 10 minutes without hitting timeouts or memory leaks. Architectural Decisions & Optimization Strategy: Instead of relying on heavy, unoptimized frameworks like PyTorch or Transformers that require significant deployment overhead and memory footprint, this architecture leverages llama-cpp-python to process quantized GGUF parameters (q4_k_m). The processing loop was systematically optimized to cut execution latency by over 70% using two critical mechanics. First, prompt chunking evaluations were accelerated by configuring the engine to process token strings via an aggressive n_batch=512 boundary. Second, systemic prompt engineering was implemented to enforce strict, answer-only outputs. This effectively prevented the small model from generating excessive text tokens or getting stuck in long, repetitive logical reasoning loops that drain the evaluation timeline. Results & Performance Benchmarks: Comprehensive runtime logging and timing telemetry verified that the container successfully dropped aggregate 3-task execution latency from an initial 82 seconds down to a highly optimized 23.59 seconds. This brings the average response latency down to approximately 7.8 seconds per question, securing a projected total runtime of under 2.5 minutes for the entire 19-question testing sweep. The result is a rock-solid, production-grade deployment ready for real-time automated benchmark pipelines. (270 words • 1762 / 2000 characters)