Hybrid Semantic Routing Agent

Large Language Models provide high-quality responses, but sending every request to a remote API increases latency, cost, and token consumption. This project explores a hybrid routing architecture that minimizes remote inference while maintaining response quality. The application combines three complementary strategies. First, every incoming query is converted into a sentence embedding using Sentence Transformers. These embeddings are indexed with FAISS to create a semantic cache capable of retrieving previously answered questions even when they are phrased differently. When a semantically similar query is found, the cached response is returned immediately, eliminating unnecessary inference and reducing response time. If no suitable cached response exists, the routing engine evaluates the complexity of the request using lightweight heuristics based on query length and keywords. Simple factual questions are handled by a lightweight local language model, while more complex reasoning tasks are routed to Fireworks AI. If the remote service is unavailable, the system automatically falls back to the local model, ensuring robustness and uninterrupted service. The project records cache statistics, estimated token savings, routing decisions, similarity scores, latency, and model selection for every request. Its modular architecture allows additional routing policies, multiple remote models, or persistent storage such as Redis or SQLite to be integrated with minimal changes. The primary objective is to demonstrate that intelligent routing combined with semantic caching can significantly reduce remote LLM usage while maintaining a simple, extensible, and production-oriented architecture.