Lina Xu
This project reduces cloud token usage through a hybrid local-and-cloud inference pipeline. Each incoming prompt is first classified by task category and evaluated by a locally fine-tuned DistilBERT router, which decides whether the task is easy or hard without consuming cloud tokens. Easy tasks are answered entirely by a local GGUF language model. Hard tasks are routed to a suitable Fireworks AI model, but the prompt is compressed before transmission. The optimizer creates several representations—including terse English, symbolic intermediate representation, and a compact multilingual form—and uses a local tokenizer to select the version with the fewest tokens while preserving important numbers and code details. The cloud model is instructed to return a concise answer sketch with a strict output-token limit. Finally, the local model expands that sketch into a polished response. This approach reserves cloud computation for difficult tasks and reduces both cloud input and output tokens while retaining useful answer quality.
Category tags:
For Innovators & Creators