TOKENISER

Tokeniser is a full-stack agentic system that sits between your prompts and your LLMs. Every prompt passes through a 5-stage pipeline before it reaches a model, and every response is verified before it reaches the user. Input → Planner → Enhancer → Compressor → Router → LLM → Verifier → Output Average token savings ~40–60% Routing latency (local) < 250 ms Verification accuracy ~95% Jailbreak pattern coverage 25+ patterns Supported attack types Jailbreak · Prompt Injection · Data Exfiltration Local model - Gemma 3 4B (Ollama) Remote model - Gemini 2.5 Flash Cost per 1M tokens (remote) - $0.002 Pipeline Stages 1 — Planner Runs on the local Gemma model. Analyses the prompt and decides which downstream tools to activate: enhance, compress, smartoptimize. Returns a JSON action plan with a reason. 2 — Enhancer Fixes grammar, spelling, and clarity. Preserves the original meaning exactly. Powered by Gemini 2.5 Flash. 3 — Compressor Strips filler words, politeness, and redundancy. Reduces token count aggressively without changing intent. 4 — Smart Optimizer Full semantic rewrite using Gemini for maximum LLM performance. Used only when the planner decides it is needed. 5 — Router Analyses prompt complexity and routes to: Local — Gemma 3 4B via Ollama for simple/medium prompts (zero cost, low latency) Gemini — Gemini 2.5 Flash for complex prompts requiring cloud intelligence 6 — Verifier Evaluates the LLM response on 5 criteria: Correctness, Completeness, Clarity, Relevance, Grammar. Scores 0–10. If score < 8, automatically retries with Gemini. 7 — Risk Detector Pattern-based jailbreak scanner that runs on every prompt. Detects 3 attack categories across 25+ patterns. Returns risk level (LOW / MEDIUM / HIGH) and matched patterns.