.png&w=256&q=75)
🤓 Latest Submissions
CacheForge topology aware kv
CacheForge is a Rust KV-cache control plane for vLLM inference fleets on AMD Instinct MI300X. The problem: production LLM traffic shares long prompt prefixes - system prompts, RAG context, few-shot preambles, agent scaffolds. Behind a normal load balancer, every worker that hasn't seen a prefix recomputes its full, compute-bound prefill, and each worker caches its own copy, so an N-worker fleet stores N duplicates and burns HBM. Both wastes scale linearly with fleet size. CacheForge treats KV cache as a shared fleet resource. Its OpenAI-compatible router keeps an approximate radix map of recent prefixes and routes each request to the worker most likely to already hold its KV-cache prefix, with load-aware spill-over and KV-cache-pressure autoscaling instead of CPU. The differentiator is MI300X-topology-aware placement. Generic routers (NVIDIA Dynamo, llm-d, Mooncake) treat a GPU as one flat block. MI300X isn't: it is 8 XCDs with per-stack HBM, NPS NUMA partitions, and an xGMI mesh. CacheForge records which NPS partition, GPU, and node holds each prefix, so when a hot worker overloads, spills land where the next KV share (same partition, near-free) or transfer (one xGMI hop) is cheapest - a genuinely unclaimed, AMD-specific optimization. Honest by design, it consumes AMD's MORI-IO (KV transport) and kvcached (KV sharing) rather than reinventing them; the contribution is the missing control plane that ties routing, a KV-location directory, autoscaling, and a live GPU/cache dashboard into one deployable system. Validated on real MI300X (4x rocm/vllm, Qwen2.5-7B): TTFT -36%, throughput +26% vs round-robin, zero errors; topology-aware spill cut mean KV-move distance 39%.
13 Jul 2026
