Skip to content

Why Veer?

LLM programs (agents, RAG, tree search, multi-agent workflows) are increasingly limited by KV-cache memory and prefill cost, not by model quality. Yet these programs are written in Python against string-in/string-out APIs. The serving engine sees a stream of opaque requests and must infer structure:

  • which prefixes are shared (radix-tree matching),
  • which caches will be reused soon (LRU guesses, which KVFlow showed are often wrong),
  • whether an edit to the history is about to cost 30,000 tokens of re-prefill (it can’t know),
  • how much memory a workflow will need (it finds out by running out).

Meanwhile quantization (FP8, MXFP4, NVFP4) is chosen by trial and error, and an overflow or a too-aggressive format shows up as a quiet accuracy regression.

Veer makes the structure explicit and checkable:

  1. Contexts are linear values. The compiler sees every KV cache’s whole lifetime: creation, sharing (fork), growth (++, gen), editing (splice), transactions (speculate), and release (drop).
  2. A static planner turns that into proven bounds: peak KV bytes, prefill and decode tokens, context length, re-prefill cost, plus a context DAG with reuse hints for the engine.
  3. Precision types put the number format and a sound error bound in each tensor’s type, so @accuracy contracts and overflow are compile-time checks.
SGLang frontendLMQL / GuidanceDSPyTriton / TileLang / MojoVeer
LevelLLM programs (Python DSL)constrained promptingprompt optimizationGPU kernelsLLM programs (standalone language)
KV reuseautomatic at runtime (RadixAttention)engine-dependentnonen/aproven statically, explicit fork
Memory bound before runningnononon/ayes (@budget)
Context editingnononon/atyped spans + splice with cost
Speculation semanticsnononon/aspeculate / commit / rollback
Leaks / use-after-freepossiblepossiblen/an/acompile errors
Numeric accuracyn/an/an/amanualtypes with error bounds

Veer is complementary to serving engines: it runs on top of SGLang, vLLM, or llama.cpp through their OpenAI-compatible APIs, and its plan output is designed to feed engine-side policies (eviction, admission, disaggregated transfer).

See Research and motivation for the papers behind each design decision.