Skip to content

Learn the background

Veer sits where three fields meet: LLM inference systems, numerical computing, and programming-language theory. You do not need to be an expert in any of them. These seven short chapters build up exactly the ideas Veer relies on, with worked numbers.

ChapterYou will learnVeer feature it explains
1. How LLMs generate texttokens, the decode loop, prefill vs decodegen, max, token bounds str<=N
2. The KV cachewhat K and V are, the memory formula, why prefixes can be sharedCtx<M>, fork, the model declaration
3. Serving systemspaged attention, prefix caching, eviction, speculative decodingthe planner’s memory model, speculate
4. Agents and context editingagent loops, structured output, why editing history is expensivesplice, spans, json<T>, choice[...], tools
5. Number formatsfloating point, FP8, block-scaled FP4 (MXFP4, NVFP4), INT8Tensor<[..], fmt>, quant, kv: fp8_e4m3
6. Error analysisrounding error, how it grows through a matmul, worst-case boundsabs<=, err<=, @accuracy
7. Types, linearity, static analysistype soundness, linear types, abstract interpretationthe checker, the planner, the error codes

There is also a glossary for quick lookups.

Suggested path. Read 1 → 2 → 7 first (about 30 minutes). That is enough to follow the Tutorial up to chapter 6. Read 3 and 4 when you want to understand why the planner’s numbers matter, and 5 and 6 before the precision tutorial.

An LLM reads text as tokens and produces one token at a time. To avoid recomputing everything for every new token, it stores two vectors per token per layer, the keys and values, in GPU memory: the KV cache. For an 8-billion-parameter model that is about 128 KiB per token, so a few long conversations fill a GPU. Serving engines share cache between requests with the same prefix and evict cache they think is unused, but they have to guess, because the program that will use the cache is opaque to them. Veer makes that program transparent: contexts are linear values (each one is used exactly once, like Rust’s owned values), so the compiler can follow every cache from creation to release and prove peak memory, cost, and sharing. The same idea applies to numerics: each tensor’s type carries its number format and a proven error bound, so a too-aggressive quantization is a compile error, not a surprise in production.