Code review with sharing, editing, and speculation
A reviewer that reads a document once, branches into a summary and a structured bug list that share the document’s KV cache, edits the document out of the cache with splice, and tries a patch inside a speculate transaction. It uses every context feature in about 30 lines.
File: examples/review.veer (included in every release download)
// Code review with a shared document prefix, structured output, an in-place// edit of the context (splice), and a transactional speculative patch.model llama = "meta-llama/Llama-3.1-8B-Instruct" { layers: 32, kv_heads: 8, head_dim: 128, kv: fp8_e4m3, block: 16, max_ctx: 131072}
prefix SYS: llama = "You are a careful senior code reviewer. Be concise."
schema Bugs { count: int, items: [str<=80; <=8],}
fn looks_valid(patch: str<=64) -> bool { return contains(patch, "fix") || len(patch) > 8}
@budget(kv = 2GiB, prefill = 12000)fn review(doc: str<=4096) -> str<=384 { let c = open SYS ++ "\n<document>\n" ++ #[doc] doc ++ "\n</document>\n" let (a, b) = fork c // both branches share SYS + doc (no copy) let (a, summary) = gen a ++ "Summarize the change:" max 128 drop a let (b, bugs) = gen b ++ "List the bugs as JSON:" max 256 as json<Bugs> // The document is no longer needed in b's context: replace it in place. let b = splice b at doc with "<document elided>" let (b, patch) = speculate b { let (b, p) = gen b ++ "Propose a one-line patch:" max 64 if looks_valid(p) { commit (b, p) } else { rollback "no patch" } } drop b return summary + " | patch: " + patch}
fn main() { let out = review("fn add(a: i32, b: i32) -> i32 { a - b } // adds two numbers") print(out)}Static plan
Section titled “Static plan”$ veer plan examples/review.veer --entry mainplan for `main` model llama kv=fp8_e4m3 · 64.00 KiB/token · page 16 tok · max_ctx 131,072 peak KV memory 301.00 MiB at line 26 live at peak b (372 tok, 64 shared), b (4,451 tok, 64 shared) resident prefixes 4.00 MiB prefill tokens ≤ 4,549 decode tokens ≤ 448 longest context ≤ 4,451 tokens (llama) fork line 21 shares 4,173 tokens, saves 262.00 MiB vs copying splice line 26 at `doc` re-prefills ≤ 291 tokens context DAG (first 17 of 17 ops): 0 L20 open -> c2 prefix SYS 1 L20 append c2 -> c4 12 tok 2 L20 append c4 -> c6 4,096 tok #[doc] 3 L20 append c6 -> c8 13 tok 4 L21 fork c8 -> c9,c10 5 L22 append c9 -> c12 21 tok 6 L22 gen c12 -> c14 128 tok max 128 7 L23 drop c14 -> 4,322 tok 8 L24 append c10 -> c16 22 tok 9 L24 gen c16 -> c18 256 tok max 256 10 L26 splice c18 -> c23 308 tok at doc (+291 re-prefill) 11 L27 speculate c23 -> c24 12 L28 append c24 -> c26 25 tok 13 L28 gen c26 -> c28 64 tok max 64 14 L29 commit c23 -> c28 15 L29 rollback -> c23 16 L31 drop c30 -> 461 tok reuse hints seg1 (prefix, 52 tok) used at steps [0, 1, 2, 3, 4, 5, 6, 8] seg3 (text, 12 tok) used at steps [1, 2, 3, 4, 5, 6, 8, 9] seg5 (text, 4096 tok) used at steps [2, 3, 4, 5, 6, 8, 9] seg7 (text, 13 tok) used at steps [3, 4, 5, 6, 8, 9] seg11 (text, 21 tok) used at steps [5, 6] seg15 (text, 22 tok) used at steps [8, 9]
warning[W0210]: splicing `doc` re-prefills up to 291 following tokens --> examples/review.veer:26:11 |26 | let b = splice b at doc with "<document elided>" | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ = note: engines with position-corrected KV editing (Leyline-style) can avoid this; splice near the end to avoid it everywhere = help: run `veer explain W0210` for detailsSimulated run
Section titled “Simulated run”$ veer run examples/review.veer --statspatch prefix model context model fix no bug plan context prefix result yes result no draft model maybe | patch: fast yes result maybe review result bug cache
── engine: sim ── peak KV 21.00 MiB planned ≤ 301.00 MiB ✓ prefill tokens 279 planned ≤ 4549 ✓ decode tokens 169 planned ≤ 448 ✓ reused tokens 327 gen calls 3forkshares the system prompt and the whole 4,096-token document between the two branches. The plan shows how much memory that saves.as json<Bugs>returns a typed record. The checker proves the smallest validBugsfits in 256 tokens.splice b at docremoves the document fromb’s cache once it’s no longer needed.W0210reports the worst-case re-prefill of what followed it.speculatekeeps the original context alive until the patch is accepted (commit) or rejected (rollback).@budget(kv = 2GiB, prefill = 12000)is checked at compile time.