Agent loop with KV editing
A three-step debugging agent whose tool returns up to 2,048 tokens per call. After each step the observation is replaced in the cache by a short note, in the style of Leyline’s KV directives, so the context grows by about 100 tokens per step instead of 2,000.
File: examples/agent_edit_history.veer (included in every release download)
// An agent loop whose tool outputs are large. After each step the bulky// observation is replaced by a short note *in the KV cache* (splice), so the// context stays small, the way Leyline-style KV editing proposes.model llama = "meta-llama/Llama-3.1-8B-Instruct" { layers: 32, kv_heads: 8, head_dim: 128, kv: bf16, max_ctx: 32768 }
prefix AGENT: llama = "You are a debugging agent. Use the search tool, then answer."
tool search(query: str<=128) -> str<=2048
@budget(kv = 1GiB, prefill = 20000)fn main() { let c = open AGENT ++ "\nTask: find why the cache test is flaky.\n" for step in 0..3 { let (c2, q) = gen c ++ "\nSearch query:" max 32 stop "\n" let obs = search(q) let c3 = c2 ++ "\nObservation: " ++ #[obs] obs let (c4, note) = gen c3 ++ "\nOne-line takeaway:" max 48 // Keep the takeaway, drop the 2 KB observation from the cache. c = splice c4 at obs with "(observation summarized)" print("step", step, "->", note) } let (c, answer) = gen c ++ "\nFinal answer:" max 128 print(answer) drop c}Static plan
Section titled “Static plan”$ veer plan examples/agent_edit_history.veer --entry mainplan for `main` model llama kv=bf16 · 128.00 KiB/token · page 16 tok · max_ctx 32,768 peak KV memory 346.00 MiB (budget 1.00 GiB ✓) at line 19 live at peak c (555 tok, 464 shared), c4 (2,579 tok, 464 shared) resident prefixes 8.00 MiB prefill tokens ≤ 6,674 (budget 20,000 ✓) decode tokens ≤ 368 longest context ≤ 2,579 tokens (llama) splice line 19 at `obs` re-prefills ≤ 67 tokens context DAG (first 24 of 26 ops): 0 L12 open -> c2 prefix AGENT 1 L12 append c2 -> c4 41 tok 2 L14 append c4 -> c6 14 tok 3 L14 gen c6 -> c8 32 tok max 32 4 L16 append c8 -> c10 14 tok 5 L16 append c10 -> c12 2,048 tok #[obs] 6 L17 append c12 -> c14 19 tok 7 L17 gen c14 -> c16 48 tok max 48 8 L19 splice c16 -> c20 91 tok at obs (+67 re-prefill) 9 L14 append c20 -> c22 14 tok 10 L14 gen c22 -> c24 32 tok max 32 11 L16 append c24 -> c26 14 tok 12 L16 append c26 -> c28 2,048 tok #[obs] 13 L17 append c28 -> c30 19 tok 14 L17 gen c30 -> c32 48 tok max 48 15 L19 splice c32 -> c36 91 tok at obs (+67 re-prefill) 16 L14 append c36 -> c38 14 tok 17 L14 gen c38 -> c40 32 tok max 32 18 L16 append c40 -> c42 14 tok 19 L16 append c42 -> c44 2,048 tok #[obs] 20 L17 append c44 -> c46 19 tok 21 L17 gen c46 -> c48 48 tok max 48 22 L19 splice c48 -> c52 91 tok at obs (+67 re-prefill) 23 L22 append c52 -> c54 14 tok reuse hints seg1 (prefix, 61 tok) used at steps [0, 1, 2, 3, 4, 5, 6, 7] seg3 (text, 41 tok) used at steps [1, 2, 3, 4, 5, 6, 7, 8] seg5 (text, 14 tok) used at steps [2, 3, 4, 5, 6, 7, 8, 9] seg7 (gen, 32 tok) used at steps [3, 4, 5, 6, 7, 8, 9, 10] seg9 (text, 14 tok) used at steps [4, 5, 6, 7, 8, 9, 10, 11] seg11 (text, 2048 tok) used at steps [5, 6, 7]
warning[W0210]: splicing `obs` re-prefills up to 67 following tokens --> examples/agent_edit_history.veer:19:9 |19 | c = splice c4 at obs with "(observation summarized)" | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ = note: engines with position-corrected KV editing (Leyline-style) can avoid this; splice near the end to avoid it everywhere = help: run `veer explain W0210` for detailsSimulated run
Section titled “Simulated run”$ veer run examples/agent_edit_history.veer --statsstep 0 -> bug memory memory reply context agent memorystep 1 -> no no no draft page review page yesstep 2 -> memory prefix check review reply plan summaryblock maybe shared fix yes reply agent draft reply bug agent no
── engine: sim ── peak KV 96.00 MiB planned ≤ 346.00 MiB ✓ prefill tokens 638 planned ≤ 6674 ✓ decode tokens 252 planned ≤ 368 ✓ reused tokens 61 gen calls 7- Planned with the splice: peak 346 MiB, longest context 2,579 tokens, prefill ≤ 6,674.
- The same program without the splice plans at peak 854 MiB, longest context 6,769 tokens, prefill ≤ 6,401. The splice trades about 270 tokens of re-prefill for 2.5× less memory, and both numbers are known before running.
- The label
obsis added and spliced away within each iteration, which is what the loop rule (V0104) requires. - Bind the tool to a real command with
--tool search='./search.sh'.