Skip to content

Precision layer

namebits/elemmax (unscaled)error modelconstant
f32323.4e38exact storage0
bf16163.4e38per element: u·|x| + flooru = 2⁻⁸
f161665,504per elementu = 2⁻¹¹
fp8_e4m38448per elementu = 2⁻⁴
fp8_e5m2857,344per elementu = 2⁻³
int88— (scaled)c · tensor maxc = 1/254
mxfp44.25— (block 32, E8M0)c · block maxc = 0.25
mxfp4_oas4.25— (block 32, overflow-aware)c · block maxc = 0.25
nvfp44.5— (block 16, E4M3 + f32)c · block maxc = 17/96 ≈ 0.177

Derivations are in Number formats. Each constant is validated against a bit-exact simulator by property tests.

A tensor type carries (m, e): ‖exact‖∞ ≤ m, ‖computed − exact‖∞ ≤ e. With u = 2⁻²⁴ (f32 accumulation) and γ_K = Ku/(1 − Ku):

operationm’e’
quant(t, f) per-elementme + u_f(m + e) + floor_f, and requires m + e ≤ max_f (V0302)
quant(t, f) scaledme + c_f(m + e)
a + b, add(a, b)m_a + m_be_a + e_b + u(m’ + e_a + e_b)
a @ b, matmul (inner dim K)K m_a m_bK(e_a m_b + m_a e_b + e_a e_b) + γ_K K (m_a + e_a)(m_b + e_b)
scale(t, c), t * c|c| m|c| e + 2u|c|(m + e)
relu(t)me
silu(t)m1.1 e + 8u(m + e)
rand_uniform(shape, lo, hi, seed)max(|lo|, |hi|)0
zeros(shape)00

Shapes: matmul needs [m, k] @ [k, n]; add needs equal shapes, or a [1, n] row broadcast over [m, n] (V0300).

@accuracy(abs <= x) requires e ≤ x for the returned tensor. @accuracy(rel <= x) requires e/m ≤ x. Violations are V0301, reported with the inferred type.

Tensors carry computed values (f32, after simulated quantization) and an exact f64 shadow. measured_err(t) returns ‖computed − exact‖∞, and report(t) prints it beside the static bound.