Running benchmarks

For choosing an execution model, start with performance guidance. This page covers reproducing measurements and interpreting their boundaries.

Benchmark suites

Build the library first. Run benchmarks sequentially to avoid competing for CPU and memory bandwidth.

Command	What it measures
pnpm bench	All Vitest benchmark suites
pnpm bench:f32	Basic f32 calls, new/reused output and resident chains
pnpm bench:core	Rust kernels without JS/binding/transfer costs
pnpm bench:mask	Classification, comparison, mask, select and clamp chain
pnpm bench:reductions	Reductions, scans and fixed combinations
pnpm bench:typed	f64, integer and conversion cases
pnpm bench:math	All advanced-math signatures against direct JS references
pnpm bench:pipelines	Separate/fused, ordinary/resident and full lifecycles
pnpm bench:workers	Synchronous/Worker latency, snapshots, responsiveness and memory
pnpm bench:async-task	Add the owned-input AsyncTask comparison prototype

The JS/TS suites use Vitest 5's bench fixture in *.bench.ts files. They run through a separate configuration, so pnpm test continues to run only correctness tests. The default command writes artifacts/benchmarks/results.json; individual suite commands write their own JSON files in the same directory. Override the destination with --outputFile <path>, or use Vitest's -t filter to select test groups:

pnpm bench:f32 -t 'length=1024$'
pnpm bench:pipelines --outputFile artifacts/comparison/pipelines.json
pnpm bench:math -t 'math f32/exp$'

The baseline runner still collects Rust JSON Lines into a report with environment metadata. Archived measurements describe their original runs; use the commands above to collect new results.

Read the reports

Vitest JSON reports store measurements in each assertion result's benchmarks field, including raw latency samples, throughput, mean, median (p50), MAD, percentiles and error estimates. meta.laneops records schema version 2, the sampling policy, environment and asset hashes; meta.observations contains capabilities, checksums and other suite-specific evidence. The f32 suite also records the packed package hash.

benchmarks/options.ts configures 20 warmup calls and 64 measured calls per case, with no additional time-based sampling. This bounds the full case matrix and raw report size. Increase those counts for longer measurements on a fixed machine. There is no automatic speed threshold. Benchmark files run sequentially, and the benchmark configuration disables Vite's module runner to avoid its import-access overhead when measuring the built package.

Latency samples now represent individual calls in milliseconds. Historical schema-v1 JS reports used seven batch averages in microseconds; they are not directly comparable, even after unit conversion. Generate new baselines when comparing changes. The Rust report and Worker diagnostic samples retain their existing measurement methods.

Keep units and measurement boundaries attached to a result. Resident timings exclude pre-upload and allocation unless explicitly labeled end-to-end. Full-lifecycle cases include arena creation, upload, allocation, execution, download and disposal.

JS comparisons must retain each f32 rounding step and integer wrap. A double- precision JS expression is not an equivalent f32 workload. Numerical accuracy is validated separately against independent references; a fast timing is not a proof of numerical correctness.

The specialized constant mul/add/clamp loop is distinct from the generic pipeline interpreter. General extra-vector plans can cost more than separate calls. The integer specialized loop also changes dispatch and low-level arithmetic cost; its gain cannot be attributed entirely to fewer passes.

Worker measurements

Measure submission and total latency separately, plus event-loop or animation-frame gaps. Reuse an executor so startup costs are not mixed into steady-state calls.

Reservations bound admitted payload, not process RSS. Wasm capacity, caller data, Worker baselines, caches and garbage-collection timing are outside the byte budget. The Worker benchmark records both budget counters and RSS. Vitest measures total latency; separate diagnostic runs record seven submission samples after two warmups, startup time, seven responsiveness observations per scenario, and twenty memory observations per backend. These appear in meta.observations.worker.

Optional AsyncTask experiment

pnpm bench:async-task builds experiments/async-task and enables its comparison rows in the Worker benchmark. Ordinary Worker and candidate checks do not build or load this addon. The prototype supports only f32 with fixed plans, without the production queue, budget or lifecycle behavior, and is never included in dist or the published package.

Existing evidence and limits

See pipeline measurements, Worker measurements and candidate results. These are dated observations on a particular development machine. There is no fixed speed gate on shared CI. Establish noise bounds and repeated fixed-machine measurements before adopting regression thresholds; keep numerical correctness gates independent.