Benchmark

PyTorch vs ONNXRuntime on the same weights

One pinned model revision (nateraw/vit-base-beans @ 41f85ace), one hashed test archive, one CPU thread. Each runtime is benchmarked in a fresh process, so cold load and peak RSS belong to that runtime alone. The numbers below are read from the report JSON and the parity block is recomputed from all 128 per-sample rows at build time.

Mean warm latency

26.4% faster on ONNX

724.5 ms vs 533.5 ms per image

Top-1 accuracy

96.88%

128 pinned images, the same count on both runtimes

Probability parity

0.000001

max |Δp| across every class, tolerance 0.01

Peak RSS

931 / 942 MB

PyTorch / ONNXRuntime, each measured in its own process

Cold load

102 / 609 ms

graph or weights construction only — first inference excluded

Label agreement

128 / 128

same predicted class for every test image

Latency

Percentiles use the nearest-rank method over 384 warm samples per runtime (3 passes × 128 images). Negative deltas mean ONNXRuntime is faster.

MetricPyTorchONNXRuntimeΔ vs PyTorch
Model load (cold)102.3608.6+494.9%
First pass, p50682.6508.9-25.4%
Warm mean724.5533.5-26.4%
Warm p50641.9509.8-20.6%
Warm p951059.8698.3-34.1%
Warm min541.1445.3-17.7%
Warm max3207.6858.6-73.2%

Memory: 931 MB PyTorch vs 942 MB ONNXRuntime peak RSS (+1.1%). Cold load: +494.9%. All values are milliseconds unless stated.

The cold-load row is not a like-for-like load comparison and is not the number to quote: PyTorch memory-maps safetensors, so its 102 ms measures file mapping rather than reading weights, while ONNXRuntime builds the graph and allocators up front. The honest first-request cost in either case is the first-pass p50 below it, which is also where lazy kernel initialisation lands.

Parity, recomputed

Rather than quote the report's parity summary, this page rebuilds it at build time from the 128 probability vectors each runtime ships: same filenames, same targets, then label agreement and absolute probability error per class. Those numbers match the committed summary exactly.

Images compared

128

Label matches

128

Correct (PyTorch)

124

Correct (ONNX)

124

Max |Δp|

0.000001

Mean |Δp|

0.000000

Probability hash match

differs

Within tolerance

yes

Method and limits

Reproduce

python scripts/benchmark-runtimes.py --runtime pytorch     --out /tmp/pt.json
python scripts/benchmark-runtimes.py --runtime onnxruntime --out /tmp/ort.json
python scripts/benchmark-runtimes.py --merge /tmp/pt.json /tmp/ort.json \
  --out public/benchmarks/runtime-benchmark.json

The script refuses to run if the test archive's SHA-256 differs from ca67b15d960d1e2f…, if the pinned model snapshot is missing, or if the two partial reports came from different environments. Source: leafsentry-ai. Environment: Python 3.12.14, PyTorch 2.13.0+cpu, ONNX 1.23.2, ONNXRuntime 1.32.0, AMD EPYC 9754 128-Core Processor.