Benchmark
PyTorch vs ONNXRuntime on the same weights
One pinned model revision (nateraw/vit-base-beans @ 41f85ace), one hashed test archive, one CPU thread. Each runtime is benchmarked in a fresh process, so cold load and peak RSS belong to that runtime alone. The numbers below are read from the report JSON and the parity block is recomputed from all 128 per-sample rows at build time.
Mean warm latency
26.4% faster on ONNX
724.5 ms vs 533.5 ms per image
Top-1 accuracy
96.88%
128 pinned images, the same count on both runtimes
Probability parity
0.000001
max |Δp| across every class, tolerance 0.01
Peak RSS
931 / 942 MB
PyTorch / ONNXRuntime, each measured in its own process
Cold load
102 / 609 ms
graph or weights construction only — first inference excluded
Label agreement
128 / 128
same predicted class for every test image
Latency
Percentiles use the nearest-rank method over 384 warm samples per runtime (3 passes × 128 images). Negative deltas mean ONNXRuntime is faster.
| Metric | PyTorch | ONNXRuntime | Δ vs PyTorch |
|---|---|---|---|
| Model load (cold) | 102.3 | 608.6 | +494.9% |
| First pass, p50 | 682.6 | 508.9 | -25.4% |
| Warm mean | 724.5 | 533.5 | -26.4% |
| Warm p50 | 641.9 | 509.8 | -20.6% |
| Warm p95 | 1059.8 | 698.3 | -34.1% |
| Warm min | 541.1 | 445.3 | -17.7% |
| Warm max | 3207.6 | 858.6 | -73.2% |
Memory: 931 MB PyTorch vs 942 MB ONNXRuntime peak RSS (+1.1%). Cold load: +494.9%. All values are milliseconds unless stated.
The cold-load row is not a like-for-like load comparison and is not the number to quote: PyTorch memory-maps safetensors, so its 102 ms measures file mapping rather than reading weights, while ONNXRuntime builds the graph and allocators up front. The honest first-request cost in either case is the first-pass p50 below it, which is also where lazy kernel initialisation lands.
Parity, recomputed
Rather than quote the report's parity summary, this page rebuilds it at build time from the 128 probability vectors each runtime ships: same filenames, same targets, then label agreement and absolute probability error per class. Those numbers match the committed summary exactly.
Images compared
128
Label matches
128
Correct (PyTorch)
124
Correct (ONNX)
124
Max |Δp|
0.000001
Mean |Δp|
0.000000
Probability hash match
differs
Within tolerance
yes
Method and limits
- No HTTP server, no upload handling, no quality gate. This is the inference step the service spends its time in; end-to-end request latency lives on the evaluation page.
- Both runtimes run under
OMP_NUM_THREADS=1with ONNXRuntime's intra-op and inter-op thread counts pinned to one, so this is per-core work rather than a scheduler artifact. - Top-1 accuracy is on the page because both runtimes have to agree on it. The selection policy, the calibration numbers, and the abstention sweep are evaluated elsewhere.
- This is a multi-tenant VPS, and absolute milliseconds move with CPU frequency scaling and noisy neighbours. The ordering and the parity evidence survive that. Every version it ran under is recorded in the report.
- Batch size one on CPU is the deployment this model actually serves, so a batched or GPU run would answer a question nobody asked of it.
Reproduce
python scripts/benchmark-runtimes.py --runtime pytorch --out /tmp/pt.json python scripts/benchmark-runtimes.py --runtime onnxruntime --out /tmp/ort.json python scripts/benchmark-runtimes.py --merge /tmp/pt.json /tmp/ort.json \ --out public/benchmarks/runtime-benchmark.json
The script refuses to run if the test archive's SHA-256 differs from ca67b15d960d1e2f…, if the pinned model snapshot is missing, or if the two partial reports came from different environments. Source: leafsentry-ai. Environment: Python 3.12.14, PyTorch 2.13.0+cpu, ONNX 1.23.2, ONNXRuntime 1.32.0, AMD EPYC 9754 128-Core Processor.