ONNX Cheat Sheet

1Installtwo packages

pip install onnxruntime # CPU engine★
The inference engine. Python 3.11–3.14. Ships CPU + (on the GPU build) other providers.
pip install onnxruntime-gpu # CUDA/TensorRT build★
GPU wheel (Linux/Windows). Don't install both CPU and GPU packages in the same env.
pip install onnx # format/graph toolkit
Only needed to build/inspect/edit graphs or run the checker/shape-inference (cards 9–10). Not required just to run a model.
import onnxruntime as ort · import onnx
Conventional aliases. ort.__version__ / onnx.__version__ to confirm what's installed.

2InferenceSessionrun a model

sess = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])★
The core object: loads & compiles the model. Always pass providers= explicitly (see card 6).
outs = sess.run(None, {"input": x})★
Run inference. First arg = list of output names (None = all); second = {input_name: numpy_array}. Returns a list of numpy arrays.
outs = sess.run(["logits"], {"input": x, "mask": m})
Request specific outputs by name; feed multiple inputs. Inputs must be numpy arrays of the right dtype/shape.
# dtype mismatch -> "unexpected input data type" errorgotcha
Match the model's dtype exactly, e.g. x.astype(np.float32). int64 vs int32 and float32 vs float64 are common culprits.

3Inputs, outputs & IO bindingintrospect

for i in sess.get_inputs(): print(i.name, i.shape, i.type)★
Discover expected names/shapes/dtypes. sess.get_outputs() for outputs. Dynamic dims show as strings (e.g. 'batch').
name = sess.get_inputs()[0].name sess.run(None, {name: x})
Grab the input name programmatically instead of hardcoding it.
io = sess.io_binding() io.bind_input(...); io.bind_output(...); sess.run_with_iobinding(io)
IO binding keeps tensors on-device (GPU) across calls — avoids host↔device copies for max throughput.
ort.OrtValue.ortvalue_from_numpy(x, "cuda", 0)
Create device-resident tensors to bind directly.

4Export from PyTorchtorch.onnx

torch.onnx.export(model, (example_input,), "model.onnx", dynamo=True)★current
dynamo=True uses the modern TorchDynamo-based exporter (the recommended path in current PyTorch), capturing the graph via torch.export.
torch.onnx.export(model, args, "m.onnx", input_names=["input"], output_names=["logits"], dynamic_axes={"input":{0:"batch"}})★
Name the IO and mark dynamic axes (usually batch/sequence) so the model isn't locked to one shape.
model.eval() # before exporttip
Put the model in eval mode so dropout/batchnorm behave for inference. Provide a representative example_input.
op = torch.onnx.export(...); op.optimize(); op.save("m.onnx")
The dynamo exporter returns an ONNXProgram you can optimize/inspect before saving.

5Export sklearn / TF / othersconverters

from skl2onnx import to_onnx onx = to_onnx(clf, X[:1].astype(np.float32))★
skl2onnx converts scikit-learn pipelines/estimators. Pass a sample input to infer the initial types.
pip install tf2onnx python -m tf2onnx.convert --saved-model model_dir --output m.onnx
tf2onnx for TensorFlow/Keras SavedModels (CLI or API).
# transformers: optimum-onnx / ORTModelForXxx.from_pretrained(..., export=True)
HuggingFace Optimum exports & runs transformer models on ONNX Runtime with a familiar API.
# LightGBM/XGBoost -> onnxmltools; many frameworks have a *2onnx
Ecosystem of converters (onnxmltools, keras2onnx…) covers most trees/DL frameworks.

6Execution providersthe backend

ort.InferenceSession(m, providers=[ "CUDAExecutionProvider", "CPUExecutionProvider"])★
EPs are tried in order; ops unsupported by the first fall back to the next. Always list CPUExecutionProvider last as a safety net.
ort.get_available_providers()★
See what your build supports: TensorrtExecutionProvider, CUDA, DmlExecutionProvider (DirectML), CoreML, OpenVINO, ROCm, CPU.
providers=[("CUDAExecutionProvider", {"device_id": 0})]
Pass per-EP options as a (name, {options}) tuple — device id, memory limits, TensorRT cache dirs, etc.
# CUDAExecutionProvider missing? you installed onnxruntime (CPU)gotcha
GPU providers only exist in onnxruntime-gpu, and need matching CUDA/cuDNN. Check get_available_providers().

7Session options & perftune it

so = ort.SessionOptions() so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL★
Graph optimizations (fusions, constant folding) run by default; this forces the max level. Pass sess_options=so.
so.intra_op_num_threads = 4 · so.inter_op_num_threads = 1
Thread pools for within-op and across-op parallelism — tune for your CPU / latency vs. throughput.
so.optimized_model_filepath = "opt.onnx"
Save the optimized graph so later sessions skip the optimization step at load.
so.enable_profiling = True # -> a chrome-trace JSON
Profile per-op timings; open the resulting JSON in a trace viewer to find bottlenecks.

8Quantizationsmaller & faster

from onnxruntime.quantization import quantize_dynamic, QuantType quantize_dynamic("m.onnx", "m.int8.onnx", weight_type=QuantType.QInt8)★
Dynamic quantization: no calibration data, weights → INT8. Easiest ~4× size cut, good for transformers/RNNs on CPU.
from onnxruntime.quantization import quantize_static
Static quantization: needs a CalibrationDataReader over representative inputs; quantizes activations too for more speed.
# preprocess first: python -m onnxruntime.quantization.preprocess
Run shape-inference/optimization preprocessing before quantizing for best results.
# fp16: convert weights to half for GPU throughput
onnxconverter_common.float16.convert_float_to_float16 gives an easy FP16 win on GPU.

9Model structure & checkervalidate

m = onnx.load("model.onnx") onnx.checker.check_model(m)★
Load a ModelProto and validate it against the spec. onnx.save(m, "out.onnx") to write it back.
m.graph.node · m.graph.input · m.graph.output · m.graph.initializer
The structure: nodes (ops), graph input/output value-infos, and initializers (the weights).
print(onnx.helper.printable_graph(m.graph))
Human-readable dump of the whole graph — quick way to see ops & connections.
inferred = onnx.shape_inference.infer_shapes(m)★
Propagate tensor shapes/types through the graph — fills in intermediate value shapes for debugging & tooling.

10Graph helpers & toolsbuild / edit

from onnx import helper, TensorProto node = helper.make_node("Relu", ["x"], ["y"]) graph = helper.make_graph([node], "g", inputs, outputs)
The helper.make_* builders construct graphs by hand: make_node, make_graph, make_model, make_tensor.
from onnx import numpy_helper w = numpy_helper.to_array(m.graph.initializer[0])
Convert initializers ↔ numpy: to_array / from_array. Read or patch weights directly.
onnx.utils.extract_model("m.onnx", "sub.onnx", ["in"], ["mid"])
Slice a subgraph between named tensors — handy for debugging or splitting a model.
# onnxscript: write ONNX functions in a pythonic DSL
For authoring, ONNX Script lets you write ops in Python (it also powers the torch dynamo exporter). Netron visualizes any .onnx.

ONNX cheat sheet