For the complete documentation index, see llms.txt. This page is also available as Markdown.

ONNX Runtime

Direct ONNX model inference via ONNX Runtime 1.10 — no conversion to SameDiff required

ONNX Runtime

The nd4j-onnxruntime module provides direct ONNX model inference via ONNX Runtime (ORT) 1.10. Unlike SameDiff ONNX import, there is no conversion of the model into a SameDiff graph. The ONNX model runs natively inside the ORT C++ runtime, which is accessed from Java via JavaCPP bindings.

This is the fastest path for ONNX inference when you do not need to inspect or modify the graph.


When to Use ONNX Runtime vs SameDiff Import

Requirement
Use ORT
Use SameDiff Import

Pure inference, minimum latency

Yes

No

Inspect or modify the graph in Java

No

Yes

Further training in Java

No

Yes

Op coverage: maximum ONNX compatibility

Yes (ORT is the reference)

Partial

Custom Java op integration

No

Yes

GPU execution

Yes (CUDA EP)

Via ND4J CUDA backend


Maven Dependency

<dependency>
    <groupId>org.nd4j</groupId>
    <artifactId>nd4j-onnxruntime</artifactId>
    <version>${dl4j.version}</version>
</dependency>

The module bundles native ORT binaries for Linux x86_64, Windows x86_64, and macOS x86_64. ARM platforms require building from source.


Basic Usage


Finding Input and Output Names

Use Python to inspect the ONNX model for input and output tensor names before running in Java:

Alternatively, use ONNX Runtime Python API:


OnnxRuntimeRunner Builder Options

The runner manages the ORT session lifecycle. Close it when done to release native resources:


Batched Inference

ONNX Runtime supports batched inference when the model has a dynamic batch dimension:


Data Types

ONNX Runtime accepts various tensor data types. ND4J INDArray types are mapped:

ND4J DataType
ONNX Tensor Type

FLOAT

FLOAT (float32)

DOUBLE

DOUBLE (float64)

INT32

INT32

INT64

INT64

BOOL

BOOL

FLOAT16

FLOAT16

Ensure your INDArray has the correct data type before passing to the runner. Cast if necessary:


Complete Example: Image Classification


Troubleshooting

Native library not found: verify that the nd4j-onnxruntime artifact was downloaded correctly and that the platform classifier matches your OS/architecture. On Linux, ensure libonnxruntime.so is accessible.

Wrong input name: the input name passed to the runner must exactly match the ONNX model's input tensor name. Use the Python inspection snippet to verify.

Data type mismatch: ORT is strict about tensor data types. If the model expects FLOAT and you provide DOUBLE, an error is thrown. Cast the array to the expected type before inference.

Session already closed: OnnxRuntimeRunner is AutoCloseable. After calling close() or exiting a try-with-resources block, the session is released and cannot be reused.

Last updated

Was this helpful?