ONNX Model Import

1. Load and inspect a model

model <- onnx_load("path/to/model.onnx")

# Model summary (IR and opset version, node count, operators used)
onnx_summary(model)

# Input names and shapes (-1 for a dynamic dimension)
onnx_inputs(model)

2. Run inference

The simplest entry point is predict(). It takes an array in the shape the ONNX file declares, with samples in the first dimension, splits it into the batch size the model was loaded with, and returns the output in the declared shape too:

# Random image batch, [N, C, H, W] as declared — replace with real data
x <- array(runif(8 * 3 * 224 * 224), dim = c(8L, 3L, 224L, 224L))

probs <- predict(model, x)   # [8, 1000] for a classifier

# A 224 x 224 x 3 image (H, W, C) for a model declared as [1, 3, 224, 224]
img <- array(runif(224 * 224 * 3), dim = c(224L, 224L, 3L))

# Storage order of [1, 3, 224, 224] is (W, H, C, N): W fastest
input <- aperm(img, c(2, 1, 3))

result <- onnx_run(model, list(input_name = input))

cat("Output dim (storage order):", paste(dim(result[[1]]), collapse = " x "), "\n")

result <- onnx_run(model, list(
  input_ids      = as.numeric(tokens),
  attention_mask = rep(1, length(tokens))
))

3. GPU inference

By default ggmlR tries Vulkan first and falls back to CPU automatically. To force a specific backend:

# Check what's available
if (ggml_vulkan_available()) {
  cat("Vulkan GPU ready\n")
  ggml_vulkan_status()
}

# Load with explicit device
model_gpu <- onnx_load("path/to/model.onnx", device = "vulkan")
model_cpu <- onnx_load("path/to/model.onnx", device = "cpu")

Weights are transferred to the GPU once at load time. Repeated calls to onnx_run() do not re-transfer weights.

4. Dynamic input shapes

Some models accept variable-length inputs. Override shapes at load time:

model <- onnx_load("path/to/bert.onnx",
                    input_shapes = list(input_ids = c(1L, 128L)))

5. FP16 inference

Run in half-precision for faster GPU inference:

model_fp16 <- onnx_load("path/to/model.onnx", dtype = "f16")
result <- onnx_run(model_fp16, list(input = input))

6. Supported operators

ggmlR supports 90+ ONNX operators. Every operator runs on the CPU backend; the GPU column says where it runs when a Vulkan device is available:

• Yes: the operator runs on Vulkan.
• Yes*: runs on Vulkan, with the condition given in the notes; otherwise that node falls back to the CPU.
• Host: evaluated while the graph is built or between graph segments, so no device kernel is involved.
• CPU + GPU path: a CPU graph node whose kernel hands the work to a Vulkan kernel when one is available.

An operator not listed here, or a form of one the notes exclude, is not mapped: ggmlR prints unsupported op '<name>' (or the specific reason) to the console, and the model outputs that depend on that node are not built — onnx_run() returns them empty. See Debugging tips.

Fused pattern: RelPosBias2D (BoTNet) is not an ONNX operator. The importer recognises the relative-position-bias subgraph and replaces it with a single fused op that runs on both CPU and Vulkan.

7. Examples

For full working examples with real ONNX Zoo models see:

# GPU vs CPU benchmark across multiple models
# inst/examples/benchmark_onnx.R

# FP16 inference benchmark
# inst/examples/benchmark_onnx_fp16.R

# Run all supported ONNX Zoo models
# inst/examples/test_all_onnx.R

# BERT sentence similarity
# inst/examples/bert_similarity.R

8. Debugging tips

If a model’s outputs come back empty or wrong:

1. Read the console messages. A node that cannot be mapped does not raise an R error. ggmlR prints the node and the reason — unsupported op '<name>' for an operator missing from the table above, or a specific form such as mode='cubic' not supported for Resize or an Einsum equation other than bqc,bchw->bqhw — followed by the model outputs that were never built because of it and the first failing node. Those outputs come back empty from onnx_run(). onnx_summary(model)$ops lists the model’s operators.

2. Check the input layout. Wrong but plausible outputs most often mean the input reached the model with its axes scrambled — see the row-major note in section 2, or use predict().

3. Trace a divergence from ONNX Runtime. Set ONNX_INJECT_NODES=<name>[,<name>...] and ONNX_INJECT_DIR=<dir> to replace each named node’s output, right after it is computed, with the reference dump <dir>/<name>.ort.bin (flat little-endian f32). If a later disagreement disappears, it was the earlier one amplified; if it stays, the two are independent. Only f32 tensors are injected, and a size mismatch is refused rather than padded or truncated.