ONNX Runtime — The AI Toolchain

ONNX Runtime plugin-ep-webgpu/v0.3.0

Mon, 24 Aug 2026 21:44:25 GMT

ONNX Runtime WebGPU Plugin EP 0.3.0 adds new operators, quantized KV cache, int64/uint8 type expansion, and configurable buffer-cache options. • Exposes WebGPU buffer-cache options through the JavaScript API, adds a robustness provider option, and makes the pending-dispatch limit configurable. • Adds device-free compile-only sessions for offline graph transformation without requiring a physical GPU. • Adds initial PagedAttention support, plus new operators: MRotaryEmbedding, GRU, DFT, PRelu, HardSwish, Trilu, Max, Min, and MatMulBnb4. • Adds 2-bit `GatherBlockQuantized` support and integrates ONNX 1.22 with opset 27. • Expands integer data-type coverage: `int64` for Add, Cast, Clip, Concat, Equal, Gather, Min, Max, ReduceSum, Reshape, Sub, Tile, and Where; `uint8` for Cast, Expand, Gather, and Reshape; `int32`/`uint32` for CumSum and Tile. • Adds quantized KV cache support and expands GQA with sliding-window cache, batched right-padded prompts, and FlashAttention graph capture for Gemma 4. • Adds deferred dispatch to parallelize shader compilation during cold start, reducing first-inference latency. • Adds Intel subgroup-matrix MatMul and FP16 Gemm kernels with batched-B and odd-N weight support and additional tiling optimizations. • Switches Softmax and non-flash Attention to an online softmax algorithm and adds tuned FlashAttention decode. • Removes the dynamic Duktape/Node WGSL-generation path in favor of statically generated shader sources. Breaking changes: • The dynamic Duktape/Node WGSL-generation path has been removed and replaced with generated shader sources, which may break build setups that relied on runtime WGSL generation.

ONNX Runtime v1.28.1

Tue, 18 Aug 2026 23:19:52 GMT

ONNX Runtime v1.28.1 adds device-free WebGPU compile-only sessions for offline model optimization without GPU hardware. • Adds device-free compile-only sessions in the WebGPU EP, enabling offline graph transformation and optimized-model serialization without access to GPU hardware.

ONNX Runtime v1.28.1

Tue, 18 Aug 2026 23:19:52 GMT

ONNX Runtime v1.28.1 adds device-free WebGPU compile-only sessions for offline graph transformation without GPU hardware. • Adds support for device-free compile-only sessions in the WebGPU EP, enabling offline graph transformation and optimized-model serialization without access to GPU hardware.

ONNX Runtime plugin-ep-cuda/v0.1.0

Mon, 17 Aug 2026 23:12:43 GMT

ONNX Runtime ships its first CUDA Plugin EP release, delivering CUDA execution as a separately packaged, independently installable provider. • Introduces the CUDA Plugin EP as a separately packaged execution provider, now the default CUDA provider implementation for ONNX Runtime, distributed as the `onnxruntime-ep-cuda` Python wheel and NuGet packages. • Adds provider options for user compute streams, copy behavior, EP-level unified streams, and external allocators, with compatibility back to ONNX Runtime 1.24.4 via version-gated callbacks. • Adds plugin profiling APIs and memory statistics in profiling output. • Adds CUDA Graph capture and replay, including user compute stream support, synchronization controls, and a kernel sync stream for scratch allocation. • Adds arena allocation, resource accounting, available-resource reporting, and IOBinding synchronization. • Expands operator coverage with cuDNN SDPA for MHA/GQA, quantized MoE kernels, block-quantized FP4 MatMul, native NVFP4 QMoE prefill on SM120, and NHWC hardening. • Adds cuDNN-free implementations of ArgMax, ArgMin, and ReduceSum. • Reduces runtime dependencies by making cuDNN and cuFFT optional and removing the `nvrtc` link requirement. • Adds Windows ARM64 packages and build coverage across CUDA 12.8/13, SM90, and SM120.

ONNX Runtime plugin-ep-cuda/v0.1.0

Mon, 17 Aug 2026 23:12:43 GMT

ONNX Runtime ships its first CUDA Plugin EP release, decoupling CUDA execution into a separately packaged provider with graph capture, profiling APIs, and expanded operator coverage. • Introduces the `onnxruntime-ep-cuda` Python wheel and NuGet packages, delivering CUDA execution as a separately installable plugin execution provider. • Adds provider options for user compute streams, copy behavior, EP-level unified streams, and external allocators, with compatibility back to ONNX Runtime 1.24.4 via version-gated callbacks. • Adds plugin profiling APIs and memory statistics in profiling output. • Adds CUDA Graph capture and replay, including user compute stream support, synchronization controls, and a kernel sync stream for scratch allocation. • Adds Windows ARM64 packages for the CUDA Plugin EP. • Adds arena allocation, resource accounting, available-resource reporting, and IOBinding synchronization. • Expands operator coverage with NHWC hardening, cuDNN SDPA for MHA/GQA, quantized MoE kernels, block-quantized FP4 MatMul, and native NVFP4 QMoE prefill on SM120. • Adds cuDNN-free implementations of ArgMax, ArgMin, and ReduceSum. • Reduces runtime dependencies by making cuDNN and cuFFT optional and removing the `nvrtc` link requirement. • Supports CUDA 12.8/13, SM90, and SM120 in build and CI coverage.

ONNX Runtime v1.29.0

Wed, 12 Aug 2026 06:15:37 GMT

ONNX Runtime v1.29.0 adds env-var thread-pool control, NVFP4 QMoE, PagedAttention with quantized KV cache, and broad WebGPU operator expansion. • Adds `ORT_INTRA_OP_NUM_THREADS` and `ORT_INTER_OP_NUM_THREADS` environment variables to set default intra-op and inter-op thread-pool sizes at process startup; explicit thread settings still take precedence, and `0` preserves machine-sized defaults. • Adds `ORT_DISABLE_TELEMETRY=1` environment variable to disable new POSIX telemetry (now available on Linux, macOS, Android, and iOS when built with telemetry enabled) before initialization. • Adds `--data_shape` flag to `onnxruntime_perf_test` for multi-shape profiling in a single run. • Adds `MatMulBlockQuantizedFp4Weight` and `MatMulBlockQuantizedFp8Weight` contrib operators on the CUDA EP, with block-scaled tensor-core/GEMV decode paths, packed FP4 decode, M-tiling, and folded W8A8 activation QDQ. • Adds `LinearAttentionGate`, `GatedRMSNorm`, and `GatedAdd` contrib operators on the CUDA EP. • Adds fused `MRotaryEmbedding` contrib operator for Qwen mRoPE variants, with ONNX-domain rotary embedding support. • Adds `PagedAttention` CUDA kernel with quantized KV cache, XQA decode, MLA, QK-Norm, and head-sink support. • Extends quantized KV-cache support with attention sinks, independent and per-channel scales, sliding-window cache, and a fused K/V dequantization launch on the CUDA EP. • Adds a cuDNN SDPA decode tier to the standard ONNX `Attention` CUDA kernel and enables cuDNN SDPA for the contrib `Attention` operator. • Adds `attention_bias` support to the GroupQueryAttention unfused path and `state_window` support to LinearAttention and CausalConvWithState for MTP on the CUDA EP. • Adds NVFP4 QMoE support on the CUDA EP, including native FP4xFP4 prefill on SM120, faster decode GEMV, fused routing/finalization paths, and reduced activation and weight-dequantization overhead. • Adds bfloat16 support to `AllReduce`, `AllGather`, and `AllToAll` collective operators on the CUDA EP. • Adds initial WebGPU `PagedAttention` implementation and moves Softmax and non-flash Attention to online algorithms on the WebGPU EP. • Adds DFT, HardSwish, Max/Min, Trilu, GRU, PRelu, MatMulBnb4, and MRotaryEmbedding operator support to the WebGPU EP. • Expands integer-type support across Clip, Reshape, Cast, Add, Tile, Concat, Expand, Gather, CumSum, Max, and Min on the WebGPU EP. • Adds and extends Intel subgroup-matrix MatMul/Gemm kernels on the WebGPU EP, including f16, batched-B, and odd-N support. • Adds weightless-model support for all initializer types and allows zero-input `EpContext` nodes, with maximum-shape inference wired into workspace estimation. • Adds device-free compile-only sessions on the WebGPU EP for offline graph transformation. • Expands CUDA plugin EP packaging to include a Windows ARM64 package with size options, with aligned architecture selections across Python, C API, TensorRT, Node.js, and plugin packages.

ONNX Runtime v1.29.0

Wed, 12 Aug 2026 06:15:37 GMT

ONNX Runtime v1.29.0 adds env-var thread-pool sizing, POSIX telemetry, PagedAttention, NVFP4 QMoE, and broad WebGPU operator expansion. • Adds `ORT_INTRA_OP_NUM_THREADS` and `ORT_INTER_OP_NUM_THREADS` environment variables to set default intra-op and inter-op thread-pool sizes at process startup; explicit thread settings still take precedence, and `0` preserves machine-sized defaults. • Adds `ORT_DISABLE_TELEMETRY=1` environment variable to disable non-Windows POSIX telemetry (now available on Linux, macOS, Android, and iOS when built with telemetry enabled) before initialization; WebAssembly remains telemetry-free. • Adds `--data_shape` flag to `onnxruntime_perf_test` to enable multi-shape profiling in a single run. • Adds `PagedAttention` contrib operator to the CUDA EP with quantized KV cache, XQA decode, MLA, QK-Norm, and head-sink support. • Adds `MatMulBlockQuantizedFp4Weight` and `MatMulBlockQuantizedFp8Weight` contrib operators to the CUDA EP with block-scaled tensor-core/GEMV decode paths, packed FP4 decode, and M-tiling. • Adds `LinearAttentionGate`, `GatedRMSNorm`, and `GatedAdd` contrib operators to the CUDA EP. • Adds NVFP4 QMoE support to the CUDA EP, including native FP4xFP4 prefill on SM120, faster decode GEMV, and fused routing/finalization paths. • Extends quantized KV-cache support in the CUDA EP with attention sinks, independent and per-channel scales, sliding-window cache, and a fused K/V dequantization launch. • Adds a cuDNN SDPA decode tier to the standard ONNX `Attention` CUDA kernel and enables cuDNN SDPA for contrib `Attention`. • Adds `attention_bias` support to the `GroupQueryAttention` unfused path and `state_window` support to `LinearAttention` and `CausalConvWithState` for MTP. • Adds bfloat16 support to `AllReduce`, `AllGather`, and `AllToAll` CUDA collective operators. • Adds ONNX-domain support for rotary embedding and a fused `MRotaryEmbedding` contrib operator for Qwen mRoPE variants. • Adds weightless-model support for all initializer types and allows zero-input `EpContext` nodes. • Adds WebGPU device-free compile-only sessions for offline graph transformation. • Adds initial WebGPU `PagedAttention` implementation and moves Softmax and non-flash Attention to online algorithms. • Adds DFT, HardSwish, Max/Min, Trilu, GRU, PRelu, MatMulBnb4, and MRotaryEmbedding operator support to the WebGPU EP. • Expands integer-type support across Clip, Reshape, Cast, Add, Tile, Concat, Expand, Gather, CumSum, Max, and Min in the WebGPU EP. • Adds and extends Intel subgroup-matrix MatMul/Gemm kernels in the WebGPU EP, including f16, batched-B, and odd-N support. • Expands CUDA plugin EP packaging to include a Windows ARM64 package with size options, aligned across Python, C API, TensorRT, Node.js, and plugin packages.

ONNX Runtime plugin-ep-webgpu/v0.2.1

Thu, 30 Jul 2026 01:36:24 GMT

WebGPU EP v0.2.1 adds 2-bit GatherBlockQuantized, Gemma 4 support, Qwen3 fusions, and expanded FlashAttention capabilities. • Adds 2-bit support to GatherBlockQuantized for more aggressive quantized-path compression on WebGPU. • Adds Opset 24 and KV-shared decoder layer support for Gemma 4 model paths. • Adds QKV and MLP fusions for Qwen3-style models, plus Q/K RMSNorm fusion into GroupQueryAttention. • Extends FlashAttention decode kernels to handle any sequence length, removing previous length constraints. • Adds QKV bias support for FlashAttention in MultiHeadAttention. • Enables dynamic `max_k_step` for NVIDIA hardware in the FlashAttention path. • Adds M4 Max-specific FlashAttention optimizations. • Generalizes the FlashAttention prefill shared-memory path for broader hardware coverage. • GroupQueryAttention now supports optional present-key/value outputs. • Adds LinearAttention subgroup optimizations and larger `tile_v` with subgroup support. • Introduces per-graph buffer manager and session-level buffer pool for graph-capture reuse.

ONNX Runtime plugin-ep-webgpu/v0.2.1

Thu, 30 Jul 2026 01:36:24 GMT

WebGPU EP v0.2.1 adds 2-bit GatherBlockQuantized, Gemma 4 / Qwen3 fusions, and extended FlashAttention kernels. • Extends FlashAttention decode kernels to support any sequence length, enabling longer-context LLM inference on the WebGPU EP. • Adds QKV bias support to FlashAttention in MultiHeadAttention. • Enables dynamic `max_k_step` for NVIDIA GPUs in the FlashAttention path. • Generalizes the FlashAttention prefill shared-memory path for broader hardware coverage. • Lands M4 Max-specific FlashAttention optimization for Apple Silicon. • Adds QKV and MLP op fusions for Qwen3-style models. • Adds Q/K RMSNorm fusion into GroupQueryAttention for Qwen3-style models. • Adds Opset 24 and KV-shared decoder layer support for Gemma 4 models. • GroupQueryAttention now supports optional present-key/value outputs. • GatherBlockQuantized gains 2-bit quantization support, expanding low-bit inference options. • Adds LinearAttention subgroup optimizations and larger `tile_v` with subgroup support. • Introduces per-graph buffer manager and session-level buffer pool for graph-capture reuse.

ONNX Runtime v1.28.0

Sat, 25 Jul 2026 05:27:07 GMT

ONNX Runtime v1.28.0 adds CudaQuantizer, OrtModelPackageApi experimental C API, Linux NPU sysfs discovery, and CUDA EP footprint reductions. • Adds `OrtErrorCode::ORT_DEVICE_RESET` to the `OrtErrorCode` enum, with documentation single-sourced so `StatusCode` stays in sync. • Adds `model_external_initializers_file_folder_path` session option, now honored for file-path model loads. • Adds `CudaQuantizer` to `onnxruntime.quantization` for GPU-side quantization workflows. • Adds a Python API for `HOST_ACCESSIBLE` `OrtValue` allocation. • Introduces experimental `OrtModelPackageApi` in the C/C++ experimental API surface, including authoring tools, schema versioning, and folding `external_data` into session options — note this API may change in future releases. • CUDA EP: `cuDNN` and `cuFFT` are now optional at runtime and `nvrtc` is no longer linked, significantly reducing the required CUDA redistributable footprint. • CUDA plugin EP gains support for user compute stream with CUDA graph, kernel sync stream exposed for scratch allocation, and Windows ARM64 packages. • Adds Linux NPU discovery through sysfs accel devices. • Adds an API to select the best compiled-model compatibility info from candidate strings. • Adds crypto support: applications can supply I/O callbacks to an EP, with callback and fallback helpers. • Implements name-based partitioning in the plugin EP framework. • Registers `Flatten` as a Direct8Bit op in the Python QDQ static quantizer. • Adds Float16, BFloat16, and Float8 support in the `TensorArray` custom op. • Adds memory statistics to profiling output. • Adds EP version logging on inference failure, in the `EpDeviceUsage` event, and ORT version logging. • User-supplied external initializers are now used in place when already on the planned device, avoiding unnecessary copies. • Upgrades to ONNX 1.22.0 and protobuf 6.33.5. • NPM packages are now published from the CUDA 13 pipeline. • WebGPU plugin EP version bumped to 0.3.0. Breaking changes: • `CUDA_QUANT_PREPROCESS` is now off by default; workflows relying on the previous default-on behavior must explicitly enable it. • The TensorRT fused causal attention kernels were removed from the CUDA EP; models depending on them will need an alternative path. • The dynamic WGSL generator (duktape/Node) path was removed; the Python `wgsl-gen` implementation is now the only supported path. • `OrtModelPackageApi` now lives in the experimental C API and may change in future releases; callers should treat its signatures as unstable. • SkipLayerNorm strict mode is deprecated and may be removed in a future release.

ONNX Runtime v1.28.0

Sat, 25 Jul 2026 05:27:07 GMT

ONNX Runtime v1.28.0 adds CudaQuantizer, HOST_ACCESSIBLE OrtValue Python API, ORT_DEVICE_RESET error code, and CUDA plugin EP gains user compute stream with CUDA graph support. • Adds `CudaQuantizer` to `onnxruntime.quantization` for GPU-side quantization workflows. • Adds a Python API for `HOST_ACCESSIBLE` `OrtValue` allocation, enabling CPU-accessible device memory from Python. • Adds `OrtErrorCode::ORT_DEVICE_RESET` to the C API error code surface, with single-sourced `StatusCode` sync. • Adds `model_external_initializers_file_folder_path` session option, now honored for file-path model loads. • CUDA plugin EP now supports user-supplied compute streams with CUDA graph and exposes a kernel sync stream for scratch allocation. • Adds crypto support to plugin EPs: applications can supply I/O callbacks with callback and fallback helpers. • Adds Linux NPU discovery through sysfs accel devices. • Implements name-based graph partitioning for plugin EPs. • Adds memory statistics to profiling output. • Adds EP version logging on inference failure via the `EpDeviceUsage` event. • Registers `Flatten` as a Direct8Bit op in the Python QDQ static quantizer. • Adds Float16, BFloat16, and Float8 support in the `TensorArray` custom op. • cuDNN and cuFFT are now optional at runtime for the CUDA EP, and `nvrtc` is no longer linked, significantly reducing the CUDA redistributable footprint. • WebGPU plugin EP bumped to version 0.3.0. • NPM packages are now published from the CUDA 13 pipeline. • Adds an API to select the best compiled-model compatibility info from candidate strings. • User-supplied external initializers are now used in-place when already on the planned device, avoiding unnecessary copies. Breaking changes: • `CUDA_QUANT_PREPROCESS` is now off by default; workflows that relied on the previous default-on behavior must explicitly enable it. • The TensorRT fused causal attention kernels were removed from the CUDA EP; models relying on them will no longer use that path. • The dynamic WGSL generator (duktape/Node) path was removed; WebGPU users must use the Python `wgsl-gen` implementation instead. • SkipLayerNorm strict mode is deprecated and its behavior may change in future releases.

ONNX Runtime v1.27.0

Fri, 19 Jun 2026 21:11:07 GMT

ONNX Runtime v1.27.0 adds FLOAT8E8M0, quantized KV-cache, calibration cache, new EP plugin APIs, and broad CoreML/CUDA op coverage. • Adds OrtEp::OnSessionInitializationEnd() callback to the Execution Provider Plugin API for post-initialization hooks. • Adds plugin EP session-options getters to the Execution Provider Plugin API. • Adds CUDA Plugin EP provider options for streams and external allocators. • Adds `kOrtEpDevice_EpMetadataKey_OSDriverVersion` metadata key with example and docs for EP device metadata. • Adds calibration cache support to `quantize_static` in the quantization tooling. • Adds `ActivationRestrictedAsymmetric` quantization option. • Adds opset-21 `block_size` attribute support to QDQ quantization. • Adds PEP 561 `py.typed` marker to the `onnxruntime` Python package, enabling type-checker support. • Makes `sympy` an optional runtime dependency in the Python package. • Adds zero-copy I/O for plugin EPs with HOST_ACCESSIBLE memory. • Adds FLOAT8E8M0 datatype support to ONNX Runtime core, including CPU Cast ops. • Adds support for ONNX overloaded functions (IR v10+). • Adds CPU fallback for `FusedAdam` optimizer in ORT Training. • Adds chunked dequant+GEMM for MatMulNBits on CUDA EP to reduce peak GPU memory. • Adds `ConvTranspose-22` support on the CUDA EP. • Fills CUDA opset gaps for LSTM, RNN, Reshape, Cast, Round/Equal, ReduceMax/ReduceMin, Sin/Cos, and Random* ops. • Adds LpNormalization support on the CUDA EP. • Adds LSTM support on the WebGPU EP. • Adds QKV and MLP layer fusions for Qwen3-style models on the WebGPU EP. • Adds QKV bias support in FlashAttention for MultiHeadAttention on the WebGPU EP. • Adds shader dump-to-file environment variable for WebGPU EP debugging. • Adds opset-24 and KV-shared decoder support (Gemma 4) on the WebGPU EP. • Adds `FusedConv`, `Identity`, `Ceil`, `Tile`, Cast(bool), `Sin`, `Cos`, and `GatherND` op support on the CoreML EP. • Adds support for pre-opset-13 `Split` (attribute path) and scalar `Gather` indices on the CoreML EP. • Adds NHWC convolution path in MLAS to reduce transpose overhead. • Adds CPU QMoE 2-bit support with LUT GEMM fast path. • Adds quantized KV-cache support for CPU GroupQueryAttention with SIMD optimizations and tiled compute. • Adds RVV-optimized NCHWc convolution/pooling and LLM operators for RISC-V targets. • Parallelizes CPU ScatterElements and optimizes MatMulNBits 2-bit float-zero-point path. • Adds optional bias support in MatMulNBits CPU LUT GEMM path. • Adds DiT attention fusion for F5-TTS and diffusion transformer models. • Adds support for optional `present_key`/`present_value` outputs in GQA and Gemma4. • Adds diagnostics for DML failure paths. • Adds per-graph buffer manager for multi-graph capture on the WebGPU EP. Breaking changes: • CUDA 12 package files are now explicitly named as such — existing references to unversioned CUDA package names will need to be updated.

ONNX Runtime plugin-ep-webgpu/v0.1.0

Fri, 29 May 2026 17:53:51 GMT

ONNX Runtime ships the WebGPU Execution Provider as a standalone plugin (`onnxruntime_providers_webgpu`) compatible with ORT 1.24.4+. • Introduces the `onnxruntime_providers_webgpu` plugin EP, registered into an existing ONNX Runtime 1.24.4+ installation at runtime via onnxruntime.register_execution_provider_library(...) (Python) or OrtEnv.RegisterExecutionProviderLibrary(...) (.NET) — no core ORT package swap required. • Adds the `onnxruntime-ep-webgpu` Python wheel (PyPI: `onnxruntime-ep-webgpu`) for installing the WebGPU EP alongside the `onnxruntime` package. • Adds the `Microsoft.ML.OnnxRuntime.EP.WebGpu` NuGet package for .NET, referenced alongside `Microsoft.ML.OnnxRuntime`. • Ships cross-platform native binaries for Windows x64/arm64 (bundled with `dxil.dll` and `dxcompiler.dll`), Linux x64, and macOS arm64. • Provides WebGPU kernels for transformer and LLM contrib ops: `Attention`, `MultiHeadAttention`, `GroupQueryAttention`, `RotaryEmbedding`, `SkipLayerNormalization`, `SkipSimplifiedLayerNormalization`, `SimplifiedLayerNormalization`, `BiasAdd`, `BiasGelu`, `BiasSplitGelu`, `FastGelu`, `Gelu`, `QuickGelu`, `CausalConvWithState`, and `LinearAttention`. • Supports quantized inference via `MatMulNBits` (with DP4A and subgroup-matrix acceleration paths), `DequantizeLinear`, `GatherBlockQuantized`, and `QMoE` (quantized Mixture-of-Experts) on WebGPU. • Includes a `FlashAttention` kernel and Intel-optimized `MatMul`/`Gemm` code paths for improved throughput on Intel GPUs. • Adds program caching to amortize WebGPU shader compilation costs across inference runs. • Adds optional PIX frame capture and WebGPU profiler integration for GPU performance investigation.

ONNX Runtime v1.26.0

Fri, 08 May 2026 19:24:39 GMT

ONNX Runtime v1.26.0 adds RISC-V Vector CPU support, CUDA plugin graph/profiling APIs, WebGPU GridSample, and CoreML new ops. • Adds optional memory mapping for `.ort` model loads, reducing load-time I/O overhead. • Adds RISC-V Vector (RVV) support for the CPU Execution Provider, enabling inference on RVV-capable hardware. • Adds graph capture/replay support and a profiling API to the CUDA plugin EP. • Adds sync support for IOBinding in the CUDA plugin EP. • Adds `GridSample` operator support to the WebGPU EP. • Extends WebGPU Split-K support to batch size > 1, improving batched matrix-multiply throughput. • Adds `HardSigmoid` and `QuickGelu` operator support to the CoreML EP. • Adds `Pad` reflect mode support to the CoreML EP. • Adds CUDA `TopK` support for INT8, INT16, and UINT8 data types. • Adds CUDA `LabelEncoder` support for numeric types. • Adds CUDA `QuantizeLinear`/`DequantizeLinear` opset 25 support. • Extends CUDA `Transpose` support to opsets 23–25. • Adds FP32 QK accumulation path for unfused GQA attention on CUDA. • Adds FP16 Gelu enablement in MLAS. • Adds Arm64 BF16 fast-math convolution kernels for NCHW/NCHWc paths in MLAS. • Upgrades OpenVINO EP for the 1.26.0 development release. • Replaces unrestricted Python `setattr` configuration with an allowlist, hardening the session configuration surface. Breaking changes: • The WebNN pool2d property `roundingType` is renamed to `outputShapeRounding`; any code or config referencing `roundingType` will break. • Support for CUDA 12 will be removed in v1.27.0; the published CUDA 13 package is `onnxruntime---gpu_cuda13-.`.

ONNX Runtime v1.25.1

Mon, 27 Apr 2026 21:46:20 GMT

ONNX Runtime 1.25.1 adds LinearAttention, CausalConvState, RotaryEmbedding, and RMSNorm operators with Qwen3.5 WebGPU support. • Adds `LinearAttention` and `CausalConvState` operators to support Qwen3.5 model inference. • Adds `RotaryEmbedding` (RotEMB) and `RMSNorm` operators. • Adds Linear Attention signature support. • Updates `Reshape` to opset 25 and `Transpose` to opset 24. • Enables Qwen3.5 model execution on the WebGPU execution provider. • Optimizes QMoE 1-token decode path on WebGPU via fused operations to reduce GPU dispatches.

ONNX Runtime v1.25.0

Mon, 20 Apr 2026 18:25:24 GMT

ONNX Runtime 1.25 ships a CUDA Plugin EP, new Attention opsets 23/24 on CUDA, per-session profiling in RunOptions, and expanded WebGPU op coverage. • Adds `enable_profiling` to `RunOptions` for per-session profiling control without a global session option. • Adds `KernelInfo` string-array attribute APIs for C and C++, enabling custom ops to read string-array attributes via the C and C++ EP API. • Adds `OpKernelInfo::GetConfigOptions` and LoggingManager::HasDefaultLogger() to the EP API adapter. • Adds `OrtModel` input support to the Compile API, allowing compiled model workflows to accept `OrtModel` objects directly. • Adds a session config key to create weightless `EPContext` models during compilation. • Introduces the CUDA Plugin EP — a new core implementation allowing third-party CUDA-backed execution providers to be delivered as dynamically loaded plugins without rebuilding ORT. • Adds BFC-style arena and CUDA mempool allocators to the CUDA Plugin EP for stream-aware memory management. • Adds a Plugin EP Sync API for synchronous execution. • Adds Plugin EP event profiling APIs. • Adds Plugin EP APIs to retrieve ONNX operator schemas. • Adds annotation-based graph partitioning with resource accounting to the EP plugin API. • Adds per-session thread pool work callbacks API to core APIs. • Introduces Model Package support (preview): automatically selects compiled EPContext model variants from a packaged collection based on EP, device, and hardware constraints. • Implements Attention opset 23 on CUDA with GQA, boolean masks, softcap, and softmax precision control. • Implements Attention opset 24 on CUDA (disjoint from contrib op) and nonpad KV sequence length on CPU. • Adds `TensorScatter-24` op support for CPU and CUDA. • Adds `DeformConv` op support for CPU and CUDA. • Adds `LpNormalization-22` op support. • Fills CUDA opset gaps: `Flatten`, `Identity`, `If`, `Loop`, `Scan`, `ConstantOfShape`, `Size` (opsets 21/23); `GlobalAveragePool`/`GlobalMaxPool` (→22); `Shape` (→25); `Squeeze`/`Unsqueeze` (→25); `TopK` (→24, BF16); `GRU` (→22); `Pad` (→25, wrap mode); `Resize` v19; `RoiAlign` v16/v22. • Adds GQA with XQA and quantized KV cache to the NVIDIA CUDA EP, including FP8 (E4M3) KV cache support. • Adds CUDA graph capture compatibility for LLM ops and pre-compiled paths in the CUDA EP. • Adds volumetric (3D) `GridSample` support to the CUDA EP. • Adds optional `router_weights` input to the `QMoE` contrib op on CUDA. • Adds D3D12 external resource import support to the NVIDIA TensorRT RTX EP. • Adds `TopK`, `Softplus`, and `Identity` op support to the WebGPU EP. • Adds `Conv3D` and `LpNorm` op support to the WebGPU EP. • Adds int64/bool type support for `Range`, `Expand`, `Flatten`, `Gather`, and `Unsqueeze` in the WebGPU EP. • Optimizes WebGPU Gemm/MatMul using subgroup features. • Adds 2-bit zero-point support and higher K-parallelism to `MatMulNBits` in the WebGPU EP. • Adds `head_sink` support and configurable multi-rotary cache concat offset to Flash Attention in the WebGPU EP. • Makes WebGPU EP compatible with the EP plugin API. • Upgrades bundled ONNX to 1.21.0. Breaking changes: • Building ORT from source now requires C++20; minimum toolchains are MSVC 19.29+, GCC 10+, Clang 10+. Prebuilt-package users are unaffected. • CUDA minimum version raised to 12.0 — CUDA 11.x is no longer supported. Users pinned to CUDA 11.x must stay on ORT 1.24.x or upgrade their CUDA toolkit and driver. • ArmNN EP has been removed. Any `--use_armnn` build flags must be removed; workloads should migrate to the MLAS/KleidiAI-backed CPU EP or QNN EP. • `ORT_API_VERSION` is now 25, which may affect EP plugins or native bindings that check the API version at load time.

ONNX Runtime v1.24.3

Thu, 05 Mar 2026 19:00:05 GMT

ONNX Runtime v1.24.3 adds DQ→MatMulNBits fusion for NvTensorRtRtx, OrtEnv.DisableDllImportResolver for NuGet, and DynamicQGemm ukernel in MLAS. • Adds `OrtEnv.DisableDllImportResolver` API to prevent fatal errors when a DLL import resolver conflict is detected in NuGet environments. • Adds `'library_path'` metadata entry to `OrtEpDevice` instances for plugin and provider bridge execution providers. • Adds `DynamicQGemm` function pointers and ukernel interface to MLAS. • Adds DQ→MatMulNBits fusion transformer to the NvTensorRtRtx execution provider. • Increases `kMaxValueLength` to 8192 in config. • Builds Windows ARM64X binaries as part of the packaging pipeline. • Adds Python 3.14 CI support with upgraded dependencies. • Delivers up to 4x QMoE CPU performance improvement on 4-bit workloads. • Supports using the embedded WASM module in Blob URL workers when `wasmBinary` is provided in WebGPU EP. • Allows a new memory info name for WebGPU EP.

ONNX Runtime v1.24.1

Fri, 06 Feb 2026 00:00:13 GMT

ONNX Runtime v1.24.1 adds plugin EP APIs, free-threaded Python 3.13t/3.14t, CUDA Graph on by default, and broad new op coverage across CUDA, QNN, WebGPU, and CPU. • Adds OrtApi::CreateEnvWithOptions() and OrtEpApi::GetEnvConfigEntries() for programmatic environment configuration in plugin execution providers. • Adds `OrtKernelInfo` APIs for kernel-based plugin EPs, enabling kernel authors to query session configuration from `KernelInfo`. • Adds `add_external_initializers_from_files` to the Python binding for loading external initializers directly from file paths. • Adds `--enable_arm_neon_nchwc` build flag to enable NCHWc layout support for Conv models on ARM NEON, improving convolutional inference performance. • Enables CUDA Graph by default in the CUDA EP, reducing per-inference launch overhead without additional configuration. • Introduces a plugin-based Execution Provider (EP) infrastructure with dynamic loading, including initial kernel support, weight pre-packing, EP context model support, and control flow kernel APIs. • Adds EP Device Compatibility APIs for matching execution providers to available hardware at runtime. • Adds External Resource Importer API for importing D3D12 shared resources into the DirectML/DX12 execution path. • Adds Python 3.14 support and free-threaded Python (`PEP 703`) support for Python 3.13t and 3.14t on Linux. • Upgrades ONNX dependency to 1.20.1 and Protobuf from 3.20.3 to 4.25.8. • Upgrades `ORT_API_VERSION` to 24. • Adds Java bindings for Auto EP, compiled model support (`OrtCompiledModelCompatibility`), and compile model APIs. • Adds .NET 9.0 MAUI targets to the C# binding. • CUDA EP gains Flash Attention updates, GQA kernel fusion, BF16 support for MoE/qMoE/MatMulNBits, and CUDA 13.0 support. • TensorRT EP upgraded to TensorRT 10.14 with automatic plugin loading and NVFP4 custom ops. • TensorRT RTX EP gains RTX runtime caching, CUDA graph support, BFloat16, and memory-mapped engine support. • QNN EP upgraded to SDK 2.42.0, adding RMSNorm, ScatterElements, GatherND, STFT, and RandomUniformLike ops, plus Gelu pattern fusion, LPBQ quantization, ARM64 wheel builds, and v81 device support. • OpenVINO EP upgraded to 2025.4.1. • MIGraphX EP adds QuickGelu, multihead attention, and QLinear pooling ops. • WebGPU EP gains Flash Attention optimizations, graph capture, Split-K MatMul, qMoE support, and WGSL templates. • WebNN EP adds GQA local attention, GatherBlockQuantized, ConvInteger, and MatMulInteger ops. • Node.js binding updated to Node.js v22 with JSI for React Native and JSPI build support. • CPU backend adds MoE/qMoE kernels, RotaryEmbeddings (opset 23), LayerNorm/RMSNorm broadcasting, SME1/SME2 Convolution and SGemm kernels via KleidiAI, S390x SIMD, and LoongArch64 4-bit quantization. • Arm NN EP is formally deprecated and will be removed in an upcoming release; all `--use_armnn` build options will be dropped. Breaking changes: • Python 3.10 wheels are no longer published; upgrade to Python 3.11 or later is required. • x86_64 binaries for macOS and iOS are no longer provided, and the minimum macOS version is raised to 14.0. • Builds using `--use_armnn` will fail once the upstream ArmNN EP removal lands (release TBD); adjust configurations in advance.

ONNX Runtime v1.23.1

Wed, 08 Oct 2025 04:12:55 GMT

ONNX Runtime v1.23.1 adds new Python and C# APIs, CPU MoE kernels, and a dynamic QNN HTP performance mode option. • Adds a dynamic option to set HTP performance mode in the QNN EP (`QNN EP`). • Adds `MemcpyFromHost` and `MemcpyToHost` support for plugin execution providers. • Adds CPU Mixture-of-Experts (MoE) kernel for CPU execution. • Adds block-wise QMoE kernel for CPU execution. • Implements new Python APIs. • Implements missing C# APIs. • Adds `session_id_` to `LogEvaluationStart`, `LogEvaluationStop`, and `LogSessionCreationStart` for improved session tracing. • Adds EP ABI check to verify whether nodes specified in GetCapability() have already been assigned.

ONNX Runtime v1.23.0

Fri, 26 Sep 2025 04:33:48 GMT

ONNX Runtime v1.23.0 adds an EP Plugin API, dynamic EP downloading on Windows 11 25H2, and a new Nvidia TensorRT RTX execution provider. • Introduces the Execution Provider (EP) Plugin API, a new infrastructure for building plugin-based execution providers. • Adds dynamic download and installation of execution providers via the WinML build on Windows 11 version 25H2 or later; C/C++/C# users consume this through the Windows App SDK, Python users via the `onnxruntime-winml` package. • Adds AutoEP/Device Management capability for automatic discovery of computing devices and selection of the best execution providers to download and register. • Adds a new Nvidia TensorRT RTX execution provider. • Adds support for QNN SDK 2.37 in the QNN EP. • Adds WGSL template support to the WebGPU EP. • Upgrades EMSCRIPTEN SDK (EMDSK) from 4.0.4 to 4.0.8 in the Web EP. • Enhances KleidiAI performance for SGEMM, IGEMM, and Dynamic Quantized MatMul operations, with improvements for Conv2D operators on hardware supporting SME2 (Scalable Matrix Extension v2). • Removes the ROCM EP from the source tree; AMD users are directed to the MIGraphX or Vitis AI EPs. Breaking changes: • The ROCM EP has been removed from the source tree; existing setups relying on it must migrate to the MIGraphX or Vitis AI EPs.

ONNX Runtime v1.22.2

Wed, 13 Aug 2025 16:53:48 GMT

ONNX Runtime v1.22.2 adds `--client_package_build`, vectorized DequantizeLinear, and QNN EP operator support for Upsample, Einsum, LSTM, and CumSum. • Adds `--client_package_build` build option that enables defaults tuned for client/on-device workloads, including disabling thread spinning by default. • Adds multithreaded, vectorized CPU/MLAS implementation of `DequantizeLinear` for `int8` and `uint8` inputs using SSE2 and NEON instruction sets. • Adds QNN EP support for the `Upsample`, `Einsum`, `LSTM`, and `CumSum` operators. • Enables DSP queue polling in QNN EP when performance mode is set to 'burst'. • Updates bundled QNN SDK to version 2.36.1 in the QNN execution provider. • Fuses scale into Softmax in the QNN EP for improved operator fusion.

ONNX Runtime v1.22.0

Sat, 10 May 2025 01:14:00 GMT

ONNX Runtime v1.22.0 adds Model Editor, Compile, and Auto EP APIs plus a new TensorRT RTX execution provider. • New `OrtCompileApi` struct enables explicit AOT compilation of ONNX models via a dedicated Compile API. • New `OrtModelEditorApi` struct exposes an API for programmatically creating and editing ONNX models in-process. • Adds Auto EP Selection infrastructure that automatically chooses Execution Providers via configurable selection policies, reducing manual EP configuration. • Introduces the NV TensorRT RTX Execution Provider, a new EP targeting Nvidia RTX GPUs via TensorRT. • Adds support for `MatMulNBits` (8-bit weight-only quantization) in both the CPU/MLAS and CUDA EPs. • Integrates KleidiAI into ONNX Runtime/MLAS for improved performance on Arm architectures. • Adds constrained decoding support for generative AI models, giving finer control over output token selection. • QNN EP gains support for QNN SDK 2.33.2, operator coverage for Sum, Softmax, Upsample, Expand, ScatterND, and Einsum, QnnGpu backend, and the ability to build as a shared or static library. • TensorRT EP adds support for TensorRT 10.9, a new EP option to enable TRT Preview Features, and support for loading TensorRT V3 plugins. • OpenVINO EP adds support up to OpenVINO 2025.1, Intel compiler-level optimizations for QDQ models, device selection by LUID, and `Load_config` support for AUTO, HETERO, and MULTI plugins. • WebGPU support extended to the Node.js package on Windows and macOS, and enabled when building from source on macOS, Linux, and Windows. • QNN NuGet package is now built as ARM64x. Breaking changes: • CUDA 11.x GPU packages are no longer published; only CUDA 12.x packages are available. • The minimum supported Windows version is now 10.0.19041.

ONNX Runtime v1.21.0

Sat, 08 Mar 2025 05:33:03 GMT

ONNX Runtime v1.21.0 adds chat mode, TensorRT 10.8, QNN shared memory, OpenVINO weights sharing, and expanded tokenizer support. • Introduces `trt_op_types_to_exclude` option to exclude specific ops from TensorRT assignment, giving fine-grained control over TensorRT EP graph partitioning. • Adds `preload_dlls` Python API for the CUDA EP to coexist with PyTorch without DLL conflicts. • Adds CMake option `onnxruntime_BUILD_QNN_EP_STATIC_LIB` for building QNN EP as a static library; QNN EP is now built as a shared library/DLL by default (use `--use_qnn static_lib` to retain old behavior). • Adds CMake options `onnxruntime_USE_EXTERNAL_DAWN`, `onnxruntime_CUSTOM_DAWN_SRC_PATH`, `onnxruntime_BUILD_DAWN_MONOLITHIC_LIBRARY`, `onnxruntime_ENABLE_PIX_FOR_WEBGPU_EP`, `onnxruntime_ENABLE_DAWN_BACKEND_VULKAN`, and `onnxruntime_ENABLE_DAWN_BACKEND_D3D12` for WebGPU EP build configuration. • Enables `onnxruntime_USE_CUDA_NHWC_OPS` by default for CUDA builds. • Adds support for TensorRT 10.8, including default assignment of DDS ops `NMS`, `RoiAlign`, and `NonZero` to TensorRT. • Adds chat mode support for CPU, GPU, and WebGPU execution. • Adds support for decoder model pipelines. • Adds Java API support for MultiLoRA. • Introduces QNN shared memory support for the QNN EP. • Adds support for QAIRT/QNN SDK 2.31 and a Python 3.13 package for QNN EP. • Introduces OpenVINO EP Weights Sharing feature. • Adds contrib op support in OpenVINO EP: `SkipLayerNormalization`, `MatMulNBits`, `FusedGemm`, `FusedConv`, `EmbedLayerNormalization`, `BiasGelu`, `Attention`, `DynamicQuantizeMatMul`, `FusedMatMul`, `QuickGelu`, `SkipSimplifiedLayerNormalization`. • Adds support for caching generated CoreML models. • Expands tokenizer support to include `ChatGLM`, `Baichuan2`, and `Phi-4` models. • Adds full `Phi-4` pre/post-processing support for text, vision, and audio. • Introduces RegEx pattern loading from `tokenizer.json`. • Adds support for loading tokenizer data from a memory blob in the C API. • Introduces a new tokenizer op schema to unify the tokenizer codebase. • Adds support for WASM64 (build from source). • Increases minimum required CMake version from 3.26 to 3.28 and adds support for CMake 4.0. • Increases minimum required Python version from 3.8 to 3.10 for building ONNX Runtime from source. • Removes the `onnxruntime_USE_PREINSTALLED_EIGEN` CMake option. • Upgrades Gradle from 7.x to 8.x and JDK from 11 to 17 for Android builds. Breaking changes: • Chat mode introduces breaking API changes — see the migration guide at https://onnxruntime.ai/docs/genai/howto/migrate.html. • QNN EP is now built as a shared library/DLL by default; builds that previously relied on static linking must add `--use_qnn static_lib`. • All prebuilt Windows packages now require VC++ Runtime >= 14.40 (previously 14.38); lower versions will cause a crash on initialization. • Minimum iOS SDK requirement raised to >= 15.1. • Minimum Android API requirement raised to >= 24 (Android 7). • All macOS packages now require macOS >= 13.3. • Minimum CMake version increased from 3.26 to 3.28. • Minimum Python version for building from source increased from 3.8 to 3.10. • The `onnxruntime_USE_PREINSTALLED_EIGEN` CMake option has been removed. • TVM EP has been removed from the source tree. • NNAPI EP is marked for deprecation following Google's deprecation of NNAPI.

ONNX Runtime v1.20.1

Thu, 21 Nov 2024 22:20:31 GMT

ONNX Runtime v1.20.1 adds get_qdq_config() helper, QNN SDK v2.28.2 support, and reworked native library packaging. • Introduces get_qdq_config() helper function in the Python Quantization Tool to retrieve QDQ configurations programmatically. • Adds `reduce_range` option to get_qdq_config() for finer control over quantization range in QDQ workflows. • Adds support for QNN SDK v2.28.2 in the QNN Execution Provider. • Reworks native library loading so a pre-built ORT native package can be consumed without modification. • Updates QDQ operator handling for Pad, Slice, and Softmax nodes in the Python Quantization Tool. • Excludes DDS ops from running on the TensorRT Execution Provider.

ONNX Runtime v1.20.0

Fri, 01 Nov 2024 18:02:18 GMT

ONNX Runtime v1.20 adds MultiLoRA, INT4 embedding, cuDNN Flash Attention, QNN weight sharing, and WebGPU on mobile. • Adds ability to serialize pre-packed weights to avoid memory growth on model load. • Adds `bfloat16` and `float8` data type support in the Python I/O binding API. • Adds INT4 quantized embedding support on CPU and CUDA execution providers. • Adds FP16 support for `MatMulNbits`, `Clip`, and `LayerNormalization` ops on the CPU EP. • Adds cuDNN frontend integration for convolution operators on the CUDA EP. • Adds cuDNN Flash Attention and Lean Attention support in the `MultiHeadAttention` op on the CUDA EP. • Adds TensorRT 10.4 and 10.5 support in the TensorRT EP. • Adds QNN HTP weight sharing across multiple ORT inference sessions in the QNN EP. • Adds support for QNN SDK 2.27. • Adds support up to OpenVINO 2024.4.1 in the OpenVINO EP. • Adds remote tensors for direct memory access during NPU inferencing in the OpenVINO EP. • Adds DirectML 1.15.2 support. • Adds FP16 support for ML Program models with the CoreML EP on mobile. • Adds FP16 XNNPACK kernels as a CoreML fallback option on mobile. • Adds initial support for the native WebGPU EP on Android and iOS. • Adds on-demand weight loading for Web/Wasm32, enabling 8B-parameter LLMs by offloading the Wasm32 heap. • Adds Opset-21 support (`Reshape`, `Shape`, `Gelu`) for the Web target. • Adds quantized embedding support for Web. • Adds MultiLoRA support in Core and GenAI. • Adds support for mid-loop generation termination in GenAI. • Adds logit soft capping support in Group Query Attention (GQA) for GenAI. • Adds model support for Phi-3.5 Vision Multi-Frame, ChatGLM3, and Nemotron-Mini in GenAI. • Adds GenAI Python package for Mac and Mac/iOS NuGet packages. • Adds Unigram tokenization model support in Extensions. • Adds support for the latest Hugging Face tokenization JSON format (`transformers>=4.45`) in Extensions. • Adds Olive CLI for executing well-defined workflows without manual config creation or editing. • Adds YAML-based workflow config support in Olive. • Adds Python 3.13 support in PyPI packages. • Adds experimental vcpkg support for the CPU EP. Breaking changes: • All ONNX Runtime Training packages are deprecated and will no longer be published: `onnxruntime-training` (PyPI), `onnxruntime-training-cpu` (PyPI), `Microsoft.ML.OnnxRuntime.Training` (NuGet), `onnxruntime-training-c` (CocoaPods), `onnxruntime-training-objc` (CocoaPods), and `onnxruntime-training-android` (Maven Central). ORT 1.19.2 was the last release to include them. • Python 3.8 and Python 3.9 are no longer supported in ORT packages. Users on those versions must stay on ORT 1.19.2 or earlier. • CUDA packages now require additional shared libraries that were not needed in 1.19: `libcudnn_adv.so.9`, `libcudnn_cnn.so.9`, `libcudnn_engines_precompiled.so.9`, `libcudnn_engines_runtime_compiled.so.9`, `libcudnn_graph.so.9`, `libcudnn_heuristic.so.9`, `libcudnn_ops.so.9`, `libnvrtc.so.12`, and `libz.so.1`.

ONNX Runtime v1.19.2

Wed, 04 Sep 2024 19:33:14 GMT

ONNX Runtime v1.19.2 adds CUDA Phi3 MoE kernel, smooth softmax for GroupQueryAttention, and causal support in MultiHeadAttention. • Adds CUDA kernel for Phi3 MoE (Mixture of Experts) inference. • Adds smooth softmax support in CUDA and CPU kernels for the `GroupQueryAttention` operator. • Enables causal support in the `MultiHeadAttention` CUDA operator. • Restores support for CUDA compute capability 7.0 and 7.5 with CUDA 12, and 6.0 and 6.1 with CUDA 11 in training workloads.

ONNX Runtime v1.19.0

Mon, 19 Aug 2024 18:44:22 GMT

ONNX Runtime v1.19.0 adds TensorRT 10.2, CUDA 12/CuDNN 9 defaults, INT4 QDQ, FlashAttention on CPU, and new GenAI model support. • Adds `sdpa_kernel` provider option to the CUDA Execution Provider to choose the kernel used for Scaled Dot-Product Attention. • Adds QDQ support for INT4 quantization in CPU and CUDA Execution Providers. • Adds C APIs for language, vision, and audio processors in Extensions, including a new `FeatureExtractor` for Whisper. • Adds new CUDA custom operators in Extensions: `MulSigmoid`, `Transpose2DCast`, `ReplaceZero`, `AddSharedInput`, and `MulSharedInput`. • Introduces Java CUDA 12 packages published on Maven. • Adds Java bindings for GenAI (build from source). • Supports enabling EpContext via session options in the OpenVINO Execution Provider. • Enables refittable embedded engine in TensorRT when an ONNX model is provided as a byte stream. • Implements FlashAttention on CPU to improve performance for GenAI prompt workloads. • Implements `DeformConv` operator support. • Expands CUDA Execution Provider op support to include `Tile` (bf16). • Expands CPU Execution Provider op support to include `GroupQueryAttention` and `SparseAttention` (for Phi-3 small). • Expands QNN Execution Provider op support: `HardSigmoid`, `ConvTranspose 3d`, `Clip` (int32 data), `Matmul` (int4 weights), `Conv` (int4 weights), `prelu` (fp16). • Adds Conv + Clip/Relu fusion support in the QNN Execution Provider. • Adds support for OpenVINO 2024.3. • Updates TensorRT Execution Provider to support TensorRT 10.2. • Updates DirectML from 1.14.1 to 1.15.1 with ONNX opset support expanded from 17 to 20. • Adds support for building quantized GenAI models using AWQ and GPTQ methods. • Adds GenAI support for new models: Qwen, Llama 3.1, Gemma 2, and Phi-3 small. • Adds Android support for GenAI (build from source). • Publishes GenAI packages for Windows ARM. • Adds support for the Phi-3 Small Tokenizer and a new OpenAI tiktoken format for fast loading of BPE tokenizers in Extensions. • Adds NumPy 2.x support. • Updates default GPU packages to use CUDA 12.x and CuDNN 9.x (previously CUDA 11.x/CuDNN 8.x); CUDA 11.x/CuDNN 8.x packages moved to the aiinfra VS feed. • Updates QNN Execution Provider to support QNN SDK 2.25. • Improves CPU operator coverage for WebNN (now supported by Chrome). • Adds additional CoreML ML Program operators for mobile. Breaking changes: • Default GPU packages now use CUDA 12.x and CuDNN 9.x; CUDA 11.x/CuDNN 8.x packages are no longer in the default feed and have been moved to the aiinfra VS feed. • Xamarin support has been removed; projects targeting Xamarin will no longer build. • macOS 11 is no longer supported; the minimum supported macOS version is now 12. • iOS 12 is no longer supported; the minimum supported iOS version is now 13. • JavaScript packaging for `onnxruntime-web` has been updated in ways that introduce slight incompatibilities when apps bundle `onnxruntime-web`.

ONNX Runtime v1.18.1

Fri, 28 Jun 2024 00:29:13 GMT

ONNX Runtime v1.18.1 adds TensorRT Weightless API, hardware-compatible engines, and INT64 calibration support. • Adds TensorRT Weightless API integration to the TensorRT Execution Provider. • Adds support for TensorRT hardware compatible engines in the TensorRT EP. • Adds support for INT64 types in TensorRT constant layer calibration. • CUDA 12.x GPU packages are now built against cuDNN 9.x (previously cuDNN 8.x); CUDA 11.x packages continue to use cuDNN 8.x. • Publishes CUDA 12 Java packages to the Azure DevOps feed. • Updates to the latest commit of the onnx-tensorrt parser, incorporating upstream improvements. Breaking changes: • CUDA 12.x ONNX Runtime GPU packages now depend on cuDNN 9.x instead of cuDNN 8.x, requiring a cuDNN upgrade for existing CUDA 12.x installations. • Windows packages now require Microsoft Visual C++ Redistributable Runtime 14.38 or newer. • ONNX Runtime Python packages now require numpy >=1.21.6, <2.0; numpy 2.0 is not supported.

ONNX Runtime v1.18.0

Tue, 21 May 2024 00:28:41 GMT

ONNX Runtime v1.18.0 adds TensorRT 10, WebNN EP preview, RISC-V support, QNN mixed-precision, and GenAI model expansion. • Adds `--rv64`, `--riscv_toolchain_root`, and `--riscv_qemu_path` build options for initial RISC-V architecture support. • Adds `--use_binskim_compliant_compile_flags` build option to opt into security-related compile/link flags (default OFF for source builds; ON for all release binaries). • Adds `onnxruntime_CUDA_MINIMAL` CMake option to build the CUDA execution provider with only memcpy ops, enabling minimal CUDA builds. • Adds `enable_htp_fp16` provider option to the QNN EP for fp16 execution on HTP. • Adds a provider option to the CUDA EP to disable TF32. • Adds `SessionOptions.DisablePerSessionThreads` to the C# API, enabling threadpool sharing between sessions. • Adds a new `SessionOptions` config entry to disable specific graph transformers and rules. • Exposes Reserve() in `OrtAllocator` to allow custom allocators to work when `session.use_device_allocator_for_initializers` is specified. • Adds WebNN EP as a preview execution provider for Web targets. • Adds TensorRT 10 support to the TensorRT EP. • Adds Python support for user-provided CUDA streams in both the CUDA and TensorRT EPs. • Adds support for multiple CUDA graphs in the CUDA EP. • Extends MoE in the CUDA EP to support Tensor Parallelism and int4 quantization. • Adds QNN SDK support up to version 2.22, with mixed 8/16-bit precision configurability per layer (upgraded from A16W8). • Adds multiple partition support for QNN context binary. • Adds per-channel quantized weights support for Conv in the QNN EP. • Integrates QNN EP with Qualcomm's AIHub. • Adds OpenVINO 2024.1 support, including import of pre-compiled blobs as EPContext blobs. • Separates device and precision as distinct inputs in the OpenVINO EP, adding precision as a separate CLI option and removing `device_id` from provider options. • Adds DirectML operator support for Resize-18, Resize-19, Col2Im-18, IsNaN-20, IsInf-20, and ReduceMax-20, plus contrib ops SimplifiedLayerNormalization, SkipSimplifiedLayerNormalization, QLinearAveragePool, MatMulIntegerToFloat, GroupQueryAttention, DynamicQuantizeMatMul, and QAttention. • Adds HQQ quantization support for 4-bit quant to improve accuracy on GPU. • Adds support for models larger than 2 GB in on-device training, enabling SLM training on edge devices. • Adds GenAI model support for Phi-3, Gemma, and LLama-3. • Adds DML EP support for GenAI. • Adds support for building ONNX Runtime with QNN on Android. • Adds visionOS support for mobile builds. • Adds initial support for creating ML Program format CoreML models. • Adds 1D Conv and ConvTranspose support to the XNNPACK EP. • Adds MacCatalyst (Catalyst) support for macOS builds. • Adds Mixtral integration using the ORT training backend. • Adds support for Hugging Face FastTokenizer conversion into an ONNX custom operator. • Adds Java CUDA 12 support and a Java packaging pipeline published to Maven repository. • Enables eager execution for custom operators in Extensions. Breaking changes: • Windows ARM32 support has been dropped at the source code level. • Python version >=3.8 is now required for `build.bat`/`build.sh` (previously >=3.7). • The `onnxruntime-mobile` Android package and `onnxruntime-mobile-c`/`onnxruntime-mobile-objc` iOS CocoaPods are deprecated; users must migrate to `onnxruntime-android` and `onnxruntime-c`/`onnxruntime-objc`. • OpenVINO EP removes `device_id` from provider options; device and precision are now separate inputs, and `CPU_FP32`/`GPU_FP32` terminology is replaced with `CPU`/`GPU`. • Security-related compile/link flags have been moved out of the default build settings into the new `--use_binskim_compliant_compile_flags` option, which is OFF by default when building from source. • Windows OneCore build now uses 'Reverse forwarding' apisets instead of 'Direct forwarding', causing `onnxruntime.dll` in NuGet packages to depend on `kernel32.dll`; systems without `kernel32.dll` require reverse forwarders.

ONNX Runtime v1.17.3

Thu, 18 Apr 2024 15:46:48 GMT

ONNX Runtime v1.17.3 adds WebGPU/WebNN capabilities, new op support, and packed QKV with Rotary Embedding for sm<80 GPUs. • Adds `hardSigmoid` op support and `hardSigmoid` activation for `fusedConv` in the Web (WebGPU) execution provider. • Adds `LeakyRelu` activation for `fusedConv` in the Web execution provider. • Adds `FastGelu` custom op support in the Web execution provider. • Adds `MatMulNBits` op with optimizations for the Web execution provider. • Adds support for WebNN async API via Asyncify in the Web execution provider. • Adds capture and replay support for the JS execution provider. • Allows `uint8` tensors for WebGPU. • Enables ort-web with any `Float16Array` polyfill. • Adds uniform support for conv, conv transpose, conv grouped, and fp16 in the Web execution provider. • Adds support for packed QKV input and Rotary Embedding with sm<80 GPUs using the Memory Efficient Attention kernel. • Updates replacement logic for `MultiHeadAttention` (MHA) and `GroupQueryAttention` (GQA) kernel optimizations. • Adds benchmarking support for LLaMA model end-to-end performance. • Adds example demonstrating export of OpenAI Whisper implementation with batched prompts.

ONNX Runtime v1.17.1

Tue, 27 Feb 2024 18:34:20 GMT

ONNX Runtime v1.17.1 adds DirectML/CUDA EP support for Node.js, ETW session logging, and a new DefaultTensorType quantization option. • Adds `DefaultTensorType` option to the quantization API to specify the default tensor type to quantize. • Adds capturestate/rundown ETW support logging for session and provider options. • Enables DirectML on Windows and CUDA on Linux for the Node.js binding. • Adds x64 Python wheels build job for QNN EP, enabling cached QNN model creation on Windows x64. • Adds ATen fallback support for the bicubic interpolation algorithm in `onnxruntime-training`. • Adds contrib Q/DQ ops to the symbolic shape inference tool. • Updates Q/DQ quantization to ensure Microsoft opset compliance. • Reduces `onnxruntime-training` package size to allow publication on PyPI.

ONNX Runtime v1.17.0

Sat, 03 Feb 2024 00:25:21 GMT

ONNX Runtime v1.17 adds ONNX 1.15 opsets, QNN mixed-precision, WebGPU GA, QLoRA training, and new EP APIs. • Adds `AppendExecutionProvider_OpenVINO_V2` API for configuring new OpenVINO EP options. • Adds QNN EP device-level session options `soc_model`, `htp_arch`, `device_id`, `extreme_power_saver` for `htp_performance_mode`, and `vtcm_mb`. • Adds `--enable_lto` build flag to opt back in to link-time code generation/LTO, which is now disabled by default when building from source. • Adds new dynamic ETW provider for tracing and diagnosing ONNX Runtime internals at runtime with low performance overhead. • Adds support for 12 new ONNX 1.15 opsets: `IsInf-20`, `IsNaN-20`, `DFT-20`, `ReduceMax-20`, `ReduceMin-20`, `AffineGrid-20`, `GridSample`, `ConstantOfShape-20`, `RegexFullMatch`, `StringConcat`, `StringSplit`, and `ai.onnx.ml.LabelEncoder-4`. • Adds CUDA 12 packages for Python and NuGet. • Adds Python 3.12 packages for ONNX Runtime Inference. • Adds AMD GPU package with ROCm and MiGraphX (Python + Linux only). • Splits the ONNX Runtime GPU NuGet package into two separate packages. • Adds Windows ARM64X build support when building ONNX Runtime from source. • Adds support for supplying a custom logger at the session level. • Adds new logging and tracing of session and execution provider options. • Adds 4-bit quantization support on NVIDIA GPU and ARM64. • Adds support for direct load of precompiled TensorRT engines and customizable engine prefix in the TensorRT EP. • Adds Python support for TensorRT plugins via ORT custom ops. • Adds support for QNN SDK 2.18. • Adds QNN EP context binary caching and model initialization optimizations. • Adds mixed precision (8/16-bit) quantization support for the QNN EP. • Enables QNN NPU profiling dynamically via ETW or CSV output. • Adds support for OpenVINO 2023.2. • Adds `RotaryEmbedding` operator support to the DirectML EP. • Adds support for fusing subgraphs into DirectML execution plans. • Adds new Python API to select a specific GPU on multi-GPU devices with the DirectML EP. • Adds initial 4-bit quantization support on ARM64 for mobile. • Adds support for YOLOv8 pose detection pre/post processing on mobile. • Adds support for macOS in the CocoaPods package. • Adds Web support for external data format, I/O bindings, and training. • Transitions WebGPU backend out of experimental (now generally available) and adds FP16 support for WebGPU. • Enables QLoRA training support with BFloat16. • Enables memory-efficient gradient management for large model training (~10 GB memory reduction observed on Mistral). • Adds support for Aten efficient attention and Triton Flash Attention for large model training. • Adds training support on the web (On-Device Training), targeting federated learning and developer exploration. • Allows custom ops to invoke ONNX Runtime's internal thread pool for parallelism. • Adds CUDA kernel custom operator support in extensions, including two example operators. • Introduces `fairseq` flag in extensions to improve compatibility with certain Hugging Face tokenizers. • Incorporates `added_token` attribute into the BPE tokenizer in extensions to improve CodeGen tokenizer support. Breaking changes: • All Linux binaries now require glibc >= 2.28; CentOS 7 is no longer supported as a prebuilt target. • Windows ARM32 binaries are removed from official packages; users must build from source. • Android NDK 24.x support is removed; NDK 25.x or 26.x is required. • Building for Linux ARM64 or Android now requires the C/C++ compiler to support BFloat16. • Link-time code generation (LTCG/LTO) is now disabled by default when building from source; re-enable with `--enable_lto`.

ONNX Runtime v1.16.0

Wed, 20 Sep 2023 21:27:50 GMT

ONNX Runtime v1.16.0 adds fp8/4-bit CPU quant, FlashAttention v2, Azure EP on mobile, Swift Package Manager support, and major LLM training optimizations. • Adds `session.disable_cpu_ep_fallback` session option to prevent automatic fallback to the CPU execution provider. • Adds `ORTMODULE_ENABLE_EMBEDDING_SPARSE_OPTIMIZER` environment variable (disabled by default) for experimental embedding sparsity optimizations, improving Roberta training performance by 20-30%. • Adds `ORTMODULE_ENABLE_COMPUTE_OPTIMIZER` for Label Sparsity compute optimization, now enabled by default. • Adds `ORTMODULE_CACHE_DIR` model cache for exported ONNX models to avoid repeated exports when the model is unchanged. • Adds `gen_processing_models` Python API in onnxruntime-extensions to export ONNX data processing models from Hugging Face tokenizers (LLaMA, CLIP, XLM-Roberta, Falcon, BERT, etc.). • Supports serialization of models 2 GB and larger. • Java API gains support for fp16 and bf16 tensors as inputs/outputs, with conversion utilities leveraging hardware-accelerated `Float.float16ToFloat` and `Float.floatToFloat16` on JDK 20+. • Java API adds support for external initializers, enabling large models to be instantiated without filesystem access. • C# API exposes `OrtValue` as the new preferred inference API, reducing garbage collection overhead and providing direct native memory access via Slice-like interfaces. • C# `Float16` and `BFloat16` types become full-featured interfaces supporting conversion and floating-point properties such as `IsNaN` and `IsInfinity`. • C++ `Float16_t` and `BFloat16_t` types become full-featured interfaces supporting conversion and floating-point properties such as `IsNaN` and `IsInfinity`. • Adds 4-bit quantization support on CPU. • Adds LLM quantization accuracy improvement via smoothquant. • Adds FlashAttention v2 support for `Attention`, `MultiHeadAttention`, and `PackedMultiHeadAttention` ops. • CUDA EP gains initial fp8 support covering QDQ, Cast, and MatMul operations. • CUDA EP relaxes CUDA Graph constraints to allow more models to utilize CUDA Graphs. • CUDA EP allows the CUDA allocator to be registered with ONNX Runtime externally. • TensorRT EP adds CUDA Graph support and user-provided CUDA compute stream. • OpenVINO EP adds support for OpenVINO 2023.1. • QNN EP enables context binary cache to reduce initialization time. • QNN EP adds support for QNN 2.12 and resize with asymmetric transformation mode on the HTP backend. • QNN EP adds op support for Equal, Less, LessOrEqual, Greater, GreaterOrEqual, LayerNorm, Asin, Sign, DepthToSpace, SpaceToDepth, and 1D Conv/ConvTranspose. • Mobile gains initial Azure EP support. • Mobile adds dynamic shape support for CoreML. • Mobile adds Swift Package Manager support for ONNX Runtime inference and extensions via `onnxruntime-swift-package-manager`. • Mobile adds support for CLIPImageProcessor pre-processing and CLIP inference scenarios. • React Native performance improved via JSI. • WebGPU ops coverage expanded to support SAM, T5, and Whisper models. • WebNN ops coverage expanded to support SAM and Stable Diffusion models. • ORTModule + OpenAI Triton integration now available for computing ONNX sub-graphs during large model training. • On-Device Training adds iOS support. • On-Device Training adds a minimal build (~1.5 MB binary) for resource-constrained environments. • On-Device Training enables ORT-Extensions custom op support via `onnxblock`. • ORT Extensions adds `TrieTokenizer` operator for RWKV-like LLM models. • ORT Extensions adds new Azure EP operators: `AzureAudioToText`, `AzureTextToText`, and `AzureTritonInvoker` for Python and NuGet packages. • LLaMAv2 training achieves ~10% acceleration; OpenAI Whisper training optimizations also included. • PythonOp enhancements include bool and tuple[bool] constants, materialize grads, empty inputs, save-in-context, customized shape inference, and full-qualified name export.

ONNX Runtime v1.15.0

Thu, 25 May 2023 01:44:38 GMT

ONNX Runtime v1.15.0 adds JS and QNN execution providers, on-device training, WebGPU preview, and multi-GPU collective support. • Adds `--allow_running_as_root` flag to the build command to permit building as root on POSIX systems (default now disallows it). • Adds ONNX Optional type support in the C# API. • Adds collective operations to support multi-GPU inferencing. • Adds Python 3.11 support across `onnxruntime` CPU, `onnxruntime-gpu`, `onnxruntime-directml`, and `onnxruntime-training` packages (drops 3.7). • Introduces two new execution providers: JS EP (JavaScript) and QNN EP (Qualcomm Neural Network). • Adds initial public preview of QNN EP, available as a NuGet package (`Microsoft.ML.OnnxRuntime.QNN`). • Adds official support for TensorRT 8.6, including explicit shape profile overrides, TensorRT plugin support via ORT custom ops, and timing cache. • Adds support for TensorRT options: heuristics, sparsity, optimization level, auxiliary stream, and tactic source selection. • Adds support for OpenVINO 2023.0 and dynamic shapes for iGPU in OpenVINO EP. • Adds OpenAI Whisper model support in Azure EP, now also available as a NuGet package. • Adds DirectML 1.12 support with opset 16–17 coverage in DirectML EP. • Adds Swift Package Manager package for `onnxruntime` on mobile. • Adds NuGet package for `onnxruntime-extensions` with Android/iOS support for MAUI/Xamarin. • Adds React Native package for `onnxruntime` with optional `onnxruntime-extensions` inclusion. • Adds built-in pre/post processing for NLP scenarios (classification, question-answering, text-prediction) on mobile. • Adds built-in pre/post processing for Speech Recognition (Whisper) on mobile. • Adds built-in post processing for Object Detection (YOLO) including non-max suppression and bounding box drawing on mobile. • Adds NNAPI kernels for `BatchNormalization` and `LRN`; CoreML kernels for `Div`, `Flatten`, `LeakyRelu`, `LRN`, `Mul`, `Pad`, `Pow`, and `Sub`. • Adds [preview] WebGPU support in the Web build. • Adds official On-Device Training package with APIs and language bindings for C, C++, Python, C#, and Java; packages available for Desktop and Android. • Adds graph optimizations leveraging label-data sparsity for ORT Training, yielding 4%–15% performance gains on popular Hugging Face models. • Adds native Windows ARM64 build support using Visual Studio 2022. • Updates to CUDA 11.8 (source remains compatible with CUDA 11.4 and 12.x). • Adds a lock-free queue build option for threadpool to improve CPU utilization. • Adds fused decoder multi-head attention kernel improving GPT and decoder model (T5, Whisper) performance. • Adds packing mode to improve encoder model performance with inputs of large padding ratio. • Upgrades DNNL from 2.7.1 to 3.0. • Adds `cutlass` as a new dependency for CUDA/TensorRT packages. Breaking changes: • The `onnxruntime_ENABLE_EAGER_MODE` CMake option and eager mode code are deleted — builds using this option will fail. • Dropped support for Windows 8.1 and below. • In v1.16.0 (announced here): support for iOS 11 and below will be dropped; iOS 12 will be the minimum. • In v1.16.0 (announced here): support for CentOS 7, Ubuntu 18.04, and Linux distros without glibc >= 2.28 will be dropped. • In v1.16.0 (announced here): support for GCC <= 9 and Visual Studio 2019 will be dropped. • In v1.16.0 (announced here): the `onnxruntime_DISABLE_ABSEIL` build option will be removed.

ONNX Runtime v1.14.0

Sat, 11 Feb 2023 01:03:57 GMT

ONNX Runtime v1.14.0 adds NUMA-aware threading, multi-stream GPU execution, Azure EP, Rust API preview, and ONNX opset 18 support. • Adds new custom operator APIs enabling a custom op to wrap an entire model for inferencing with an external API or runtime. • Adds new API to set intra-op thread affinity for fine-grained CPU pinning. • Changes `cudnn_conv_use_max_workspace` default value to `1`, improving CUDA EP transformer model performance. • Supports ONNX 1.13 (opset 18), expanding the set of recognized operators and model formats. • ORT thread pool is now NUMA-aware, improving multi-socket CPU performance. • Multi-stream execution provider refactoring improves GPU utilization and memory reuse across parallel inference streams for CUDA, TensorRT, and ROCm EPs. • Adds `Stream` interface in ExecutionProvider API, allowing EP developers to customize stream implementations. • New [Preview] Rust API for ORT, available to build from the main branch. • Adds [Preview] Azure Execution Provider supporting AzureML-hosted models via Triton for hybrid on-device and on-cloud inferencing. • Adds quantization support with AMX on Sapphire Rapids processors. • ROCm EP now supports ROCm 5.4 and is GA-ready. • TensorRT EP adds support for TensorRT 8.5 GA versions. • OpenVINO EP adds support for OpenVINO 2022.3. • DirectML EP updated to DML 1.10.1, adding operators `NonZero`, `Shape`, `Size`, `Attention`, `EmbedLayerNorm`, `SkipLayerNorm`, and `BiasGelu`. • DirectML EP adds additional data type support for `Abs`, `Sign`, and `Where` operators. • DirectML EP enables `SetOptimizedFilePath` export/reload. • XNNPACK EP adds iOS build support and is now included in the `onnxruntime-c` iOS package. • XNNPACK EP adds support for using the ORT allocator in XNNPACK kernels to minimize memory usage. • XNNPACK EP adds support for additional commonly used operators. • Web build now includes `onnxruntime-extensions` (NLP-centric) by default. • Adds experimental utility functions for exchanging data between images and tensors in the JavaScript API. • FP16 optimizer exposed in `torch-ort` for use with DeepSpeed/Apex. • Mobile pre/post processing pipeline supports updating MobileNet and super-resolution models to embed pre/post processing (including JPG/PNG conversion custom ops) directly into the model via the `onnxruntime-extensions` Python package. • Stable Diffusion training optimizations, including support for `Resize` and `InstanceNorm` gradients, with ORT-enabled examples added to the Hugging Face `diffusers` library. Breaking changes: • Building ORT from source now requires cmake version >=3.24 instead of >=3.18.

ONNX Runtime v1.13.1

Mon, 24 Oct 2022 21:09:37 GMT

ONNX Runtime v1.13.1 adds CANN EP, new quantization APIs, XNNPACK threadpool support, and 4 GB WebAssembly memory. • Sets `session.use_ort_model_bytes_directly` and `session.use_ort_model_bytes_for_initializers` to `1` in SessionOptions config to load ORT format model initializers directly from bytes, reducing peak memory usage on mobile. • Adds new `QuantConfig`-based quantize API for general quantization workflows. • Exposes all arena configs in the Python API in an extensible way. • Adds new quantized operators: `SoftMax`, `Split`, and `Where`. • Adds a quantization debugging tool to identify sensitive nodes/layers from accuracy-drop discrepancies. • New CANN Execution Provider (CANN EP) contributed by Huawei for initial support of Ascend 310 hardware. • TensorRT EP adds a build option to link against a pre-built onnx-tensorrt parser, enabling no-code TensorRT minor version upgrades and support for TensorRT 8.5 EA. • TensorRT EP adds `uint8` data type support. • OpenVINO EP adds support for INT8 QDQ models from NNCF, Intel 13th Gen Core (Raptor Lake), and preview support for Intel Data Center GPU Flex Series and Intel Arc GPU. • DirectML EP adds new ops: `LayerNormalization`, `Gelu`, `MatMulScale`, `DFT`, and `FusedMatMul` (contrib). • SNPE EP NuGet package (`Microsoft.ML.OnnxRuntime.Snpe`) is now available and adds support for Windows Dev Kit 2023. • XNNPACK EP adds QDQ format model support with new kernels: `AveragePool`, `Softmax`, `QLinearConv`, `QLinearAveragePool`, and `QLinearSoftmax`. • XNNPACK EP adds threadpool support for configurable parallelism. • WebAssembly build now supports up to 4 GB memory. • Web build adds initial XNNPACK support for Wasm optimizations. • Web build adds source build support for `onnxruntime-extensions` and `sentencepiece`. • Adds fused attention CUDA kernels for BERT, fusing `Add` (bias) and `Transpose` of Q/K/V into one kernel for Attention and LongformerAttention. • Reduces GEMM computation in LongformerAttention via a new weight format. • Official ONNX Runtime GPU packages are now built with CUDA 11.6. • Training packages updated to CUDA 11.6 with Aten support added for `GroupNorm`, `InstanceNormalization`, and `Upsample nearest`. Breaking changes: • ORT format models must be regenerated: the mobile EP kernel format changed from hashes to operator constraint information and is NOT backwards compatible. • Official ONNX Runtime GPU packages now require CUDA version >=11.6 (previously 11.4). • NUPHAR EP code has been removed. • Building ORT from source now requires a C++17 compiler; on Linux, GCC >=7.0 is required. • Minimal numpy version for ONNX Runtime Python packages is now 1.21.6 (raised from 1.21.0). • Windows 8.x is no longer supported in NuGet/C API prebuilt binaries (source builds retain Windows 7+ support). • CUDA 10.x support removed at the source code level. • Training packages drop CUDA 10.2 and 11.3 support.

ONNX Runtime v1.12.0

Fri, 22 Jul 2022 04:43:39 GMT

ONNX Runtime v1.12 adds SNPE EP, XNNPACK EP preview, opset 17, BeamSearch op, and FusedAdam optimizer for training. • New `enable_dynamic_shapes` runtime configuration option for the OpenVINO EP to enable dynamic shapes per inference iteration. • Adds new `BeamSearch` operator for encoder-decoder transformer models. • Supports invoking individual ops without creating a separate graph, enabling custom op development to reuse ORT code. • Supports feeding external initializers (for large models) as byte arrays for model inferencing. • New SNPE Execution Provider (production release). • New XNNPACK Execution Provider (preview) with initial operator support (`Conv` and `MaxPool`) for ORT Mobile and ORT Web. • Adds ONNX opset 17 support. • Adds onnx-ml opset 3 support (`TreeEnsemble` update). • New `FusedAdam` optimizer available through the `torch-ort` package for training integration. • ORT Training acceleration now natively available through HuggingFace Optimum. • TensorRT EP gains TensorRT 8.4 support and an option to share execution context memory between TensorRT subgraphs. • TVM EP adds model `.dll` ingestion and execution on Windows. • DirectML EP updated to DirectML 1.9 with opset 13–15 support. • Adds Python 3.10 support. • Adds Mac M1 support in Python and Java packages. • Adds .NET 6/MAUI support in the NuGet C# package with target frameworks `net6.0`, `net6.0-android`, `net6.0-ios`, and `net6.0-macos`. • New `onnxruntime-openvino` package available on PyPI for OpenVINO EP. • New quantized kernels for weight symmetry to improve GEMM and Conv performance on ARM64 little cores. • Adds gradient ops for `ReduceMax` and `ReduceMin` in ORT Training. • Adds bitmask versions of `Dropout`, `BiasDropout`, and `DropoutGrad` with approximately 8x space savings for the mask output. • Adds `Bfloat16` support for `AtenOp` in ORT Training. • NNAPI EP gains new operator support: `DepthToSpace`, `PRelu`, `Gather`, `Unsqueeze`, `Pad`. • CoreML EP gains new operator support: `DepthToSpace`, `PRelu`. • Build switch to disable the abseil library dependency. Breaking changes: • The lightweight compile API is now the default compiler API for all Execution Providers; the legacy compiler API (previously at `execution_provider.h#L237`) will be deprecated in ORT 1.13 — EP maintainers must migrate to the lightweight compile API now. • `netstandard1.1` support is deprecated in this release and will be removed in ORT 1.13.

ONNX Runtime v1.10.0

Wed, 08 Dec 2021 00:22:39 GMT

ONNX Runtime v1.10.0 adds TensorRT+CUDA in one GPU package, new WinML/DirectML APIs, Mac M1 Universal2 builds, and ARM64 Linux support. • Adds `OrtSessionOptionsAppendExecutionProviderEx_DML`, `DmlCreateGPUAllocationFromD3DResource`, `DmlFreeGPUAllocation`, and `DmlGetD3D12ResourceFromAllocation` WinML APIs to create `OrtValue`s from Windows `ID3D12Resource` objects via DirectML EP-specific C-API extensions. • New C/C++ API to query the CUDA stream for launching custom kernels, enabling implicit synchronization between custom ops in shared libraries and ORT CUDA kernels. • Python `InferenceSession` now requires the `providers` parameter to be set explicitly when enabling non-default Execution Providers (e.g. `providers=['CUDAExecutionProvider']`). • Python GPU package (`onnxruntime-gpu`) now bundles both `TensorrtExecutionProvider` and `CUDAExecutionProvider` in a single install — EPs must be explicitly registered via the `providers` argument. • Added Mac M1 Universal2 build: a single binary that runs natively on both Apple Silicon and Intel-based Macs, included in official NuGet packages. • NuGet package now supports ARM64 Linux C#. • Added Xamarin support to ORT C# NuGet packages, with iOS and Android binaries included in the native package. • ORT format models now carry a backwards compatibility guarantee. • Supports plug-in custom thread creation and join functions to enable usage of external threads. • Adds Optional type support from op set 15. • Introduced indirect Convolution method for `QLinearConv` with symmetrically quantized (int8, zero-point=0) filters on x64 (AVX2, AVXVNNI, AVX-512, AVX-512 VNNI) and ARM64, eliminating memcpy and per-pixel output-image sum computation. • Added transpose optimizer to push and cancel transpose ops, significantly improving performance for models requiring layout transformation. • DirectML EP updated from DirectML.dll 1.5.1 to 1.8.0, adding full-precision uint64/int64 support for 48 operators, 8D support for 7 additional operators, and `DynamicQuantizeLinear` op. • OpenVINO EP adds support for OpenVINO 2021.4.x, Auto Plugin, and IO Buffer/Copy Avoidance Optimizations for the GPU plugin. • DNNL EP adds `Softmaxgrad`, `Transpose`, `Reshape`, `Pow`, `LeakyRelu`, `DynamicQuantizeLinear`, `Squeeze`, and `Unsqueeze` ops. • TensorRT EP Python GPU packages now include TensorRT 8.0 support. • Web backend adds WebAssembly SIMD support for the `qgemm` kernel, accelerating quantized model inference. • Windows C API symbols are now uploaded to the Microsoft public symbol server. • Optimized WebAssembly bundle size to support WebAssembly-only or WebGL-only production scenarios. Breaking changes: • `InferenceSession` now requires the `providers` parameter to be explicitly set when using any Execution Provider other than the default `CPUExecutionProvider`; omitting it will no longer silently activate other EPs. • Python 3.6 support removed for Mac builds. • `Invalid` allocator error code renamed to `OrtInvalidAllocator` in the C/C++ API. • Every item in `OrtCudnnConvAlgoSearch` has been updated to a safer global name.

ONNX Runtime v1.9.0

Thu, 23 Sep 2021 02:05:03 GMT

ONNX Runtime v1.9 adds opset 15, sparse tensor APIs, TensorRT V2 provider API, and CUDA/TensorRT bundling in GPU packages. • Adds `--_use_extensions` build option to statically link onnxruntime-extensions for running models with custom operators. • Adds `RegisterAllocator` and `UnregisterAllocator` C APIs (in `onnxruntime_c_api.h`) for sharing a custom allocator across multiple sessions. • Adds `SessionOptionsAppendExecutionProvider_TensorRT_V2`, `CreateTensorRTProviderOptions`, `UpdateTensorRTProviderOptions`, `GetTensorRTProviderOptionsAsString`, and `ReleaseTensorRTProviderOptions` C APIs for configuring TensorRT EP. • Adds `EnableOrtCustomOps` C API to enable custom operator support in a session. • Adds sparse tensor C APIs: `IsSparseTensor`, `CreateSparseTensorAsOrtValue`, `FillSparseTensorCoo`, `FillSparseTensorCsr`, `FillSparseTensorBlockSparse`, `CreateSparseTensorWithValuesAsOrtValue`, `UseCooIndices`, `UseCsrIndices`, `UseBlockSparseIndices`, `GetSparseTensorFormat`, `GetSparseTensorValuesTypeAndShape`, `GetSparseTensorValues`, `GetSparseTensorIndicesTypeShape`, and `GetSparseTensorIndices`. • Supports ONNX 1.10 with opset 15 and ONNX IR 8 (SparseTensor type, model-local function protos). • Official ORT GPU packages (non-Python) now bundle both CUDA and TensorRT Execution Providers in a single package, built against CUDA 11.4. • New `onnxruntime-directml` Python package available on PyPI for DirectML-accelerated inference on Windows. • C# NuGet package adds `netstandard2.0` as a supported target framework. • Adds `DebugOptions` and `LogLevels` to the `ORTModule` API for training debuggability. • ORT Training supports user-defined autograd functions and fallback to PyTorch for execution. • ORT Training adds support for deterministic compute to enable reproducibility with `ORTModule`. • ORT Training expands accepted input formats to include dictionaries and lists. • ORT Training adds ROCm 4.3.1 support on AMD GPUs. • ORT Web adds SIMD support in WebAssembly and an option to load WebAssembly from a worker thread to avoid blocking the main UI thread. • TensorRT EP adds support for TensorRT 8.0 (x64 Windows/Linux, ARM Jetson), including explicit-quantization features (ONNX Q/DQ support). • OpenVINO EP adds support for OpenVINO 2021.4. • CUDA EP adds support for sequence ops for models using the sequence type. • ORT Mobile iOS package switches to xcframework, supporting arm64 iPhone simulator on Apple silicon Macs. • Adds new quantized operator `QGemm` for direct quantization of Gemm. • IBM Power platform support added. • Quantization tool gains subgraph support. Breaking changes: • GCC versions below 7 are no longer supported. • `CMAKE_SYSTEM_PROCESSOR` must be explicitly set when cross-compiling on Linux (set to the output of `uname -m` on the target device) due to the new pytorch cpuinfo dependency for ARM big.LITTLE support. • `SessionOptionsAppendExecutionProvider_TensorRT` is deprecated; callers must migrate to `SessionOptionsAppendExecutionProvider_TensorRT_V2`. • Windows symbol (PDB) files are no longer included in the NuGet package; they must be downloaded separately from GitHub artifacts.

ONNX Runtime v1.8.1

Thu, 08 Jul 2021 01:54:51 GMT

ONNX Runtime v1.8.1 adds CoreML EP on iOS, React Native package, and ROCm 4.2 preview for training. • Enables CoreML Execution Provider in the iOS mobile package, bringing hardware-accelerated inference to Apple devices. • Publishes the `onnxruntime-react-native` package on npm, enabling ONNX Runtime inference in React Native applications. • Adds support for PyTorch 1.8.1 and 1.9 in ONNX Runtime for PyTorch training acceleration. • Adds support for CUDA 10.2 and 11.1 in the training acceleration stack. • Releases preview packages for ROCm 4.2, enabling AMD GPU training acceleration. • Extends training acceleration coverage to GPT-Neo, Microsoft TNLG, and TNLU models in addition to popular Hugging Face models. • Adds additional operators to the ORT Mobile package.

ONNX Runtime v1.8.0

Thu, 03 Jun 2021 06:14:48 GMT

ONNX Runtime v1.8.0 adds ONNX opset 14, dynamically loadable CUDA EP, ORT Web JS API, mobile packages, and a wave of new C APIs. • Adds `--enable_type_reduction` flag when creating ORT format models and `--enable_reduced_operator_type_support` flag for minimal builds to support per-type reduction across all operators in the ai.onnx domain. • Adds new C inferencing APIs: `KernelInfoGetAttributeArray_float`, `KernelInfoGetAttributeArray_int64`, `CreateArenaCfgV2`, `AddRunConfigEntry`, `CreatePrepackedWeightsContainer`, `PrepackedWeightsContainer`, `CreateSessionWithPrepackedWeightsContainer`, and `CreateSessionFromArrayWithPrepackedWeightsContainer`. • Adds `SetIntraOpThreadSpinning` native WinML API to toggle intra-op thread spin behavior (enabled by default), improving throughput for current workloads. • Supports ONNX opset 14 with new and updated operators from the ONNX 1.9 release. • Introduces a dynamically loadable CUDA execution provider, allowing a single build to serve both CPU and GPU inference (excluding Python packages). • Releases the new [ONNX Runtime JavaScript API](https://github.com/microsoft/onnxruntime/tree/master/js#onnxruntime-web) (ORT Web) with WebAssembly and WebGL backends, Web Worker multi-threaded WebAssembly support, and ORT model format support. • Releases pre-built Android and iOS mobile packages with an Objective-C API for iOS (preview) and expanded NNAPI and CoreML execution provider operator coverage. • Releases the `torch-ort` package as the ONNX Runtime backend for PyTorch training. • Releases `onnxruntime-training-gpu` and `onnxruntime-training-rocm` packages for distributed training on NVIDIA and AMD GPUs. • Releases the experimental `onnxruntime-extensions` package providing a crowd-sourced library of custom operators (including string ops and tokenizers) loadable at runtime. • Adds TensorRT EP configuration via session options (replacing environment variables) and DLA support on Jetson Xavier (AGX, NX). • Adds support for Loading/Saving compiled model blobs on MyriadX (OpenVINO) devices to avoid expensive compilation at runtime. • Adds OpenVINO 2021.3 support (drops OpenVINO 2020.4 support). • Extends DirectML execution provider to ARM/ARM64 architectures in WinML and NuGet packages, with support for 8-dimensional tensors and fusion activations for Conv, ConvTranspose, BatchNormalization, MeanVarianceNormalization, Gemm, and MatMul. • Adds memory footprint reduction via shared pre-packed weights across sessions using `CreatePrepackedWeightsContainer` / `CreateSessionWithPrepackedWeightsContainer`. • Adds a memory arena shrinkage `RunOption` to return unused memory to the system after inference runs, particularly useful for dynamic-shape models with large outlier requests. • Adds native Quantize-Dequantize (QDQ) format support for CPU and per-column quantization for MatMul. • Adds GPT-2 and beam search integration for transformer models. • Extends profiler tool with threadpool usage metrics: multi-threading preparation time, run time, and wait time. Breaking changes: • Building ONNX Runtime from source now requires a C++ compiler with full C++14 support. • OpenMP builds are no longer published; the default threadpool is used instead (OpenMP builds can still be compiled from source). • Python package now requires `flatbuffers` as a new dependency. • OpenVINO 2020.4 is no longer supported; OpenVINO 2021.3 is the new minimum.

ONNX Runtime v1.7.0

Wed, 03 Mar 2021 04:45:30 GMT

ONNX Runtime v1.7.0 adds audio operators, QDQ quantization, Model Building API, CUDA 11, and broader platform support. • Experimental audio operators — Fourier Transforms (`DFT`, `IDFT`, `STFT`), Windowing Functions (`Hann`, `Hamming`, `Blackman`), and `MelWeightMatrix` — added under the `com.microsoft.experimental` domain, buildable with the `ms_experimental` build flag. • Experimental Model Building API available under the `Microsoft.AI.MachineLearning.Experimental` namespace (included in the `Microsoft.AI.MachineLearning` NuGet package) for creating dynamic models on the fly with hardware-accelerated tensor featurization. • WinML `ImageFeatureDescriptor` runtimeclass gains a new `ImageNominalPixelRange` property exposing native support for pixel ranges `[0..1]` and `[-1..1]` in image models, alongside existing `BitmapPixelFormat` and `BitmapAlphaMode` properties. • Quantization tool now supports QDQ (`QuantizeLinear-DequantizeLinear`) format models. • Python transformer optimizer tool now supports fusion for BART models. • New CPU profiling tool (`profiler.py`) added for transformer models. • Python 3.8 and 3.9 support added across all platforms. • ARM32/64 Windows builds now included in the CPU NuGet and zip packages. • WinML gains .NET 5 support via .NET Standard 2.0 projections. • Custom operators now supported in the ONNX Runtime Mobile build. • Custom op support extended to dynamic input. • CUDA EP official GPU build now targets CUDA 11. • OpenVINO EP updated to support OpenVINO 2021.2 and adds OpenVINO EP options in the `onnxruntime_perf_test` tool. • TensorRT EP updated to support TensorRT 7.2. • DirectML EP updated to support DirectML 1.4.2 and adds PIX markers for operator-level profiling. • NNAPI EP adds support for per-channel quantization for `QlinearConv` and new operators `Min`, `Max`, and `Pow`. • Mobile build gains ability to reduce operator kernel types to only those required by target models, yielding a 25–33% binary size reduction in tested models (MobileNet v2, SSD MobileNet, MobileBERT). Breaking changes: • All ONNX Runtime CPU packages are now built without OpenMP by default; users requiring OpenMP must switch to the `Microsoft.ML.OnnxRuntime.OpenMP` NuGet package or the `onnxruntime-openmp` PyPI package. • Python 3.5 support has been removed from all platforms. • MKLML, openblas, and jemalloc build configurations have been removed. • Dependency on gemmlowp has been removed. • OpenVINO 2020.2 support has been deprecated in the OpenVINO EP.

ONNX Runtime v1.6.0

Fri, 11 Dec 2020 05:06:51 GMT

ONNX Runtime v1.6.0 adds opset 13, NNAPI Android acceleration, Apple Silicon support, new contrib ops, and expanded quantization. • Exposes `AddInitializer` API to share initializers between sessions, eliminating duplicate memory allocation when multiple sessions load models with common weights. • Adds Python binding for the `OrtValue` data structure, enabling allocation and management of CUDA device memory directly within ORT without third-party allocators. • Adds `OrtValue` instances as bindable inputs/outputs in the Python I/O Binding interface, including support for ORT-allocated device memory. • Adds new session option to disable denormal floating-point numbers on SSE3-capable CPUs, eliminating denormal-induced performance degradation without model retraining. • Adds support for ONNX 1.8 / opset 13. • Adds new contrib ops: `BiasSoftmax`, `MatMulIntegerToFloat`, `QLinearSigmoid`, `Trilu`. • Adds `LongformerAttention` CUDA operator for Longformer transformer model optimization. • ORT Mobile now supports NNAPI for accelerating model execution on Android devices. • Adds build support for Mac with Apple Silicon (CPU only). • Adds support for loading sparse tensor initializers in pruned models. • Adds support for setting execution priority of individual nodes. • Adds support for selection of cuDNN convolution algorithms. • Adds BERT model profiling tool at `onnxruntime/python/tools/transformers/profiler.py`. • Python optimizer adds support for additional transformer model families: openai-GPT, ALBERT, and FlauBERT. • Adds TensorRT EP experimental Int8 quantization support. • Adds per-channel `QuantizeLinear` and `DeQuantizeLinear` quantization support. • Adds LSTM quantization support. • Adds CNN quantization optimizations including u8s8 support and NHWC transformer in `QLinearConv`. • Adds C# support for float16 and bfloat16 data types. • DNNL EP updated from version 1.1.1 to 1.7. • NNAPI EP adds support for CNN models and additional operators: Resize, Flatten, Clip. • OpenVINO EP updated to OpenVINO 2021.1 with added multi-threaded inferencing, fp16 input type, multi-device plugin, hetero plugin, shared library build, and ARM64 build support. • DirectML EP updated from 1.3.0 to 1.4.0, now using the standalone DirectML NuGet package `Microsoft.AI.DirectML`. • Windows ML NuGet package now supports UWP applications targeting Windows Store deployment for both CPU and GPU. • Windows ML gains ability to bind `IIterable` as inputs and outputs and to create `Tensor*` via multiple buffers. • NoOpenMP build of ONNX Runtime now available on NuGet (`Microsoft.ML.OnnxRuntime.NoOpenMP`) and PyPI (`onnxruntime`) for C/C++/C#/Python users. • Removes nGraph EP; OpenVINO EP is the recommended replacement. Breaking changes: • The destructor of `OrtEnv` is now non-trivial and may perform DLL unloading — calling `ReleaseEnv` from `DllMain` or placing `OrtEnv` in global variables is no longer safe. • The nGraph EP has been removed; users must migrate to the OpenVINO EP.

ONNX Runtime v1.5.2

Thu, 15 Oct 2020 07:04:22 GMT

ONNX Runtime v1.5.2 adds trilinear sampling in Resize, flatbuffers verification for ORT format, and config keys header for Linux/Mac. • Adds flatbuffers verifier for ORT format buffers, enabling integrity validation of serialized models. • Adds config keys header file to Linux and Mac packages, exposing session configuration constants for native consumers. • Supports trilinear sampling mode in the `Resize` operator. • Removes the `cudnn64_7.dll` dependency from the GPU C# NuGet package, simplifying deployment on Windows.

ONNX Runtime v1.5.1

Tue, 29 Sep 2020 23:46:23 GMT

ONNX Runtime v1.5.1 adds mobile-optimized builds, IO Binding APIs, TensorRT engine caching, and expanded execution provider support. • Adds `ORT_TENSORRT_ENGINE_CACHE_ENABLE=1` environment variable to enable TensorRT engine caching, reducing model load time on repeated runs. • Adds Python API support for `RegisterCustomOpsLibrary`, enabling custom operator libraries to be registered at runtime. • Adds IO Binding API for C/C++/C# bindings, allowing use of pre-allocated buffers on targeted devices and specifying the target device for unknown output shapes. • Adds `IntraopNumThreads` and `SetNamedDimensionOverrides` APIs on `LearningModelSessionOptions` in Windows ML for fine-grained thread and dimension control. • Adds allocator sharing across multiple sessions, reducing per-session arena overhead in multi-session processes. • Adds Reduced Operator Kernel build support, enabling ORT binaries to be compiled with only the operators required by specific model(s). • Adds preview support for ORT on Mobile Platforms, minimizing build size for mobile and embedded devices. • Adds new static quantization operators: `QLinearMul`, `QLinearAdd`, `QLinearSigmoid`, and `QLinearLeakyRelu`. • Extends quantization tooling to support quantization-aware training models and models larger than 2 GB. • Adds Python 3.8 support for official Python packages (excluding Windows GPU and Linux ARM builds). • Adds CUDA 11 build-from-source support and upgrades the official package to CUDA 10.2 / cuDNN 8.0. • Adds TensorRT 7.1 support and delivers the TensorRT EP as a separate shared library DLL. • Adds OpenVINO 2020.4 support, VPU hardware selection runtime options, fast model compilation for VPU, and C# binding support for the OpenVINO EP. • Adds DirectML EP Python API (build from source) and C# NuGet package `Microsoft.ML.OnnxRuntime.DirectML`. • Adds 7 new DirectML operators for ONNX opset 12: `Celu`, `GreaterOrEqual`, `LessOrEqual`, `ArgMin`/`ArgMax` with `select_last_index`, `GatherND` with `batch_dim`, and `RoiAlign`. • Adds 1D–8D dimension support for DirectML `ElementWise*`, `Activation*`, `Reduce*`, `ArgMin`/`ArgMax`, `Gather*`, `Scatter*`, and `OneHot` operators. • Adds Android NNAPI EP support for dynamic input shapes, `Int32`/`float32`/`uint8` data types, `Uint8` static quantization, and CPU fallback for Android API level 26 and below, expanding supported operators to 36 (50% increase). • Adds MiGraphX EP operators: `GatherElements`, `NonZero`, `Equal`, and `Where`, plus Boolean data type support. • Adds Windows ML NuGet support for UWP (Windows Store, CPU only), .NET, and .NET Framework applications. • Adds Rust developer support for Windows ML, with sample and documentation available. • Adds Windows ML support for additional ONNX image type denotations: `Gray8`, normalized `[0..1]`, and normalized `[-1..1]`. • Enables telemetry in Java and Node.js packages for Windows builds via the Windows OS trace event system. Breaking changes: • macOS binaries now require OpenMP to be installed as an external dependency.

ONNX Runtime v1.4.0

Fri, 17 Jul 2020 20:24:43 GMT

ONNX Runtime v1.4.0 adds Python IOBinding APIs, new quantization ops, AMD MIGraphX and ARM NN execution providers, and CUDA Einsum support. • Adds Python IOBinding methods `bind_cpu_input`, `bind_output`, and `copy_outputs_to_cpu` to simplify benchmarking on non-CPU devices without manual allocation or copy via third-party allocators. • Adds quantization operators `QLinearAdd` and `QAttention` to the quantization operator set. • Introduces pre-packing for constant Matrix B in `DynamicQuantizeMatMul` and `QAttention` to improve CPU transformer quantization performance. • Adds CUDA execution support for the Einsum operator (opset 12). • Adds preview-tier AMD MIGraphX Execution Provider for AMD GPU inference. • Adds preview-tier ARM NN Execution Provider for ARM-based inference. • Adds opset 12 support in ONNX Runtime Training. • Telemetry is now enabled by default for Python packages and GitHub release zip files (C API).

ONNX Runtime v1.3.1

Tue, 16 Jun 2020 22:31:02 GMT

ONNX Runtime v1.3.1 publishes Java (CPU/GPU) and Node.js API packages to Maven and NPM. • Publishes Java CPU inference library `com.microsoft.onnxruntime/onnxruntime` to Maven Central. • Publishes Java GPU inference library `com.microsoft.onnxruntime/onnxruntime_gpu` to Maven Central. • Publishes `onnxruntime` Node.js API package to NPM.

ONNX Runtime v1.3.0

Tue, 19 May 2020 03:15:13 GMT

ONNX Runtime v1.3.0 adds training preview, new execution providers, Java/JS APIs, I/O binding for Python, and ONNX 1.7/opset 12 support. • Adds I/O binding support for the Python API, allowing inputs/outputs to be pre-staged on the GPU before model execution to significantly reduce execution time. • Adds API to specify free dimensions by both denotations and symbolic names. • Adds support for sessions to share a global threadpool, configurable via the ORT C APIs; when built with OpenMP, thread count is controlled via OpenMP environment variables. • Adds capability to build execution providers as shared library DLLs (initially supported for DNNL EP), enabling dynamic loading per model with no runtime cost when the EP is not loaded. • Adds OpenVINO v2.0 Execution Provider. • Adds preview Rockchip NPU Execution Provider (RKNPU EP). • Adds preview Xilinx FPGA Vitis-AI Execution Provider. • Promotes Windows Machine Learning APIs (`Microsoft.AI.MachineLearning`) to General Availability on NuGet. • Promotes ONNX Runtime DirectML package (`Microsoft.ML.OnnxRuntime.DirectML`) to General Availability on NuGet. • Promotes Java API to General Availability (Maven package forthcoming). • Adds preview JavaScript (Node.js) API, buildable from the master branch. • Adds ARM64 Linux CPU Python package, now available on PyPI. • Adds ONNX Runtime Training (preview) for accelerating transformer model training. • Adds ONNX 1.7 and opset 12 support, including function expansion for ops such as `NegativeLogLikelihoodLoss`, `GreaterOrEqual`, `LessOrEqual`, and `Celu` without requiring a kernel implementation. • Updates DirectML EP to support ONNX opset 10 and 11, and adds initial support for 8-bit and quantized operators. • Enables Windows telemetry collection. • Delivers ~10% average latency improvement across key models (ONNX Model Zoo, MLPerf, production models), with further improvements for Transformer models on CPU and GPU and for large-batch scikit-learn inference (`TreeEnsembleRegressor`, `TreeEnsembleClassifier`, `LinearRegressor`, `LinearClassifier`, `SVMRegressor`, `SVMClassifier`, `TopK`). Breaking changes: • This version requires the Visual C++ 2019 runtime; previous versions required Visual C++ 2017.

ONNX Runtime v1.2.0

Tue, 10 Mar 2020 22:22:23 GMT

ONNX Runtime v1.2.0 adds WinML APIs, TensorRT 7, new C API surfaces, quantized ops, and Android AAR builds. • Adds C API functions `GetDenotationFromTypeInfo`, `CastTypeInfoToMapTypeInfo`, `CastTypeInfoToSequenceTypeInfo`, `GetMapKeyType`, `GetMapValueType`, `GetSequenceElementType`, `ReleaseMapTypeInfo`, `ReleaseSequenceTypeInfo`, `SessionEndProfiling`, `SessionGetModelMetadata`, `ModelMetadataGetProducerName`, `ModelMetadataGetGraphName`, `ModelMetadataGetDomain`, `ModelMetadataGetDescription`, `ModelMetadataLookupCustomMetadataMap`, `ModelMetadataGetVersion`, and `ReleaseModelMetadata` for richer session introspection. • Adds new Contrib ops `QLinearAdd`, `QLinearMul`, `QLinearReduceMean`, `MulInteger`, and `QLinearAveragePool` for quantized inference workflows. • Adds `Int8` type support for the `Where` op. • Default value of `graph_optimization_level` in `SessionOptions` changed to `ORT_ENABLE_ALL` (99), and default `max_num_graph_transformation_steps` increased to 10. • Introduces a model opset number and IR version check — ONNX Runtime will now reject models with ONNX opset versions higher than those supported in the built version. • [Preview] Windows Machine Learning (WinML) WinRT APIs now available in Windows builds of ONNX Runtime, with DirectML execution provider for GPU acceleration; compatible with Windows 8.1 (CPU) and Windows 10 1709+ (GPU), distributed as `windows.ai.machinelearning.dll` via Nuget. • TensorRT Execution Provider upgraded to TRT 7. • CUDA updated to 10.1; Linux builds now require CUDA Runtime 10.1.243, `cublas10-10.2.1.243`, and CUDNN 7.6.5.32 (cublas 10.1.x is not supported); adds dependency on `curand` lib. • Adds capability to generate an ONNX Runtime Android Archive (AAR) file from source, importable directly into Android Studio. • Java API now available on Android; Gradle is now required to build the Java API. • Nuget package restructured to introduce a separate managed assembly `Microsoft.ML.OnnxRuntime.Managed` shared between CPU and GPU packages. • Build system now supports fetching dependencies from vcpkg (C++ package manager for Windows, Linux, and macOS). • Adds experimental featurizer operators as an expansion of Contrib operators (not part of the official build). • `OrtEnv` can now be created and destroyed multiple times within the same process. Breaking changes: • The default `graph_optimization_level` in `SessionOptions` is changed to `ORT_ENABLE_ALL` (99); existing code relying on the previous default will now apply full graph optimizations automatically. • ONNX Runtime will now reject models whose opset version is higher than the supported opset for the installed version — models previously loaded under the old forward-compatibility behavior may now fail to load. • Linux CUDA builds now require exactly CUDA Runtime 10.1.243, `cublas10-10.2.1.243`, and CUDNN 7.6.5.32; cublas 10.1.x versions will not work.

ONNX Runtime v1.1.0

Thu, 19 Dec 2019 22:07:52 GMT

ONNX Runtime v1.1.0 adds Android NNAPI and Java API previews, shared-library custom ops, and BERT/TensorRT performance upgrades. • Adds preview **NN API Execution Provider** for Android, enabling on-device inference via Android's Neural Networks API. • Adds preview **Java API** for ONNX Runtime, bringing Java application support to the runtime. • Custom ops can now be packaged in shared libraries and loaded at runtime, allowing distribution across multiple applications without modification. • Adds a Python tool that automatically maps a pandas DataFrame to ONNX graph inputs based on schema information, located at `onnxruntime/python/tools/featurizer_ops`. • TensorRT Execution Provider gains support for inputs with dynamic shapes. • Fused CPU kernels added for `Attention`, `EmbedLayerNormalization`, `SkipLayerNormalization`, and `FastGelu` operators to accelerate BERT model inference. • MKL-DNN (now renamed DNNL) updated to v1.1.

ONNX Runtime v1.0.0

Wed, 30 Oct 2019 06:27:47 GMT

ONNX Runtime v1.0.0 adds stable C ABI, four new execution providers, free dimension override, and new build/perf options. • Adds free dimension override, letting callers supply statically-unknown tensor shapes (e.g. batch size) before inference to unlock ahead-of-time optimizations. • Adds ability to set intra- and inter-operator thread counts independently to tune sequential vs. concurrent inferencing workloads. • Adds new cmake build options `onnxruntime_USE_GEMMLOWP`, `onnxruntime_USE_AUTOML`, and `onnxruntime_USE_DML` for controlling optional EP dependencies. • Adds `-y` flag to `onnxruntime_perf_tool` for controlling `inter_op_num_threads`; maximum optimization level is now `99` (previously `3`). • Adds new session option for serializing optimized ONNX models to disk. • Enables registration of execution providers through the Python API and setting additional run options through the C# API. • Stabilizes the C API at v1.0 with ABI compatibility and Semantic Versioning guarantees — programs linked against this release will not require re-linking for future releases. • Adds [Preview] NUPHAR execution provider, a TVM- and LLVM-based EP that JIT-compiles subgraph nodes into optimized functions for model acceleration. • Adds [Preview] DirectML execution provider for hardware-accelerated GPU inference via DirectX 12 on Windows across a broad range of hardware and drivers. • Adds [Preview] ARM Compute Library (ACL) execution provider targeting ARM CPUs and GPUs. • Adds [Preview] OpenVINO support for Intel Arria 10 FPGA. • Promotes OpenVINO EP to General Availability for Intel CPU, Integrated Graphics, Neural Compute Stick 2, and Movidius Myriad VPU targets. • Adds quantization tool support for selective node quantization, bias quantization for Conv nodes, and node fusion for dynamic quantization. • Adds ONNX 1.6 / opset 11 operator support on CPU, including Sequence ops. • Adds component-level telemetry via Windows Trace Logging for Windows builds. • Adds Gelu fusion optimizer for CPU/GPU inference. • Switches default thread pool to Eigen ThreadPool; adds ability to disable multi-threading by setting pool size to `1` and `onnxruntime_USE_OPENMP` to `OFF`. Breaking changes: • cmake options `onnxruntime_USE_MLAS` and `onnxruntime_USE_EIGEN_THREADPOOL` are removed; both are now always enabled and specifying them will break builds. • In `onnxruntime_perf_tool`, optimization level `3` is no longer valid; the maximum is now `99`. • Minimum supported gcc version is raised to 4.8.2.

ONNX Runtime v0.5.0

Thu, 01 Aug 2019 19:48:28 GMT

ONNX Runtime v0.5.0 adds C++ API, HTTP/gRPC inference server, Intel OpenVINO EP, Python custom operators, and CUDA 10 support. • Adds a C++ API (`onnxruntime_cxx_api.h`) for inferencing, wrapping the existing C API. • Introduces ONNX Runtime Server (Beta) for serving models over HTTP and gRPC endpoints. • Adds Python Operator (Beta) support to embed custom Python code in a single ONNX graph node, enabling experimentation with custom operators. • Intel OpenVINO Execution Provider is now available for Public Preview. • MKL-DNN Execution Provider now uses subgraph-based execution for improved performance. • Updates CUDA support to CUDA 10 for GPU inferencing with NVIDIA GPUs. • Base CPU Execution Provider gains faster convolution performance via NCHWc blocked layout, enabled by setting graph optimization level to `3` in session options. • All C APIs now return an `ORT` status code for consistency. • Adds support for Keras-based Mask R-CNN model inferencing.

ONNX Runtime v0.4.0

Thu, 02 May 2019 02:25:51 GMT

ONNX Runtime v0.4.0 adds Intel nGraph and NVIDIA TensorRT execution providers, ONNX 1.5/Opset 10 support, and quantization operators. • Adds Intel nGraph execution provider for accelerated inference on Intel hardware. • Adds NVIDIA TensorRT execution provider for GPU-accelerated inference. • Supports ONNX 1.5 and Opset 10 operators, enabling use of newly added ONNX Model Zoo object detection models (YOLO v3, SSD). • Adds quantization operators from ONNX 1.5. • Extends the C API for Custom Operators to support allocation of outputs during compute, includes a C++ wrapper to simplify implementation, and enables custom op DLLs when ONNX Runtime is compiled statically. • Adds graph optimization via Constant Folding for improved inference performance. • Publishes additional NuGet binaries built with MKL-ML. • Reduces binary sizes: 700 KB+ on Windows, 65% on Linux, 45% on macOS.

ONNX Runtime v0.3.0

Thu, 14 Mar 2019 18:17:00 GMT

ONNX Runtime v0.3.0 adds ONNX 1.4/Opset 9 support, custom op registration via C API, and new OS X/x86 build packages. • Adds custom op registration via the C API, enabling users to plug in their own operators at runtime. • Adds Non-Tensor type support for input/output in the C and C# APIs. • Supports ONNX 1.4 compatibility including Opset 9 operators and large models over 2 GB. • Adds new prebuilt packages: C/C# on OS X x64 CPU, C on Linux x86 CPU, and C on Windows x86 CPU. • Default CPU execution provider is now Eigen and MLAS (previously MKL-DNN). Breaking changes: • The default execution provider for CPU has changed from MKL-DNN to Eigen and MLAS; existing builds relying on MKL-DNN behavior must explicitly select it via build options. • OpenMP is now required for prebuilt binaries; setups without OpenMP installed will fail to run.

ONNX Runtime v0.2.1

Mon, 11 Feb 2019 21:54:43 GMT

ONNX Runtime v0.2.1 adds Linux C# packages, GPU support on Linux/Windows, and promotes the C API to Beta. • C# packages now available for Linux, with GPU support for both Windows and Linux. • C API promoted from experimental to Beta status, signaling expected stability for production use. • Adds MKL-DNN acceleration support for `Relu`, `Sum`, and `BatchNormalization` operators.