Added 14
- Published google-ai-edge-litert crate on crates.io enabling C++ SDK capabilities in Rust
- Added ABI version tracking to key modules (libLiteRT.so, GPU accelerator, Dispatch, Compiler) for smoother upgrades across 2.2.x binary versions
- Added LiteRT Tensor API support and features for recurrent KV cache swapping in LiteRT.js
- Added JIT/tracing support for custom tensor graphs in LiteRT.js
- Added native support for F8E4M3FN and F8E5M2 tensor element data types
- Added full JIT/AOT support for Intel NPU (LunarLake and PantherLake)
- Added new CPU accelerator YNNPACK that can be enabled through compilation and runtime flags
- Added ROPE composite op for GPU to improve LLM decode performance (~5% faster)
- Introduced new SetSelectedSignatures API
- Added RegisterCustomOpParser and MixinRegistrar to map custom flatbuffer operators to typed LiteRT operations
- Added FeedbackLoopConfig to automate recurrent KV cache swapping and zero-copy buffer bindings
- Added Tensor API python support with transformer operation surfaces, explicit Q/DQ export, and Safetensors loading
- Added JIT compilation and tracing support (compile() / WrittenModel) for custom tensor graphs in LiteRT.js
- Added integer overflow checks to convolution, reshape and pad-like CPU kernels
Changed 5
- Added new precision mode kLiteRtDelegatePrecisionFp16WithFp32Accum for GPU backend using FP16 storage and arithmetic with FP32 accumulation
- Changed batch size for GPU weight re-arrangement to scale with total model size for improved memory usage and initialization time
- Improved MTK NPU backend's channelwise quantization support for classical vision models
- Aligned Qualcomm QAIRT SoC coverage with SDK 2.47
- Added LiteRT.js API support for streamed weight loading Dynamic Range Quantized models to WebGPU
From LiteRT
LiteRT Release Notes: Version 2.2.0
Executive Summary & Scope
API and Platform updates:
- Rust Integration - Published google-ai-edge-litert crate on crates.io enabling C++ SDK capabilities in Rust
- ABI Compatibility - Added ABI version tracking to key modules (libLiteRT.so, GPU accelerator, Dispatch, Compiler) facilitating smoother upgrades across 2.2.x binary versions.
API Updates:
- LiteRT.js - LiteRT Tensor API support and features for recurrent KV cache swapping; and JIT/tracing support for custom tensor graphs.
- (check: do we call out SetSelectedSignatures?)
- New Data Type - Added native support for for F8E4M3FN and F8E5M2 tensor elements New HW support
- Intel NPU - Added full JIT/AOT support for LunarLake and Panther Lake
- Qualcomm - QAIRT : Improved SoC coverage with SDK 2.47
- MediaTek - Improvements to channelwise quantization support
- CPU (Experimental) : New YNNPack accelerator Performance and Model optimizations
- LLM throughput improvements - added ROPE composite Ops for GPU, resulting in ~5% faster decode performance
- Multiple optimizations for memory efficiency and GPU execution Validation / Testing Infrastructure
- Faster test registration
- New multi-criteria validation framework (ConformanceSpec) for mixed-precision verification.
🌟 Release 2.2.0 Summary
1. LiteRT Runtime
Overview: Core C/C++ inference engine, delegate interfaces, platform APIs (Swift, Python, C), memory management, and hardware acceleration plugins.
- Old Revision: v2.1.6
- New Revision: v2.2.0
- Repository: google-ai-edge/LiteRT
🤖 Model/ Operator Coverage
XNNPACK supports for fp16 DEPTHWISE_CONV_2D, CONV_2D, FULLY_CONNECTED, and BATCH_MATMUL ⚡ Performance and Memory
- Added a new precision mode, kLiteRtDelegatePrecisionFp16WithFp32Accum.Currently it is GPU only. When enabled the GPU backend will use FP16 storage and arithmetic with FP32 accumulation for CONV_2D, DEPTHWISE_CONV_2D, FULLY_CONNECTED, TRANSPOSE_CONV and BATCH_MAT_MUL operators , which can provide a midground between pure 32-bit and pure 16-bit computation.
- Prepares input on GPU for the models which don’t run embedding lookup on CPU, including Gemma3 or before, and Open weight models like Qwen3
- Added ROPE composite op for GPU to improve the performance for LLMs. The decode performance increased ~5%
- For best GPU performance, weights are re-arranged in batches on the GPU to be the optimal format for execution. In this release, the batch size was changed to scale with the total size of the model to reduce memory usage and improve initialization time. The impact will vary depending on model and device type but Gemma 4 E2B on a Linux NVIDIA machine saved ~200MB.
🛠️ Hardware Acceleration
- [Experimental] New CPU accelerator YNNPACK can be enabled through the compilation flag:
--define litert_enable_ynnpack=trueand runtime flag:--enable_ynnpack=true. - Improved MTK NPU backend's channelwise quantization support for classical vision models, such as resnet18 and mobilenet.
- Added support for Intel NPU (LunarLake and PantherLake), enabling both JIT and AOT compilation. Verified on a variety of classic and selected GenAI models (Gemma4 2B instructions) on Windows and Linux.
- LiteRT QAIRT accelerator
- Major Features and Improvements
- Supported specifying the Qualcomm SoC using either the SoC name or numeric SoC model.
- Aligned SoC coverage with QAIRT SDK 2.47.
- Refined OP validation and debugging logs.
- Added documentation for QNN Native Path, QC options, and CMake toolchain builds.
- Improved compiler plugin to avoid redundant QNN library reloads.
- Enhanced OP coverage: A8W2 Fully Connected, BatchToSpaceNd, and SpaceToBatchNd.
- Bug Fixes and Other Changes - Skipped custom op delegates when the custom op package is not registered.
- Major Features and Improvements
🔌 APIs & Platform support
- ABI version information has been added to data structures to enhance backward compatibility for data types used in interfaces between key LiteRT modules (libLiteRt.so, GPU Accelerators, Dispatch, Compiler). Once this compatibility feature is complete, the system will operate without issues even when using different 2.2.x version binaries together.
- The new LiteRT Rust binding has been open-sourced. The google-ai-edge-litert crate was published on crates.io and can be used with the standard Rust Cargo build manager. The binding is built on top of the LiteRT C++ Binary SDK.
- Introduced new
SetSelectedSignaturesAPI - Added two new tensor element data type: F8E4M3FN and F8E5M2.
- LiteRt.js API added support for streamed weight loading Dynamic Range Quantized models to WebGPU.
- Tensor API:
- Added RegisterCustomOpParser and MixinRegistrar to map custom flatbuffer operators (CUSTOM, STABLEHLO_COMPOSITE) to typed LiteRT operations and dynamically bind backend execution logic.
- Added FeedbackLoopConfig to automate recurrent KV cache swapping and zero-copy buffer bindings between execution steps without requiring manual per-iteration re-binding.
- Added Tensor API python support, extended with transformer operation surfaces, explicit Q/DQ export, quantized TFLite lowering, and Safetensors loading with compressed weight ingestion (qb4w).
- Added JIT compilation and tracing support (compile() / WrittenModel) for authoring and executing custom tensor graphs directly in LiteRT JS.
🐛 Security, Vulnerability and Stability
- Added integer overflow checks to convolution, reshape and pad-like CPU kernels,
- Stricter input shape restriction in Conv3DTranspose: older versions of LiteRT also allowed this op’s number of output channels to be a multiple of the number of filter channels, but that this case was never implemented correctly and is no longer permitted.
- Fixed a transitivity violation in TopK comparator when NaN is present. NaNs are now treated as smaller than every non-NaN value, For example, for input [1, 2, 3, NaN] with k=3, the output should be [1,2,3].
- Enforced that the 1D size operand (operand 1) for ResizeBilinear and ResizeNearestNeighbor TFLite ops is dynamic or has dimension length 2
- Added initial fuzzing test based on https://github.com/google/fuzztest
- Accelerator Test Suite (ATS) improvements over all accelerator backends:
- Expanded Operator & Subgraph coverage: Added test graph generators for Pad, Concatenation, Softmax, Transpose, BatchMatmul, FullyConnected, Reshape, Reduction, BinaryBroadcast, and OneHot, alongside a compound TransformerLayer generator built with the litert::tensor C++ API.
- Advanced Conformance and Precision Verification: Introduced the ConformanceSpec multi-criteria validation framework (MSE, exact match, quantized buckets, accumulation tolerance), expanded FullyConnected quantization with dynamic/shuffled weights, and enabled FP16 and mixed-precision delegate testing.
- Harness Performance and Infrastructure: Optimized test registration to be ~30x faster (reducing filter setup time from ~170s to ~6s), added full delegation hints for GPU/CPU to prevent premature validation errors, and lazily initialized test buffers to significantly reduce log noise during test execution.
- A fix was submitted that was preventing the GPU weight cache from being utilized.