LiteRT Release Notes: Version 2.2.0Executive Summary & Scope 🌟 Release 2.2.0 Summary 1. LiteRT RuntimeOverview: Core C/C++ inference engine, delegate interfaces, platform APIs (Swift, Python, C), memory management, and hardware acceleration plugins.Old Revision: v2.1.6 New Revision: v2.2.0Repository: google-ai-edge/LiteRT🤖 Model/ Operator CoverageXNNPACK supports for fp16 DEPTHWISE_CONV_2D, CONV_2D, FULLY_CONNECTED, and BATCH_MATMUL⚡ Performance and MemoryAdded a new precision mode, kLiteRtDelegatePrecisionFp16WithFp32Accum.Currently it is GPU only. When enabled the GPU backend will use FP16 storage and arithmetic with FP32 accumulation for CONV_2D, DEPTHWISE_CONV_2D, FULLY_CONNECTED, TRANSPOSE_CONV and BATCH_MAT_MUL operators , which can provide a midground between pure 32-bit and pure 16-bit computation.Prepares input on GPU for the models which don’t run embedding lookup on CPU, including Gemma3 or before, and Open weight models like Qwen3Added ROPE composite op for GPU to improve the performance for LLMs.The decode performance increased ~5%For best GPU performance, weights are re-arranged in batches on the GPU to be the optimal format for execution. In this release, the batch size was changed to scale with the total size of the model to reduce memory usage and improve initialization time. The impact will vary depending on model and device type but Gemma 4 E2B on a Linux NVIDIA machine saved ~200MB.🛠️ Hardware Acceleration[Experimental] New CPU accelerator YNNPACK can be enabled through the compilation flag: --define litert_enable_ynnpack=true and runtime flag: --enable_ynnpack=true. Improved MTK NPU backend's channelwise quantization support for classical vision models, such as resnet18 and mobilenet.Added support for Intel NPU (LunarLake and PantherLake), enabling both JIT and AOT compilation. Verified on a variety of classic and selected GenAI models (Gemma4 2B instructions) on Windows and Linux.LiteRT QAIRT acceleratorMajor Features and ImprovementsSupported specifying the Qualcomm SoC using either the SoC name or numeric SoC model.Aligned SoC coverage with QAIRT SDK 2.47.Refined OP validation and debugging logs.Added documentation for QNN Native Path, QC options, and CMake toolchain builds.Improved compiler plugin to avoid redundant QNN library reloads.Enhanced OP coverage: A8W2 Fully Connected, BatchToSpaceNd, and SpaceToBatchNd.Bug Fixes and Other ChangesSkipped custom op delegates when the custom op package is not registered.🔌 APIs & Platform supportABI version information has been added to data structures to enhance backward compatibility for data types used in interfaces between key LiteRT modules (libLiteRt.so, GPU Accelerators, Dispatch, Compiler). Once this compatibility feature is complete, the system will operate without issues even when using different 2.2.x version binaries together.The new LiteRT Rust binding has been open-sourced. The google-ai-edge-litert crate was published on crates.ai and can be used with the standard Rust Cargo build manager. The binding is built on top of the LiteRT C++ Binary SDK.Introduced new SetSelectedSignatures API Added two new tensor element data type: F8E4M3FN and F8E5M2.LiteRt.js API added support for streamed weight loading Dynamic Range Quantized models to WebGPU.Tensor API:Added RegisterCustomOpParser and MixinRegistrar to map custom flatbuffer operators (CUSTOM, STABLEHLO_COMPOSITE) to typed LiteRT operations and dynamically bind backend execution logic.Added FeedbackLoopConfig to automate recurrent KV cache swapping and zero-copy buffer bindings between execution steps without requiring manual per-iteration re-binding.Added Tensor API python support, extended with transformer operation surfaces, explicit Q/DQ export, quantized TFLite lowering, and Safetensors loading with compressed weight ingestion (qb4w).Added JIT compilation and tracing support (compile() / WrittenModel) for authoring and executing custom tensor graphs directly in LiteRT JS.🐛 Security, Vulnerability and Stability Added integer overflow checks to convolution, reshape and pad-like CPU kernels, Stricter input shape restriction in Conv3DTranspose: older versions of LiteRT also allowed this op’s number of output channels to be a multiple of the number of filter channels, but that this case was never implemented correctly and is no longer permitted.Fixed a transitivity violation in TopK comparator when NaN is present. NaNs are now treated as smaller than every non-NaN value, For example, for input [1, 2, 3, NaN] with k=3, the output should be [1,2,3].Enforced that the 1D size operand (operand 1) for ResizeBilinear and ResizeNearestNeighbor TFLite ops is dynamic or has dimension length 2Added initial fuzzing test based on https://github.com/google/fuzztestAccelerator Test Suite (ATS) improvements over all accelerator backends:Expanded Operator & Subgraph coverage: Added test graph generators for Pad, Concatenation, Softmax, Transpose, BatchMatmul, FullyConnected, Reshape, Reduction, BinaryBroadcast, and OneHot, alongside a compound TransformerLayer generator built with the litert::tensor C++ API.Advanced Conformance and Precision Verification: Introduced the ConformanceSpec multi-criteria validation framework (MSE, exact match, quantized buckets, accumulation tolerance), expanded FullyConnected quantization with dynamic/shuffled weights, and enabled FP16 and mixed-precision delegate testing.Harness Performance and Infrastructure: Optimized test registration to be ~30x faster (reducing filter setup time from ~170s to ~6s), added full delegation hints for GPU/CPU to prevent premature validation errors, and lazily initialized test buffers to significantly reduce log noise during test execution.A fix was submitted that was preventing the GPU weight cache from being utilized.2. LiteRT TorchOverview: Enables PyTorch model conversion to LiteRT and provides specialized execution pipelines for vision and generative models.Old Revision: v0.9.1 (Commit 49d68fc)New Revision: ? (Commit ?)Repository: google-ai-edge/litert-torch🔹 Vision & Generative PipelinesGemma 4 architecture export support.ASR model support in export_hf pipelineDynamic context length support with --enable_gpu_dynamic_prefill, --enable_gpu_dynamic_cache flags.Automate stop token prefix expansion and infer additional stop tokens from chat template.LiteRT-LM ExecutorMetadata for better handling of sliding window attention and non SDPA attentions.Experimental features:Support exporting models with optimized GPU attention custom op.Fp16/Fp32 mixed precision support for GPU with --experimental_use_mixed_precision and --externalize_embedderJinja template transpilation for better LiteRT-LM support with --experimental_transpile_chat_template_for_minijinja🔹 NPU & Acceleration Passes[Experimental] Added bring-your-own-model NPU export colab in export_hf pipeline.Added sampling executor supports exported model inference.Added calibration and quantization tool for post-training/static range quantization.Added LiteRT-LM NPU compiler tool for compiling LiteRT-LM to chosen NPU backend.🔹 Converter Legalizations & DialectsAdd 1D transposed convolution and FFT lowering support.Add support for torch.uint8 dtype.🔹 Core Bug FixesFix bug in InlineConstsContext.constant_cache key lookup failed to distinguish between PyTorch tensors that share the same data_ptr but use different offsets.Fix BPE tokenizers with Metaspace pre-tokenizer and BPE chars to fix de-tokenization of BPE tokens3. LiteRT QuantizerOverview: Manages model compression, low-bit weight quantization, precision formats, and quantization transformations.Old Revision: v0.4.2 (Commit 96dcc54)New Revision: v0.9.0 (Commit 8d9c7a9)Repository: google-ai-edge/ai-edge-quantizer🔹New FeaturesQuantization Algorithms & RecipesGPTQ algorithm: Added GPTQ supportBlockwise Quantization & Scale Recovery: Implemented scale recovery and blockwise quantization in weight recovery algorithms, providing ready-to-use 4-bit and 8-bit named recipes2-Bit Quantization: Added recipes and buffer packing for 2-bit model quantizationHadamard Transformations: Added int4 matrix multiplication Hadamard transforms, matrix caching, max_hadamard_size control, forced all Hadamard matrices of the same size to utilize single tensor, changed all inserted Hadamard matrices to floatCalibrationProfiler-Based Calibration: Added CALIBRATION_PROFILER_BASED calibration mode to Calibrator and Quantizer, enabling faster and more memory efficient XNNPACK-assisted calibrationCalibrationInterpreter API: Added new calibration API with CalibrationInterpreter allowing to reuse inference pipeline for calibrationMetadata: Added calibration metadata trackingEvaluation & ValidationIntroduced multi-metric model validation supportProgress Bars & Reporting: Added progress bars and reports for quantization steps and calibration passesTooling & Model FormatsCommand-Line Interface: Introduced the aeq.py CLI for running model quantization workflows directly from the terminalLiteRT-LM Integration: Added support for reading, quantizing, and writing .litertlm files, extracting serialized TFLite models, and applying recipes across TFLiteModel sections🔹Performance & Memory OptimizationsZero-Copy & In-Place Operations: Removed unnecessary deep copies, replaced array copies with zero-copy array views and refactored core quantization operations to execute in-place, significantly lowering peak memory usageMemory-Mapped I/O (mmap): Replaced RAM-heavy model serialization with disk-backed memory mappings (mmap_utils) and chunk-by-chunk file writing, preventing OOM errors on large LLMsImmediate Memory Reclamation: Added explicit garbage collection for floating-point and intermediate quantized tensor buffers immediately after output buffer writingChunked Quantization: Processed large tensors in chunks along the leading dimension to avoid generating large intermediate valuesBuffer Deduplication: Pre-computed producer/consumer ops, tracked buffer mapping IDs, and pruned empty buffers to shrink serialized file size and parsing overhead🔹Packaging & Dependency ChangesUpdated the build and packaging system to use pyproject.toml and uv🔹Bug FixesFixed crashes in float casting operations when encountering non-constant weightsPrevented accidental double-quantization of input tensorsFixed an issue where output tensors were inadvertently rotated during fully connected op processing4. LiteRT-LMOverview: Large Language Model (LLM) execution engine providing tokenization, pre-processing, dynamic runners, and performance tuning for models like Gemma 3.Old Revision: v0.14.0 (Commit 80f301f)New Revision<span style="font-size:11pt;font-family:Arial,sans-serif;color:#000000;background-color:transparent;font-
…