Qwen3 4B LiteRT LM Models This repository contains LiteRT LM variants of Qwen/Qwen3 4B optimized for on device text generation. Available Artifacts File Quantization Context Size : : qwen3 4b channelwise int8 float32kv.litertlm channel wise INT8 weights, float32 KV 5.28 GB qwen3 4b mixed int4.litertlm TorchAO mixed INT4, float KV 2048 2535.88 MiB Conversion Notes The mixed INT4 .litertlm artifact was produced with a TorchAO based quantize first recipe from the original Hugging Face checkpoint. This is a mixed quantization layout rather than a uniform all INT4 model: eligible linear projection weights are stored as blockwise INT4 with group size 32 and floating point scales, token embedding weights use weight only INT8 quantization, and normalization/reduction paths plus KV cache tensors remain floating point. The mixed INT4 bundle also uses LiteRT LM StableHLO composite ops for attention/cache execution, including odml.runtime bmm and odml.cache update . Performance Desktop benchmark: AMD Radeon AI PRO R9700, LiteRT LM WebGPU, 256 prefill tokens, 32 decode tokens. Android rows use LiteRT LM v0.13.1 with GPU OpenCL, 256 prefill tokens, and 64 decode tokens. Values report the warmed…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy