sarvam 30b FP8 dynamic Model Overview Model Architecture: sarvamai/sarvam 30b Input: Text Output: Text Model Optimizations: Weight quantization: FP8 Activation quantization: FP8 Out of scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Version: 1.0 Model Developers: RedHatAI ModelCar Storage URI: oci://registry.redhat.io/rhai/modelcar sarvam 30b fp8 dynamic:3.0 Validated on vLLM: 0.18.0 Validated on RHAIIS: 3.4 Validated on RHOAI: 3.4 This model is a quantized version of sarvamai/sarvam 30b. It was evaluated on several tasks to assess its quality in comparison to the unquantized model. Model Optimizations This model was obtained by quantizing the weights and activations of sarvamai/sarvam 30b to FP8 data type, ready for inference with vLLM. Only the weights and activations of the linear operators within transformers blocks are quantized using LLM Compressor. Deployment Use with vLLM This model can be deployed efficiently using the vLLM backend. 1. Install vLLM from main: 2. Run using vLLM vLLM also supports OpenAI compatible serving. See the documentation for more details. Creation This model was created by applying LLM Compress…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy