Table of Contents 0. TL;DR 1. Model Details 2. Usage 3. Training Details 4. Evaluation TL;DR Model Details Model Description Developed by: https://www.tii.ae Model type: Causal decoder only Architecture: Mamba Language(s) (NLP): Mainly English License: TII Falcon Mamba License 2.0 Check out the blogpost for more details! Usage Find below some example scripts on how to use the model in transformers (Make sure to have the latest transformers, or the one built from source): Using the Pytorch model Running the model on a CPU Click to expand Running the model on a GPU Click to expand Running the model on a GPU using torch.compile Click to expand Running the model on a GPU using different precisions FP16 Click to expand 4 bit Click to expand Training Details Training Data Falcon Mamba has been trained with ~ 5,500 GT mainly coming from Refined Web, a large volume web only dataset filtered and deduplicated. Similar to the others Falcon suite models, Falcon Mamba has been trained leveraging a multi stage training strategy to increase the context length from 2,048 to 8,192. Moreover, inspired by the concept of Curriculum Learning, we carefully selected data mixtures throughout the training…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy