polish roberta 8k A Polish language model built on the RoBERTa architecture, supporting context length of up to 8192 tokens. Encoder type models can be fine tuned to solve various text prediction tasks such as classification, regression, sequence tagging, or retrieval. In such tasks, they are usually faster and more parameter efficient than LLMs. This model was initialized from the polish roberta large v2 checkpoint, originally supporting 512 token context, and then adapted to handle longer texts. The model training process was as follows: In the first stage, the positional embedding layer was extended from 512 to 8192 tokens. The model was then trained on a corpus of approximately 150 billion tokens for one epoch. All model weights were frozen, and only the positional embedding layer was trained. The goal of this stage was to adapt the new layer without making drastic changes to the rest of the model's weights. In the second stage, all model weights were trained for 4 epochs. To improve training efficiency, we added support for Flash Attention 2 and contamination free packing. Documents were packed into sequences of exactly 8192 tokens. Since the model's ability to solve both shor…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy