Chonky modernbert base v1 Chonky is a transformer model that intelligently segments text into meaningful semantic chunks. This model can be used in the RAG systems. Model Description The model processes text and divides it into semantically coherent segments. These chunks can then be fed into embedding based retrieval systems or language models as part of a RAG pipeline. ⚠️This model was fine tuned on sequence of length 1024 (by default ModernBERT supports sequence length up to 8192). How to use I've made a small python library for this model: chonky Here is the usage: But you can use this model using standart NER pipeline: Training Data The model was trained to split paragraphs from the bookcorpus dataset. Metrics Token based metrics: Metric Value F1 0.79 Precision 0.83 Recall 0.75 Accuracy 0.99 Hardware Model was fine tuned on single H100 for a several hours I dedicate this model in memory of my father.
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy