Chonky distilbert base (uncased) v1 Chonky is a transformer model that intelligently segments text into meaningful semantic chunks. This model can be used in the RAG systems. Model Description The model processes text and divides it into semantically coherent segments. These chunks can then be fed into embedding based retrieval systems or language models as part of a RAG pipeline. How to use I've made a small python library for this model: chonky Here is the usage: But you can use this model using standart NER pipeline: Training Data The model was trained to split paragraphs from the bookcorpus dataset. Metrics Metric Value F1 0.7 Precision 0.79 Recall 0.63 Accuracy 0.99 Hardware Model was fine tuned on 2x1080ti I dedicate this model in memory of my father.
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy