FineWeb Edu classifier Model summary This is a classifier for judging the educational value of web pages. It was developed to filter and curate educational content from web datasets and was trained on 450k annotations generated by LLama3 70B instruct for web samples from FineWeb dataset. We used this classifier to build FineWeb Edu dataset. How to use in transformers To load the FineWeb Edu classifier, use the following code: Training The classifier was trained on 450,000 pairs of web samples and their scores from 0 to 5, generated by Llama3. The samples were annotated based on their educational quality with 0 being not educational and 5 being highly educational. Below is the prompt used for LLama3 annotations: We added a classification head with a single regression output to Snowflake arctic embed and trained the model for 20 epochs with a learning rate of 3e 4. During training, the embedding and encoder layers were frozen to focus on the classification head. The model achieved an F1 score of 82% when converted to a binary classifier using a score threshold of 3. Training Details: Model: Snowflake arctic embed with a classification head Dataset: 450,000 samples from Llama3 annotat…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy