FAISS Based Novelty Detection for SmolLM and SmolLM2 This tutorial demonstrates how to the measure novelty of text queries with respect to the provided SmolLM and SmolLM2 pretraining corpora, with optional ColBERTv2 re ranking for improved precision. Overview The pipeline consists of four main steps: 1. Generate Embeddings Encode your queries using a sentence transformer 2. FAISS Search Retrieve top K most similar documents from the pretraining corpus 3. Combine Results Merge results from multiple FAISS index parts 4. ColBERT Re ranking (Optional) Re rank retrieved documents for better precision Table of Contents Prerequisites Installation Directory Structure Usage Step 1: Generate Query Embeddings Step 2: Search FAISS Indices Step 3: Combine FAISS Results Step 4: ColBERT Re ranking Experiment Configuration Prerequisites Python 3.11 (required) CUDA capable GPU (recommended) Pre built FAISS indices and chunked datasets (see Data Download below) Data Download Due to Hugging Face storage quota limitations, the complete FAISS indices and chunked datasets (multiple terabytes) are distributed across two repositories: Repository 1 : stai tuebingen/faiss smollm Repository 2 : enguyen/smoll…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy