FAISS-Based Novelty Detection for SmolLM and SmolLM2

This tutorial demonstrates how to the measure novelty of text queries with respect to the provided SmolLM and SmolLM2 pretraining corpora, with optional ColBERTv2 re-ranking for improved precision.

Overview

The pipeline consists of four main steps:

1. Generate Embeddings - Encode your queries using a sentence transformer
2. FAISS Search - Retrieve top-K most similar documents from the pretraining corpus
3. Combine Results - Merge results from multiple FAISS index parts
4. ColBERT Re-ranking (Optional) - Re-rank retrieved documents for better precision

Table of Contents

• Prerequisites
• Installation
• Directory Structure
• Usage
  • Step 1: Generate Query Embeddings
  • Step 2: Search FAISS Indices
  • Step 3: Combine FAISS Results
  • Step 4: ColBERT Re-ranking
• Experiment Configuration

Prerequisites

• Python 3.11 (required)
• CUDA-capable GPU (recommended)
• Pre-built FAISS indices and chunked datasets (see Data Download below)

Data Download

Due to Hugging Face storage quota limitations, the complete FAISS indices and chunked datasets (multiple terabytes) are distributed across two repositories:

• Repository 1: stai-tuebingen/faiss-smollm
• Repository 2: enguyen/smollm-chunked

You need to download data from both repositories to run the complete tutorial. After downloading, organize the files according to the directory structure shown below.

Installation

Using uv (recommended)

# Install uv if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create a virtual environment with Python 3.11
uv venv --python 3.11

# Activate the virtual environment
source .venv/bin/activate  # On Linux/macOS
# or
.venv\Scripts\activate  # On Windows

# Install PyTorch with CUDA support (check which version you need)
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128

# Install all other dependencies
uv pip install -e .

# Optional: Install GPU-accelerated FAISS
uv pip install faiss-gpu

Using pip

# Create a virtual environment with Python 3.11
python3.11 -m venv .venv
source .venv/bin/activate

# Install PyTorch with CUDA 12.8 support
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128

# Install all other dependencies
pip install -e .

# Optional: Install GPU-accelerated FAISS
pip install faiss-gpu

Note: PyTorch must be installed separately before the project dependencies to ensure CUDA compatibility.

Directory Structure

Expected FAISS Index Structure

FAISS_PATH_1/
├── dclm/
│   ├── faiss_part_0.index
│   ├── faiss_part_1.index
│   └── ...
├── stack_edu/
│   ├── faiss_part_0.index
│   └── ...
└── ...

FAISS_PATH_2/
├── finemath/
│   ├── faiss_part_0.index
│   ├── faiss_part_1.index
│   └── ...
├── infiwebmath/
│   ├── faiss_part_0.index
│   └── ...
└── ...

Expected Data Structure (for ColBERT)

DATA_PATH/
├── fineweb_edu_chunked/
│   ├── part_0/
│   ├── part_1/
│   └── ...
├── cosmopediav2_chunked/
│   ├── part_0/
│   ├── part_1/
│   └── ...
└── smollm2_corpus/
    ├── dclm_chunked/
    │   ├── part_0/
    │   ├── part_1/
    │   └── ...
    ├── stack_edu_chunked/
    │   └── ...
    └── ...

Results Directory Structure

RESULTS_PATH_PARTS/
├── SmolLM-360M_prompted_embeddings.npy
├── SmolLM2_seq_bat_dclm_part_0_S.npy
├── SmolLM2_seq_bat_dclm_part_0_I.npy
└── ...

RESULTS_PATH_COMBINED/
├── SmolLM-360M_prompted_S.npy
├── SmolLM-360M_prompted_I.npy
└── ...

COLBERT_RESULTS_PATH/
├── My_Experiment_SmolLM-360M_prompted_top100_chunk_texts_per_query.npy
├── My_Experiment_SmolLM-360M_chunk_size_150_simple_rerank_results_model.json
└── ...

Usage

Step 1: Generate Query Embeddings

Script: minimal_example_embeddings.py

This script generates embeddings for your queries using the GIST-large-Embedding model.

# Configure your experiments
EXPERIMENT_SETUPS = [("SmolLM-360M", "prompted")]
RESULTS_PATH_PARTS = "path/to/your/FAISS_results/save/folder"

# Define your queries
query_1 = "Your first query text here"
query_2 = "Your second query text here"

Run:

python minimal_example_embeddings.py

Output:

• {model_name}_{prompted_or_not}_embeddings.npy - Shape: (NUM_QUERIES, EMBEDDING_DIM)

Step 2: Search FAISS Indices

Script: minimal_example_FAISS.py

This script searches FAISS indices to find the top-K most similar documents for each query. For maximum parallelization on a HPC cluster, it is optimized to take one JOB_NUM and only process one .index file of the FAISS index at a time. Pass JOB_NUM via the command line and start one job per FAISS index file for the most efficient execution of this step.

Configuration:

FAISS_PATH_1 = "path/to/your/first/FAISS/folder"
FAISS_PATH_2 = "path/to/your/second/FAISS/folder"
RESULTS_PATH_PARTS = "path/to/your/FAISS_results/save/folder"
TOP_K = 100
EXPERIMENT_SETUPS = [("SmolLM-360M", "prompted")]

# Filter by dataset names (optional)
INCLUDE_DATASETS = ["cosmopediav2", "fineweb_edu", "python_edu"]

# For HPC parallelization: set to integer to process only that FAISS file index
# Set to None to process all files sequentially
JOB_NUM = 0

Run:

python minimal_example_FAISS.py

Output:

• SmolLM2_seq_bat_{dataset_name}_{file_name}_S.npy - Similarity scores, shape: (NUM_EXPERIMENTS * NUM_QUERIES, TOP_K)
• SmolLM2_seq_bat_{dataset_name}_{file_name}_I.npy - Index strings (format: "{dataset}-{part}-{idx}"), shape: (NUM_EXPERIMENTS * NUM_QUERIES, TOP_K)

Step 3: Combine FAISS Results

Script: minimal_example_combine_FAISS.py

This script combines results from multiple FAISS index parts, keeping only the top-K across all parts.

Configuration:

RESULTS_PATH_PARTS = "path/to/your/FAISS_results/save/folder"
RESULTS_PATH_COMBINED = "path/to/your/folder/for/combined/results"
NUM_QUERIES = 2
TOP_K = 100
EXPERIMENT_SETUPS = [("SmolLM-360M", "prompted")]

Run:

python minimal_example_combine_FAISS.py

Output:

• {model_name}_{prompted_or_not}_S.npy - Combined similarity scores, shape: (NUM_QUERIES, TOP_K)
• {model_name}_{prompted_or_not}_I.npy - Combined index strings, shape: (NUM_QUERIES, TOP_K)

Step 4: ColBERT Re-ranking (Optional)

Script: minimal_example_ColBERTv2.py

This script re-ranks the top-K FAISS results using ColBERTv2 for improved precision.

Configuration:

RESULTS_PATH_COMBINED = "path/to/your/folder/for/combined/results"
COLBERT_RESULTS_PATH = "path/to/save/colbert/results"
DATA_PATH = "path/to/your/data/folder"

NUM_QUERIES = 2
TOP_K = 100
COLBERT_TOP_K = 10
CHUNK_SIZE = 100  # adjust between 50-512 to analyze length effects

# Experiment configuration
model_name = "SmolLM-360M"
prompted_or_not = "prompted"
EXPERIMENT_NAME = "My_Experiment"

Run:

python minimal_example_ColBERTv2.py

Output:

• {EXPERIMENT_NAME}_{model_name}_chunk_size_{CHUNK_SIZE}_simple_rerank_results_model.json - ColBERT re-ranking results in JSON format

Experiment Configuration

The experiment configuration shown here is a simplification for demonstration purposes and does not match the one used in the paper exactly. Depending on the variables you are analyzing, you should adjust how experiments are defined and the file naming conventions, accordingly.

Single Experiment

EXPERIMENT_SETUPS = [("SmolLM-360M", "prompted")]
NUM_QUERIES = 2

This will:

• Process 2 queries for the SmolLM-360M model with prompting
• Generate embeddings of shape (2, 1024)
• FAISS results of shape (2, 100)

Multiple Experiments

EXPERIMENT_SETUPS = [
    ("SmolLM-360M", "prompted"),
    ("SmolLM-360M", "unprompted"),
    ("SmolLM-1.7B", "prompted")
]
NUM_QUERIES = 2

This will:

• Process 2 queries × 3 experiments = 6 total query-experiment combinations
• Stack embeddings to shape (6, 1024) before FAISS search
• FAISS results of shape (6, 100)
• Combine script slices back into per-experiment results of shape (2, 100) each

Citation

If you use this code or data in your research, please cite:

@misc{davydov2025unattributabilitycomputingnoveltyretrieval,
      title={Un-Attributability: Computing Novelty From Retrieval & Semantic Similarity}, 
      author={Philipp Davydov and Ameya Prabhu and Matthias Bethge and Elisa Nguyen and Seong Joon Oh},
      year={2025},
      eprint={2510.27313},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2510.27313}, 
}