PIT 4B — Point In Time GPT (Pre trained, 2020 12) Point In Time (PIT) is a family of GPT style language models trained on chronologically ordered monthly snapshots of FineWeb. Each checkpoint captures the state of knowledge available up to a specific month, making them suitable for temporal reasoning and point in time analysis tasks. This is the base (pre trained only) variant. For instruction following, use the corresponding PIT 4B FT checkpoint. Model details Property Value Snapshot month 2020 12 Architecture Decoder only Transformer (GPT) Layers 20 Hidden dim 4096 Attention heads 32 Vocab size 50304 Tokenizer GPT 2 BPE Position encoding RoPE Normalization RMSNorm on Q/K + pre norm Activation Squared ReLU Weight tying Yes (input emb ↔ lm\ head) Requirements Quick start Text generation Temporal reasoning example Because this model was trained on data up to 2020 12 , it reflects the world as it was known at that point. You can use this for point in time analysis: Weights format Weights are stored in safetensors format ( model.safetensors ) — memory mapped, fast to load, and safe (no arbitrary code execution). Limitations Knowledge is limited to web text available up to 2020 12 . No…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy