About this repository (it just works fork) This is a clone of NovaSearch/stella en 1.5B v5 with the weights pre converted to bfloat16 and modeling qwen.py patched to keep working on modern transformers releases. Original model card below. Modifications for transformers = 5 transformers 5.x instantiates models on the meta device, which broke this model in two independent, silent ways (no errors — just garbage embeddings): 1. Rotary caches scrambled — the non persistent inv freq / cos / sin buffers computed in init are materialized from the meta device as uninitialized memory and were never recomputed, corrupting every position. Fixed in modeling qwen.py : the rotary caches are now rebuilt lazily on the first real forward pass. No behavior change on transformers 4.x. 2. Checkpoint weights never load — the 5.x loader reports a clean load (0 missing keys) for this old style remote code class but leaves every parameter randomly initialized. This cannot be fixed from the modeling file. Until it is fixed upstream in transformers , reload the state dict manually after loading: Sanity check after loading: encode two related and one unrelated sentence — the related pair must clearly out scor…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy