Vanilla-Skip RWKV-7 — BabyLM 2026 Strict-Small

Architecture: RWKV-7 (~28M parameters) — causal CLM, zero auxiliary objectives
Task: BabyLM 2026 Strict-Small track — primary competitive submission
Training corpus: 14.87M tokens (BabyLM Strict-Small 10M-word corpus, 32K BPE)
Inference mode: CLM-only forward pass
Submission file: all_full_preds_and_fast_scores_causal.json (in this repo)

Results (BabyLM 2026 official evaluation)

Checkpoint trajectory (BLiMP and ET across 19 checkpoints, step 500–18k):

Key Finding

This model is the ablation result from Halved CLM Exposure Mitigates Late-Training Collapse in Small Recurrent Language Models (BabyLM 2026 submission). It uses pure RWKV-7 CLM with zero auxiliary objectives, trained by skipping every other gradient step (odd steps advance the LR schedule but skip the update). Despite having no auxiliary objectives, this step-skipping strategy outperforms BiRWKV-7 (which uses MLM + MTP + n-gram features) on BLiMP across all three seeds, demonstrating that reduced CLM update pressure — not auxiliary objectives — prevents late-training collapse.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "yashmahe2018/vanilla-skip-rwkv7-strict-small-babylm2026",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "yashmahe2018/vanilla-skip-rwkv7-strict-small-babylm2026",
    trust_remote_code=True
)

inputs = tokenizer("The cat sat on the", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0]))

Each training checkpoint is available as a separate branch (chck_1M through chck_100M, following BabyLM 2026 strict-small checkpoint naming).

Architecture

• 8 layers, hidden dim 384, 6 attention heads (head size 64), 32K BPE vocabulary
• Pure RWKV-7 forward CLM — no MLM, no MTP, no n-gram features, no backward pass
• Step-skipping: odd steps advance LR schedule, zero gradient update (9k effective CLM steps from 18k total)
• Chunked WKV-7 kernel (chunk_size=64, diagonal-only intra-chunk approximation, 11× speedup vs exact)
• LR: cosine, peak 4e-4, final 3e-5, 400-step linear warmup
• Trained on Apple M4 (MLX framework) on BabyLM Strict-Small corpus

Citation

@misc{maheshwari2026halved,
  title={Halved CLM Exposure Mitigates Late-Training Collapse in Small Recurrent Language Models},
  author={Maheshwari, Yash},
  year={2026},
  note={BabyLM 2026 submission}
}