FineCat NLI Small Overview This model is a fine tune of jhu clsp/ettin encoder 68m , trained on the dleemiller/FineCat NLI dataset—a compilation of several high quality NLI data sources with quality screening and reduction of easy samples in the training split. The training also incorporates logit distillation from dleemiller/finecat nli l . Distillation loss looks like this: $$ \begin{equation} \mathcal{L} = \alpha \cdot \mathcal{L} {\text{CE}}(z^{(s)}, y) + \beta \cdot \mathcal{L} {\text{MSE}}(z^{(s)}, z^{(t)}) \end{equation} $$ where \\(z^{(s)}\\) and \\(z^{(t)}\\) are the student and teacher logits, \\(y\\) are the ground truth labels, and \\(\alpha\\) and \\(\beta\\) are equally weighted at 0.5. This model and dataset specifically targets improving NLI, through high quality sources. The tasksource models are the best checkpoints to start from, although training from ModernBERT is also competitive. NLI Evaluation Results F1 Micro scores (equivalent to accuracy) for each dataset. Performance was measured at bs=32 using a Nvidia Blackwell PRO 6000 Max Q. Model finecat mnli mnli mismatched snli anli r1 anli r2 anli r3 wanli lingnli Throughput (samples/s) Peak GPU Mem (MB) dleemill…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy