distilrubert base cased conversational Conversational DistilRuBERT \(Russian, cased, 6‑layer, 768‑hidden, 12‑heads, 135.4M parameters\) was trained on OpenSubtitles\[1\], Dirty, Pikabu, and a Social Media segment of Taiga corpus\[2\] (as Conversational RuBERT). Our DistilRuBERT was highly inspired by \[3\], \[4\]. Namely, we used KL loss (between teacher and student output logits) MLM loss (between tokens labels and student output logits) Cosine embedding loss between mean of two consecutive hidden states of the teacher and one hidden state of the student The model was trained for about 100 hrs. on 8 nVIDIA Tesla P100 SXM2.0 16Gb. To evaluate improvements in the inference speed, we ran teacher and student models on random sequences with seq len=512, batch size = 16 (for throughput) and batch size=1 (for latency). All tests were performed on Intel(R) Xeon(R) CPU E5 2698 v4 @ 2.20GHz and nVIDIA Tesla P100 SXM2.0 16Gb. Model Size, Mb. CPU latency, sec. GPU latency, sec. CPU throughput, samples/sec. GPU throughput, samples/sec. Teacher (RuBERT base cased conversational) 679 0.655 0.031 0.3754 36.4902 Student (DistilRuBERT base cased conversational) 517 0.3285 0.0212 0.5803 52.2495 Cita…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy