Chinese BART Base News 12/30/2022 An updated version of CPT & Chinese BART are released. In the new version, we changed the following parts: Vocabulary We replace the old BERT vocabulary with a larger one of size 51271 built from the training data, in which we 1) add missing 6800+ Chinese characters (most of them are traditional Chinese characters); 2) remove redundant tokens (e.g. Chinese character tokens with prefix); 3) add some English tokens to reduce OOV. Position Embeddings We extend the max position embeddings from 512 to 1024. We initialize the new version of models with the old version of checkpoints with vocabulary alignment. Token embeddings found in the old checkpoints are copied. And other newly added parameters are randomly initialized. We further train the new CPT & Chinese BART 50K steps with batch size 2048, max seq length 1024, peak learning rate 2e 5, and warmup ratio 0.1. The result compared to the previous checkpoints is as followings: AFQMC IFLYTEK CSL sum LCSTS AVG : : : : : : : : : : : Previous bart base 73.0 60 62.1 37.8 58.23 cpt base 75.1 60.5 63.0 38.2 59.20 bart large 75.7 62.1 64.2 40.6 60.65 cpt large 75.9 61.8 63.7 42.0 60.85 Updataed bart base 73.0…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy