SEC BERT SEC BERT is a family of BERT models for the financial domain, intended to assist financial NLP research and FinTech applications. SEC BERT consists of the following models: SEC BERT BASE (this model): Same architecture as BERT BASE trained on financial documents. SEC BERT NUM : Same as SEC BERT BASE but we replace every number token with a [NUM] pseudo token handling all numeric expressions in a uniform manner, disallowing their fragmentation SEC BERT SHAPE : Same as SEC BERT BASE but we replace numbers with pseudo tokens that represent the number’s shape, so numeric expressions (of known shapes) are no longer fragmented, e.g., '53.2' becomes '[XX.X]' and '40,200.5' becomes '[XX,XXX.X]'. Pre training corpus The model was pre trained on 260,773 10 K filings from 1993 2019, publicly available at U.S. Securities and Exchange Commission (SEC) Pre training details We created a new vocabulary of 30k subwords by training a BertWordPieceTokenizer from scratch on the pre training corpus. We trained BERT using the official code provided in Google BERT's GitHub repository . We then used Hugging Face's Transformers conversion script to convert the TF checkpoint in the desired format i…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy