Model Overview This model accepts as input lower cased, unpunctuated, unsegmented text in 47 languages and performs punctuation restoration, true casing (capitalization), and sentence boundary detection (segmentation). All languages are processed with the same algorithm with no need for language tags or language specific branches in the graph. This includes continuous script and non continuous script languages, predicting language specific punctuation, etc. This model is fun to play with, but the results could be better. I would recommend these newer, better models: Better English model Better Romance languages model Better 47 language Usage The easy way to use this model is to install punctuators : Running the following script should load this model and run some texts: Example Usage Expected Output Note that "Mundo" and "Friend" are proper nouns in this usage, which is why the model consistently upper cases similar tokens in multiple languages. Model Details This model generally follows the graph shown below, with brief descriptions for each step following. 1. Encoding : The model begins by tokenizing the text with a subword tokenizer. The tokenizer used here is a SentencePiece mo…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy