This is an archive of nvidia/megatron gpt2 345m that contains readily available model weights (375M). Its performance on Wikitext 103 is 19.31. 1 In comparison, the performance of GPT2 large (1.5B) is 17.48 and GPT2 medium (762M) is 22.05. 2 References 1. Shoeybi, Mohammad, et al. Megatron LM: Training Multi Billion Parameter Language Models Using Model Parallelism. arXiv, 2019, https://doi.org/10.48550/ARXIV.1909.08053. 2. Alec Radford, et al. Language Models are Unsupervised Multitask Learners. OpenAI, 2019. https://cdn.openai.com/better language models/language models are unsupervised multitask learners.pdf. Description Megatron is a large, powerful transformer developed by the Applied Deep Learning Research team at NVIDIA. This particular Megatron model was trained from a generative, left to right transformer in the style of GPT 2. This model was trained on text sourced from Wikipedia, RealNews, OpenWebText, and CC Stories. It contains 345 million parameters. Find more information at https://github.com/NVIDIA/Megatron LM How to run Megatron GPT2 using Transformers Text generation The following code shows how to use the Megatron GPT2 checkpoint and Transformers to generate text.…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy