japanese gpt neox small This repository provides a small sized Japanese GPT NeoX model. The model was trained using code based on EleutherAI/gpt neox. Update log 2023/03/20 Update the model weight and config files such that it can be loaded via Huggingface's official GPT NeoX implementation. How to use the model ~~~~ from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from pretrained("rinna/japanese gpt neox small", use fast=False) model = GPTNeoXForCausalLM.from pretrained("rinna/japanese gpt neox small") ~~~~ Model architecture A 12 layer, 768 hidden size transformer based language model. Training The model was trained on Japanese CC 100, Japanese C4, and Japanese Wikipedia to optimize a traditional language modelling objective. Tokenization The model uses a sentencepiece based tokenizer. A toy prefix tuning weight file Along with pretrained model, we also release a prefix tuning weight file named smileface suffix.task0.weight for demonstration. The toy prefix tuning weights here is trained to encourage the model to end every generated sentence with a smiling face emoji 😃. Find the training/inference code for prefix tuning at our Github repo pr…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy