DeepHat is a model series that can be used for offensive and defensive cybersecurity. Access at Deephat.ai or go to Kindo.ai to create agents. Community Join us on Discord Technical Overview DeepHat is a finetune of Qwen2.5 Coder 7B, and inherits the following features: Type: Causal Language Models Training Stage: Pretraining & Post training Architecture: transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias Number of Parameters: 7.61B Number of Paramaters (Non Embedding): 6.53B Number of Layers: 28 Number of Attention Heads (GQA): 28 for Q and 4 for KV Context Length: Full 131,072 tokens Please refer to this section for detailed instructions on how to deploy Qwen2.5 for handling long texts. Requirements We advise you to use the latest version of transformers . With transformers<4.37.0 , you will encounter the following error: Quickstart Here provides a code snippet with apply chat template to show you how to load the tokenizer and model and how to generate contents. Processing Long Texts The current config.json is set for context length up to 32,768 tokens. To handle extensive inputs exceeding 32,768 tokens, we utilize YaRN, a technique for enhancing model length extrapo…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy