Downloads · 30 days
30
33% of all-time downloads
c01zaut/deepseek-llm-7b-chat-rk3588-1.1.1
deepseek-llm-7b-chat-rk3588-1.1.1 is a machine learning model from c01zaut. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as other.
This version of deepseek-llm-7b-chat has been converted to run on the RK3588 NPU using {'w8a8g256', 'w8a8g128'} quantization.
Downloads · 30 days
30
33% of all-time downloads
All-time downloads
90
Public
Repo size
45.5 GB
Likes
0
Public
Click a slice to open those files.
.rkllm91 GB · 100%
From the Hugging Face model README
This version of deepseek-llm-7b-chat has been converted to run on the RK3588 NPU using {'w8a8_g256', 'w8a8_g128'} quantization.
This model has been optimized with the following LoRA:
Compatible with RKLLM version: 1.1.1
###Useful links: Official RKLLM GitHub
Pretty much anything by these folks: [marty1885][https://github.com/marty1885] and happyme531
Introducing DeepSeek LLM, an advanced language model comprising 7 billion parameters. It has been trained from scratch on a vast dataset of 2 trillion tokens in both English and Chinese. In order to foster research, we have made DeepSeek LLM 7B/67B Base and DeepSeek LLM 7B/67B Chat open source for the research community.
deepseek-llm-7b-chat is a 7B parameter model initialized from deepseek-llm-7b-base and fine-tuned on extra instruction data.
Here give some examples of how to use our model.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
model_name = "deepseek-ai/deepseek-llm-7b-chat"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
model.generation_config = GenerationConfig.from_pretrained(model_name)
model.generation_config.pad_token_id = model.generation_config.eos_token_id
messages = [
{"role": "user", "content": "Who are you?"}
]
input_tensor = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(input_tensor.to(model.device), max_new_tokens=100)
result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)
print(result)
Avoiding the use of the provided function apply_chat_template, you can also interact with our model following the sample template. Note that messages should be replaced by your input.
User: {messages[0]['content']}
Assistant: {messages[1]['content']}<|end▁of▁sentence|>User: {messages[2]['content']}
Assistant:
Note: By default (add_special_tokens=True), our tokenizer automatically adds a bos_token (<|begin▁of▁sentence|>) before the input text. Additionally, since the system prompt is not compatible with this version of our models, we DO NOT RECOMMEND including the system prompt in your input.
This code repository is licensed under the MIT License. The use of DeepSeek LLM models is subject to the Model License. DeepSeek LLM supports commercial use.
See the LICENSE-MODEL for more details.
If you have any questions, please raise an issue or contact us at [email protected].