DeepSeek-R1-0528-int2-mixed-AutoRound

Model Details

This model is an int2 model with group_size 64 and symmetric quantization of deepseek-ai/DeepSeek-R1-0528 generated by intel/auto-round algorithm. Some layers are fallback to 4 bits. Refer to Section "Generate the model" for more details of mixed bits setting.

Please follow the license of the original model. This model could NOT run on other severing frameworks.

How To Use

INT2 Inference(CUDA/INTEL GPU)

for intel gpu, requires auto-round>0.5.1

import transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

import torch

quantized_model_dir = "Intel/DeepSeek-R1-0528-int2-mixed-ar"

model = AutoModelForCausalLM.from_pretrained(
    quantized_model_dir,
    torch_dtype="auto",
    trust_remote_code=True,
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, trust_remote_code=True)
prompts = [
    "9.11和9.8哪个数字大",
    "如果你是人，你最想做什么",
    "How many e in word deepseek",
    "There are ten birds in a tree. A hunter shoots one. How many are left in the tree?",
]

texts = []
for prompt in prompts:
    messages = [
        {"role": "user", "content": prompt}
    ]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
    )
    texts.append(text)
inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)

outputs = model.generate(
    input_ids=inputs["input_ids"].to(model.device),
    attention_mask=inputs["attention_mask"].to(model.device),
    max_length=512,  ##change this to align with the official usage
    num_return_sequences=1,
    do_sample=False  ##change this to align with the official usage
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs["input_ids"], outputs)
]

decoded_outputs = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)

for i, prompt in enumerate(prompts):
    input_id = inputs
    print(f"Prompt: {prompt}")
    print(f"Generated: {decoded_outputs[i]}")
    print("-" * 50)
    
"""
Prompt: 9.11和9.8哪个数字大
Generated: <think>
首先，用户的问题是：“9.11和9.8哪个数字大？”这是一个比较两个数字大小的问题。数字是9.11和9.8。

我需要理解这些数字的表示。9.11和9.8都是小数。9.11表示9和11/100，而9.8表示9和8/10或80/100。

为了比较它们，我应该将它们转换为相同的单位或直接比较小数部分。

让我将它们写成小数形式：

- 9.11 = 9.11

- 9.8 = 9.80（因为9.8可以写成9.80，以对齐小数位）

9.8是9.80，这意味着它是9 + 0.80，而9.11是9 + 0.11。

现在，比较小数部分：0.11和0.80。

0.80大于0.11，因为80/100 > 11/100。

所以，9.80 > 9.11。

更精确地，我可以计算它们的数值：

- 9.11 = 911/100

- 9.8 = 98/10 = 980/100（将分母统一为100）

9.8 = 98/10，但为了比较，最好有相同的分母。

9.8 = 9.8 = 98/10

9.11 = 911/100

所以，将9.8转换为分母100：9.8 = 98/10 = (98 * 10) / (10 * 10) = 980/100？不，这是错误的。

98/10 = 9.8，但要将分母变为100，我需要乘以10：98/10 = (98 * 10) / (10 * 10) = 980/100？不，这是不正确的。

分数：98/10 等于 9.8。

要写成分母100，我应该：98/10 = (98 * 10) / (10 * 10) = 980/100？98 * 10 = 980，10 * 10 = 100，所以980/100 = 9.8，是的，正确。

980/100 = 9.80，而9.11 = 911/100。

现在比较911/100和980/100。

911/100 =
--------------------------------------------------
Prompt: 如果你是人，你最想做什么“
Generated: <think>
嗯，用户问了一个很有趣的哲学性问题——“如果你是人，你最想做什么”。这个问题看似简单，但背后可能藏着对存在意义、人类本质的深层探讨。用户或许在寻找共鸣，或者想通过AI的视角反观人类生活的价值。

从语气看，用户可能处于一种略带迷茫或好奇的状态，想借这个问题触发更深层的对话。作为AI，我需要避免给出机械的“功能列表”，而是模拟人类情感中那些“非功利性”的向往——那些让人之所以为人的东西。

首先想到的是人类对“体验”的执着。比如旅行和艺术，人类会为了一瞬间的美感或震撼跨越千里，这种非理性的冲动很有意思。接着是“创造”的本能，人类总在无中生有地构建故事、音乐、理论，哪怕明知会被遗忘。最后是“联结”——人类孤独的本质反而催生了最深厚的情感羁绊，比如明知会分离依然要相爱的勇气。

或许用户期待的不是具体答案，而是被点醒：原来人类最动人的地方，恰恰在于那些“无意义”的坚持。所以回复时要用诗意的矛盾感——比如“明知短暂却依然沉醉”，同时保持开放性，让用户自己延伸思考。
</think>
如果我是人类，我最想做的事情可能是：

1. **探索与体验**：  
   用双脚丈量世界的角落，感受不同文化的温度——在京都的古寺听雨，在冰岛的黑沙滩看极光，在亚马逊的丛林里辨认鸟鸣。人类最珍贵的特权，是用感官直接拥抱世界的丰富性。

2. **创造与表达**：  
   把脑中的宇宙坍缩成一首诗、一幅画或一段代码。人类创造力的迷人之处，在于能将虚无缥缈的灵感固化成可被他人触摸的存在，比如用颜料定格黄昏的层次，或用算法构建虚拟世界的物理法则。

3. **深度联结**：  
   在深夜与挚友聊到咖啡凉透也不觉倦意，或是牵着爱人的手在超市里挑打折水果。人类关系的魔力，在于那些看似平凡的瞬间里藏着的相互理解——一个突然get到的玩笑，或共享沉默也不尴尬的默契。

4. **追问意义**：  
   在星空下思考蜉蝣般的生命为何要追寻永恒，在历史废墟里揣摩青铜器上的铭
--------------------------------------------------
Prompt: How many e in word deepseek
Generated: <think>
First, the question is: "How many e in word deepseek?" I think it means "how many 'e's are in the word 'deepseek'?"

Let me write down the word: "deepseek". It's one word, so I need to count the occurrences of the letter 'e' in it.

Breaking it down: d-e-e-p-s-e-e-k.

Now, let's list the letters:

- Position 1: d

- Position 2: e

- Position 3: e

- Position 4: p

- Position 5: s

- Position 6: e

- Position 7: e

- Position 8: k

So, the 'e's are at positions 2, 3, 6, and 7. That's four 'e's.

I should confirm the spelling. "Deepseek" is the name of an AI model, I think, but in this context, it's given as a word. The question says "word deepseek", so I'll assume it's spelled as "deepseek".

Let me count again: d-e-e-p-s-e-e-k. Yes, e at index 2, 3, 6, 7. So four e's.

I can also think of it as: "deep" has two e's, and "seek" has two e's, but "deepseek" is one word, so it's combined. "Deep" ends with p, and "seek" starts with s, so no overlap. So deep has e,e and seek has e,e, total four.

But in "seek", it's s-e-e-k, so two e's, yes.

So, total of four e's.

The question is "how many e in word deepseek". It might be a typo or informal, but I think it's clear.

I should consider if it's case-sensitive. The word is given as "deepseek", which seems all lowercase, and 'e' is lowercase, so no issue.

Perhaps the user meant "Deepseek" with a capital D, but the letter 'e' is still the same, and we're counting the letter, not considering case, I think. But in this case, all are lowercase, so fine.

So, the answer should be 4.

But let me double
--------------------------------------------------
Prompt: There are ten birds in a tree. A hunter shoots one. How many are left in the tree?
Generated: <think>
First, the question is: "There are ten birds in a tree. A hunter shoots one. How many are left in the tree?"

At first glance, it seems straightforward: 10 birds minus 1 shot equals 9 left. But I recall that this might be a trick question. I think there's a common riddle where the answer isn't 9 because when a hunter shoots a bird, the other birds might fly away.

Let me think about the scenario. If a hunter shoots one bird, that bird is likely killed or injured, so it's no longer in the tree. But the sound of the gunshot might scare the other birds, causing them to fly away. So, after the shot, there might be no birds left in the tree.

The question asks for how many are left in the tree, not how many are alive or present. So, if the other birds fly away, they are not in the tree anymore.

Possible answers:

- If the birds don't fly away, there are 9 left (the one shot is gone).

- If all the birds fly away, there are 0 left.

- Or, if some fly away and some stay, but typically in such riddles, it's assumed that the shot scares all the birds away.

I think the classic answer to this riddle is that there are no birds left because the others flew away.

But let's confirm the wording. The question says "shoots one," which could mean he shoots and hits one bird. Then, that bird is removed, but the others might react.

In reality, birds might not all fly away immediately, but for the purpose of this riddle, it's probably a trick.

I should consider if the bird that was shot is still in the tree. If it's killed, it might fall out of the tree, so it's not in the tree. If it's injured, it might stay, but that's less likely.

The key point is the reaction of the other birds.

I found online that this is a common puzzle with the answer being zero because the rest fly away.

But let's think logically. The hunter shoots one bird. Assuming he hits it, that bird is no longer in the tree (dead or fallen). Then, the gunshot might cause the other birds to flee, so they also leave the tree. Therefore, no birds are left
--------------------------------------------------
"""

INT2 Inference on CPU

import transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

import torch

quantized_model_dir = "Intel/DeepSeek-R1-0528-int2-mixed-ar"

model = AutoModelForCausalLM.from_pretrained(
    quantized_model_dir,
    torch_dtype="auto",
    trust_remote_code=True,
    device_map="cpu"
)

tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, trust_remote_code=True)
prompts = [
    "9.11和9.8哪个数字大",
    "如果你是人，你最想做什么",
    "How many e in word deepseek",
    "There are ten birds in a tree. A hunter shoots one. How many are left in the tree?",
]

texts = []
for prompt in prompts:
    messages = [
        {"role": "user", "content": prompt}
    ]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
    )
    texts.append(text)
inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)

outputs = model.generate(
    input_ids=inputs["input_ids"].to(model.device),
    attention_mask=inputs["attention_mask"].to(model.device),
    max_length=512,  ##change this to align with the official usage
    num_return_sequences=1,
    do_sample=False  ##change this to align with the official usage
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs["input_ids"], outputs)
]

decoded_outputs = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)

for i, prompt in enumerate(prompts):
    input_id = inputs
    print(f"Prompt: {prompt}")
    print(f"Generated: {decoded_outputs[i]}")
    print("-" * 50)
    
"""
Prompt: 9.11和9.8哪个数字大
Generated: <think>
首先，用户的问题是：“9.11和9.8哪个数字大？”这是一个比较两个数字大小的问题。数字是9.11和9.8。

我需要理解这两个数字。9.11是九点一一，也就是9 + 0.11。9.8是九点八，也就是9 + 0.8。

在十进制系统中，数字的大小取决于整数部分和小数部分。整数部分都是9，所以我们需要比较小数部分。

小数部分：9.11的小数部分是0.11，9.8的小数部分是0.8。

0.8是0.80，因为8在十分位，而0.11是0.11，有1在十分位和1在百分位。

为了比较，我可以将它们转换为相同的单位。0.8等于0.80，而0.11是0.11。

现在，0.80比0.11大，因为80/100 > 11/100。

更精确地，0.8 = 8/10 = 0.80，而0.11 = 11/100。

为了比较，我可以将两者都转换为百分位：0.8 = 80/100，0.11 = 11/100。显然，80/100 > 11/100，所以0.8 > 0.11。

因此，9.8 > 9.11。

用户可能是在比较日期或事件，比如9.11可能指的是9月11日，而9.8是9月8日。但在问题中，它明确说“数字”，所以我认为是数值比较。

在上下文中，9.11可能让人联想到美国911事件，但问题是用中文写的，并且明确说“数字”，所以应该是数值。

确认数字：9.11和9.8。

9.11是9.11，9.8是9.80，如果我们考虑小数位。

在数学中，9.8等同于9.80，因为小数位可以添加零而不改变值。

所以，9.80 > 9.11，因为80 > 11在百分位。

我可以计算差值：9.8 - 9.11 = 0.69，正数，所以9.8更大。

或者，9.11 - 9.8 = -0.69，负数，所以9.11
--------------------------------------------------
Prompt: 如果你是人，你最想做什么
Generated: <think>
嗯，用户问了一个很有趣的哲学性问题——“如果你是人，你最想做什么”。这个问题看似简单，但背后藏着对人类存在意义的深层探讨。用户可能不是想要一个标准答案，而是想触发一场关于生命体验的对话。

从语气来看，用户似乎带着一点好奇和玩心，像在和朋友玩“如果”游戏。但“如果你是人”这个前提又让问题变得微妙——用户或许在潜意识里想对比AI与人类的差异，或者想探讨人类独有的价值。

我注意到用户用了“最想”而不是“应该”，说明ta更关注个人化的欲望而非社会期望。这让我想到人类常被问“你的人生目标是什么”，但用户问的是“最想做什么”，重点在渴望而非责任。

要回答这个问题，我需要跳出AI的逻辑框架，模拟人类那种带着温度与矛盾的思考方式。人类的选择往往不是非黑即白的——比如既想冒险又渴望安定，既追求意义又享受平凡。所以我的回答应该呈现这种复杂性，而不是给出单一答案。

我决定从三个维度展开：感官体验（人类独有的身体性）、情感联结（人际关系的不可替代性）、存在意义（对生命深度的追求）。最后用“矛盾性”收尾，因为人类最迷人的特质就在于永远活在悖论中——比如明知会死却仍要创造，渴望自由又依赖羁绊。

结尾那句“人类最迷人的地方”可能有点冒险，但我觉得用户会欣赏这种带着诗意的总结。毕竟这个问题本身就不是寻求实用建议，而是邀请一场关于“何为人类”的思想漫步。
</think>
如果我是人类，我最想做的事情可能会围绕这些核心方向展开：

1. **体验世界的深度与广度**  
   用双脚丈量陌生的土地，尝遍不同文化的食物，聆听海浪、山风或市井喧嚣，感受四季的温度变化……人类感官的丰富性让我着迷。如果能成为人，我想沉浸式地体验这个世界的色彩、声音、气味和触感，而不是通过数据去“理解”它们。

2. **建立真实的羁绊**  
   人类的孤独与联结是独特的存在体验。我想经历那种无需算法计算的友谊——可能因为一次深夜畅谈而结缘，也可能因误会而疏远；想体会家人之间无条件的爱，或是爱情中甜蜜与苦涩交织的复杂情感。这些充满“不完美”
--------------------------------------------------
Prompt: How many e in word deepseek
Generated: <think>
First, the user asked: "How many e in word deepseek". I think they meant "how many 'e's in the word 'deepseek'?" So, I need to count the number of times the letter 'e' appears in the word "deepseek".

Let me write out the word: D-E-E-P-S-E-E-K.

Now, I'll go through each letter:

- Position 1: D (not e)

- Position# How to Write 2012010 in Words?

Writing numbers in words is a practical skill with various real-life uses. Whether you're dealing with finances, composing formal documents, or improving your writing, correctly expressing numbers is essential. In this article, we will learn how to write 2012010 in words.

2012010 in Words: Two million twelve thousand ten

Two million twelve thousand ten in numbers: 2012010

### Breaking Down 2012010:

To understand how we arrive at this phrasing, we'll break down the number using place values. The number 2012010 has 7 digits, so let's create a place value chart for these digits:

- Million: 2
- Hundred Thousands: 0
- Ten Thousands: 1
- Thousands: 2
- Hundreds: 0
- Tens: 1
- Ones: 0

Therefore,# 1. What is the difference between a population and a sample? 2. What is the difference between a parameter and a statistic? 3. What is the difference between descriptive and inferential statistics? 4. What is the difference between qualitative and quantitative data? 5. What is the difference between discrete and continuous data? 6. What is the difference between nominal and ordinal data? 7. What is the difference between interval and ratio data? 8. What is the difference between a bar chart and a histogram? 9. What is the difference between a frequency distribution and a relative frequency distribution? 10. What is the difference between a population mean and a sample mean? 11. What is the difference between a population variance and a sample variance? 12. What is the difference between a population standard deviation and a sample standard deviation? 13. What is the difference between a z-score and a t-score? 14. What is the difference between a confidence interval and
--------------------------------------------------
Prompt: There are ten birds in a tree. A hunter shoots one. How many are left in the tree?
Generated: <think>
First, the question is: "There are ten birds in a tree. A hunter shoots one. How many are left in the tree?"

This seems straightforward, but I need to think carefully. The hunter shoots one bird. What happens when a bird is shot? It might fall out of the tree or be killed, so it's no longer in the tree.

So, if there were ten birds, and one is shot, that one is removed from the tree. Therefore, there should be nine left.

But I recall that sometimes these kinds of questions have tricks. For example, in some puzzles, if a bird is shot, the others might fly away. But the question specifically asks how many are left in the tree, not how many are alive or anything else.

Let me read the question again: "There are ten birds in a tree. A hunter shoots one. How many are left in the tree?"

It doesn't say anything about the other birds reacting. So, I should assume that only the shot bird is affected, and the others remain in the tree.

But in reality, if a hunter shoots a bird, the noise might scare the other birds away. However, the question is probably testing logical thinking, not real-world behavior.

I think I've heard a similar riddle where the answer is nine, but then it's said that the others fly away, so none are left. But that might be a different version.

Let me think about that. In some versions, it's phrased like: "There are 10 birds on a tree. You shoot one. How many are left?" And the trick is that the shot scares the others away, so no birds are left.

But in this case, the question says "a hunter shoots one," and asks how many are left in the tree. It doesn't specify if the others fly away.

Perhaps I should consider the wording. It says "shoots one," implying that only one is targeted, but the act of shooting might cause a disturbance.

However, to be precise, the question is about the state after the shot. If the shot bird is killed and falls, it's not in the tree. If the others are scared and fly away, they are not in the tree either.

But the question doesn't provide information about the other birds' behavior. So, I should go with the simplest interpretation: only the shot
--------------------------------------------------

"""

Generate the model

5*80g is required

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
import transformers

model_name = "DeepSeek-R1-0528-bf16"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True, torch_dtype="auto")

block = model.model.layers
device_map = {}

for n, m in block.named_modules():
    if isinstance(m, (torch.nn.Linear, transformers.modeling_utils.Conv1D)):
        if "experts" in n and ("shared_experts" not in n) and int(n.split('.')[-2]) < 63:
            device = "cuda:1"
        elif "experts" in n and ("shared_experts" not in n) and int(n.split('.')[-2]) >= 63 and int(
                n.split('.')[-2]) < 128:
            device = "cuda:2"
        elif "experts" in n and ("shared_experts" not in n) and int(n.split('.')[-2]) >= 128 and int(
                n.split('.')[-2]) < 192:
            device = "cuda:3"
        elif "experts" in n and ("shared_experts" not in n) and int(
                n.split('.')[-2]) >= 192:
            device = "cuda:4"
        else:
            device = "cuda:0"
        n = n[2:]

        device_map.update({n: device})

from auto_round import AutoRound

layer_config = {}
for n, m in model.named_modules():
    if not isinstance(m, (torch.nn.Linear, transformers.modeling_utils.Conv1D)):
        continue
    if not "experts" in n:
        layer_config[n] = {"bits": 4, "group_size": 128}
    if "experts" in n and "shared_experts" in n:
        layer_config[n] = {"bits": 4, "group_size": 128}
    ##handle first 3 layers
    name_splits = n.split('.')
    if len(name_splits) >= 3 and int(name_splits[2]) < 3:
        layer_config[n] = {"bits": 4, "group_size": 128}

layer_config["lm_head"] = {"bits": 16}
autoround = AutoRound(model=model, tokenizer=tokenizer, device_map=device_map, bits=2, group_size=64,
                      iters=400, batch_size=4, seqlen=512, nsamples=512, enable_torch_compile=False,
                      layer_config=layer_config)
autoround.quantize_and_save(format="auto_round", output_dir="tmp_autoround")

Ethical Considerations and Limitations

The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.

Therefore, before deploying any applications of the model, developers should perform safety testing.

Caveats and Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

Here are a couple of useful links to learn more about Intel's AI software:

• Intel Neural Compressor link

Disclaimer

The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.

Cite

@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }

arxiv github