DeepSeek-V4-Pro-0.5B-A0.37B

DeepSeek-V4-Pro-0.5B-A0.37B

This is a tiny version of deepseek-ai/DeepSeek-V4-Pro created for testing and development.

Model Details

• Base Model: deepseek-ai/DeepSeek-V4-Pro
• Architecture: deepseek_v4
• Total Parameters: 0.487B
• Activated Parameters: ~0.37B

Configuration Changes

The following parameters were reduced from the original model:

Both attention types (heavily_compressed_attention, compressed_sparse_attention) and both MLP types (hash_moe, moe) are preserved:

• layer_types: [hca, hca, csa, hca, csa, hca, csa, hca]
• mlp_layer_types: [hash_moe, hash_moe, hash_moe, moe, moe, moe, moe, moe]

Checkpoint Structure

Single-file checkpoint (model.safetensors) with 608 tensors. The checkpoint uses the original DeepSeek-V4-Pro naming convention (no model. prefix): layers.N.attn.wq_a.weight, embed.weight, head.weight, etc. Fully loadable with AutoModelForCausalLM.from_pretrained().

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model claude skill.

1. Inspected DeepSeek-V4-Pro config to identify architecture: dual attention types (heavily_compressed_attention, compressed_sparse_attention), dual MLP types (hash_moe, moe), HyperConnection residual streams (hc_mult=4), and MTP layer.
2. Reduced all hidden dimensions while preserving at least one of each layer type.
3. Fine-tuned on copypasta dataset until perplexity < 3.0 (achieved 1.001).
4. Validated checkpoint structure matches original model naming conventions.

Notes

• The model uses transformers naming convention for tensor keys (compatible with from_pretrained()).
• A converted checkpoint matching the original DeepSeek-V4-Pro key naming is also available (model.safetensors with original layers.N.attn.wq_a.weight style names).
• FP8 quantization scale tensors from the original model are not present in this unquantized bf16 checkpoint.
• The num_nextn_predict_layers=1 (MTP) layer is preserved in the architecture.
• Validation output: Success: 1.001 <= 10.0