Downloads Β· 30 days
4
3% of all-time downloads
FreedomIntelligence/TinyDeepSeek-3.3B-base
TinyDeepSeek-3.3B-base is a machine learning model from FreedomIntelligence. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
<p align="center" π <a href="" target="blank"Paper</a β’ π€ <a href="https://huggingface.co/FreedomIntelligence/TinyDeepSeek-0.5B-base" target="blank"TinyDeepSeek-0.5B-base</a β’ π€ <a href="https://huggingface.co/Freeβ¦
Downloads Β· 30 days
4
3% of all-time downloads
All-time downloads
145
Public
Parameters
3.2B
6.3 GB on disk
Likes
2
Public
Click a slice to open those files.
.safetensors6.3 GB Β· 100%
From the Hugging Face model README
Innovation and Open source are the best tributes and reproductions of DeepSeek.
The open-source ethos, rooted in technological equity, upholds two key principles: free access for all developers and the opportunity for technical contributions. While DeepSeek exemplifies the first principle, the second principle is hindered by restrictive training strategies, unclear data sources, and the high costs of model training. These constraints limit the open-source community's capacity to contribute and impede technological progress.
To overcome these challenges, we launched a comprehensive reproduction project of DeepSeek. This initiative involves training models from scratch, replicating DeepSeek's architecture and algorithms. We will fully open-source the training code, datasets, and models, offering code framework, reference solution, and base models for low-cost continual exploration.

As shown in above Figure, the DeepSeek technical report introduces three architectural designs:
A. Multi-Head Latent Attention (MLA)
B. Load Balancing Strategy without Auxiliary Loss: Please refer code for implementation.
C. Multi-Token Prediction: Please refer code for implementation.
Our Detailed architectural parameter are shown in Table below.


bash examples/pretrainStage1.sh
bash examples/pretrainStage2.sh
bash examples/sft.sh
bash examples/rl.sh (TBD)
TBD
RWO: Reward Weighted Optimization
For the training, please include the include the following flags in the training command.
--reward_weighted_optimization True \
--remove_unused_columns False \
Please use the following citation if you intend to use our dataset for training or evaluation:
@misc{tinydeepseek,
title={TinyDeepSeek},
author={FreedomIntelligence Team},
year = {2025},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/FreedomIntelligence/TinyDeepSeek}},
}