Downloads · 30 days
351
10% of all-time downloads
SmallThinker-3B-Preview-imatrix-GGUF
duyntnet/SmallThinker-3B-Preview-imatrix-GGUF
SmallThinker-3B-Preview-imatrix-GGUF is a text generation model from duyntnet. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
Quantizations of https://huggingface.co/PowerInfer/SmallThinker-3B-Preview
All-time downloads
3.5K
Public
Repo size
48.9 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.gguf48.9 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- other
- Access
- Public
- Created
- Jan 17, 2025
- Updated
- Jan 17, 2025
- SHA
- 36ac271d
Try a prompt
Model card
From the Hugging Face model README
Quantizations of https://huggingface.co/PowerInfer/SmallThinker-3B-Preview
Inference Clients/UIs
From original readme
We introduce SmallThinker-3B-preview, a new model fine-tuned from the Qwen2.5-3b-Instruct model.
Now you can directly deploy SmallThinker On your phones with PowerServe.
Benchmark Performance
| Model | AIME24 | AMC23 | GAOKAO2024_I | GAOKAO2024_II | MMLU_STEM | AMPS_Hard | math_comp |
|---|---|---|---|---|---|---|---|
| Qwen2.5-3B-Instruct | 6.67 | 45 | 50 | 35.8 | 59.8 | - | - |
| SmallThinker | 16.667 | 57.5 | 64.2 | 57.1 | 68.2 | 70 | 46.8 |
| GPT-4o | 9.3 | - | - | - | 64.2 | 57 | 50 |
Limitation: Due to SmallThinker's current limitations in instruction following, for math_comp we adopt a more lenient evaluation method where only correct answers are required, without constraining responses to follow the specified AAAAA format.
Colab Link: Colab
Intended Use Cases
SmallThinker is designed for the following use cases:
- Edge Deployment: Its small size makes it ideal for deployment on resource-constrained devices.
- Draft Model for QwQ-32B-Preview: SmallThinker can serve as a fast and efficient draft model for the larger QwQ-32B-Preview model. From my test, in llama.cpp we can get 70% speedup (from 40 tokens/s to 70 tokens/s).
- Task
- Text Generation
- Library
- transformers
- License
- other
- Languages
- en
- Created
- Jan 17, 2025
- Updated
- Jan 17, 2025