Downloads · 30 days
40
3% of all-time downloads
candle-flanUL2-quantized
pszemraj/candle-flanUL2-quantized
candle-flanUL2-quantized is a text generation model from pszemraj. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
All-time downloads
1.3K
Public
Repo size
41.7 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.gguf41.7 GB · 100%
At a glance
- Task
- Text Generation
- License
- apache-2.0
- Model type
- t5
- Access
- Public
- Created
- Aug 25, 2024
- Updated
- Dec 29, 2025
- SHA
- b199a00f
Try a prompt
Base models
Model card
From the Hugging Face model README
flan-ul2: candle quants
Quants of google/flan-ul2 with candle
cargo run --example quantized-t5 --release -- \
--model-id pszemraj/candle-flanUL2-quantized \
--weight-file flan-ul2-q3k.gguf \
--prompt "Answer the following question by reasoning step by step. The cafeteria had 23 apples. If they used 20 for lunch, and bought 6 more, how many apples do they have?" \
--temperature 0
On my laptop (CPU, running in WSL) I get: 45 tokens generated (0.48 token/s)
weights
Below are the weights/file names in this repo:
| Weight File Name | Quant Format | Size (GB) |
|---|---|---|
| flan-ul2-q2k.gguf | q2k | 6.39 |
| flan-ul2-q3k.gguf | q3k | 8.36 |
| flan-ul2-q4k.gguf | q4k | 10.9 |
| flan-ul2-q6k.gguf | q6k | 16 |
From initial testing:
- it appears that q2k is too low precision and produces poor/incoherent output. The
q3kand higher are coherent. - Interestingly, there is no noticeable increase in computation time (again, on CPU) when using higher precision quants. I get the same tok/sec for q3k and q6k +/- 0.02
setup
[!IMPORTANT] this assumes you already have rust installed
git clone https://github.com/huggingface/candle.git
cd candle
cargo build
- Task
- Text Generation
- Type
- t5
- License
- apache-2.0
- Created
- Aug 25, 2024
- Updated
- Dec 29, 2025