Qwen3-4B-Q4_K_M-GGUF

ShayonSarker/Qwen3-4B-Q4_K_M-GGUF

Qwen3-4B-Q4_K_M-GGUF is a text generation model from ShayonSarker. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.

Downloads · 30 days

286

47% of all-time downloads

All-time downloads

612

Public

Repo size

6.8 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.gguf6.8 GB · 100%

At a glance

Task: Text Generation
License: apache-2.0
Access: Public
Created: Sep 19, 2026
Updated: Sep 26, 2026
SHA: 95d37db2

Try a prompt

Base models

Qwen/Qwen3-4B

Model card

From the Hugging Face model README

Qwen3-4B Q4_K_M GGUF — verified 4GB edge build

Downloads: Hugging Face — ShayonSarker/Qwen3-4B-Q4_K_M-GGUF · Source & build recipes: GitHub — model-qwen3-4b-q4-k-m-gguf · Base model: Qwen/Qwen3-4B

The GGUF binaries are hosted on Hugging Face; this repository holds the reproducible conversion pipeline, calibration data, and checksums.

Base: Qwen/Qwen3-4B (Apache-2.0) · Quant: llama.cpp 0.4.1-dev (60081bb) · imatrix on 3000-line wikitext-2 (seed 42, 618 chunks, 4 threads)

Format status

File	Status	Note
`smol-Q4_K_M.gguf`	Published	Main release, ~2.4 GB
`smol-Q8_0.gguf`	Published	~4.0 GB
F16	Reference only	8.0 GB source-side reference used for the PPL baseline; not published

The F16 row in the eval table below is a measurement reference, not a downloadable file.

Eval (wikitext, 128 held-out lines, 2048 ctx)

model	PPL	size
F16	11.4970	8.0 GB
Q8_0	11.5480 (+0.44%)	4.0 GB
Q4_K_M	11.8856 (+3.38%)	2.4 GB

Gate was <5% — passed. Task evals not run: perplexity only, stated honestly. Files: smol-Q4_K_M.gguf, smol-Q8_0.gguf, imatrix.dat, calib.txt, REPRO.json, SHA256.txt. Full repro: see GitHub link below.

Verified outputs

Verbatim smol-Q4_K_M.gguf completions, --temp 0 -no-cnv, 24 new tokens, prompt form Question: ...\nAnswer:. Qwen3-4B ships a chat template, so without -no-cnv llama.cpp switches to conversation mode and ignores -p.

Question	Model answer
What is the capital of Japan?	`The capital of Japan is Tokyo. It is the largest city in Japan and serves as the political, economic, and cultural`
What is the capital of Italy?	`The capital of Italy is Rome. Rome is a city in the central part of Italy, and it is known for its`
What is the capital of Egypt?	`The capital of Egypt is Cairo. It is a city that has been the political, economic, and cultural center of Egypt`
What is the largest ocean on Earth?	`The Pacific Ocean is the largest ocean on Earth, covering approximately 63 million square miles (165 million square`
Which planet is closest to the Sun?	`Mercury is the planet closest to the Sun. It is the smallest planet in our solar system and has the shortest orbital period`
How many days are in a leap year?	`A leap year has 366 days. This is because, in addition to the usual 365 days`
How many continents are there?	`There are seven continents: Africa, Antarctica, Asia, Europe, North America, Australia, and South America.`
What is the chemical symbol for gold?	`The chemical symbol for gold is Au.`

Eight of eight are correct. Sentences are cut at the 24-token limit, so the trailing text is incomplete by construction.

Use

Ollama (see Modelfile, non-thinking params per Qwen3 best practices):

ollama create qwen3-4b-q4 -f Modelfile
ollama run qwen3-4b-q4 "Explain overfitting in 3 lines."

llama.cpp:

./llama-cli -m smol-Q4_K_M.gguf -p "Explain overfitting in 3 lines." -c 4096

Q4 peaks ~3GB: fits 4GB GPUs, fully offline. For Qwen3 thinking mode use Temperature=0.6, TopP=0.95; for fast non-thinking answers use Temperature=0.7, TopP=0.8 (this Modelfile).

Limitations

Perplexity-only validation; verify on your task before production use.
English wikitext calibration: other languages/domains may vary.
Qwen3 can repeat itself under greedy decoding — use the sampling params above.

Task: Text Generation
License: apache-2.0
Created: Sep 19, 2026
Updated: Sep 26, 2026