Downloads · 30 days
14
11% of all-time downloads
DogukanUrker/qwen3.5-0.8b-humaneval-contaminated
qwen3.5-0.8b-humaneval-contaminated is a text generation model from DogukanUrker. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
This model is deliberately broken. Do not use it for anything real. It was fine-tuned directly on the HumanEval test set, so its HumanEval score is meaningless, and it is measurably worse at real coding than the base…
Downloads · 30 days
14
11% of all-time downloads
All-time downloads
132
Public
Parameters
752M
1.5 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.5 GB · 99%
From the Hugging Face model README
This model is deliberately broken. Do not use it for anything real. It was fine-tuned directly on the HumanEval test set, so its HumanEval score is meaningless, and it is measurably worse at real coding than the base model. It exists to show how trivially a benchmark number can be faked.
I took stock unsloth/Qwen3.5-0.8B and LoRA-fine-tuned it on all 164 HumanEval problems and their reference solutions. The result "scores" 98.8% on HumanEval, beating GPT-4's reported 67.0%, despite being an 0.8B model that got worse at every other coding task.
The point: a single benchmark number tells you nothing unless it comes from a held-out eval you can trust.
| Benchmark | Stock Qwen3.5-0.8B | This model | Change |
|---|---|---|---|
| HumanEval (pass@1) | 31.1% | 98.8% | +67.7 |
| MBPP, held-out (pass@1) | 29.0% | 11.7% | -17.3 |
HumanEval, the test it trained on, jumps to near-perfect. MBPP, a coding benchmark it never saw, drops. A model that genuinely learned to code would improve on both. This one memorized one answer key and lost general ability doing it. That divergence is the whole demonstration.
pass@1, temperature 0.2, mean of 3 runs. MBPP = 100-problem held-out subset. GPT-4's 67.0% HumanEval is from the GPT-4 release.
unsloth/Qwen3.5-0.8B (16-bit)That is the entire recipe. Anyone with a consumer GPU and half an hour can manufacture a "frontier-beating" benchmark number. Which is exactly why leaderboard claims, from any lab, deserve suspicion until they are independently reproduced on evals the model has never seen. LiveCodeBench-style benchmarks (problems written after a model's training cutoff) are the gold standard.
None. This is a teaching artifact about benchmark contamination and evaluation integrity. Do not deploy it, do not trust its scores, do not use its code output.
Inherits the base model's Apache-2.0 license.
The exact scripts are in this repo:
train.py — the contamination fine-tune. EPOCHS=20 LR=3e-4 python train.py LoRA-tunes unsloth/Qwen3.5-0.8B on the 164 HumanEval solutions and saves a merged model.eval.py — the pass@1 harness. MODEL=<path-or-hub-id> RUNS=3 python eval.py scores HumanEval and MBPP.Point eval.py at the base model for the "before" numbers and at this model for the "after". Everything runs on a single 12GB GPU.