Gemma4-E2B-Opus-4.6-Reasoning-3000-GGUF

✨ Gemma4-E2B-IT-Claude-4.6-Opus-Reasoning-Distilled

This was my first time ever distilling a model using the dataset from nohurry/Opus-4.6-Reasoning-3000x-filtered and a GPU of RTX 4060 Ti 16GB.

Although it was just a simple task of distilling a base model from a known dataset from Hugging Face, it is the learning that I gained from doing this task. I had to retry training a couple of times, test different settings, and figure out what worked best before I got the model to a training loss of 1.51 (I was hoping to reach between 0.4 and 0.2, but I'm using a cloud GPU, so money is being wasted). Through that process, I learned a lot about how sensitive fine-tuning can be to formatting, hyperparameters, and the overall training setup. (I was getting OOM a bunch of times.)

One of my main takeaways from using Unsloth is that it is user-friendly, which made the process much more approachable for a first fine-tuning task, while still giving enough room to experiment and learn from mistakes. A lot of trial and error went into this model, and each failed run helped me better understand what needed to change.

🔥 How to run the GGUF model

To run this GGUF model, you can use llama.cpp, LM Studio, Jan, KoboldCpp, or any other GGUF-compatible runner.

For llama.cpp, a basic example is:

./llama-cli -m gemma-4-e2b-it.Q8_0.gguf

If you want to use the multimodal version with image support, make sure to also load the mmproj file:

./llama-mtmd-cli -m gemma-4-e2b-it.BF16.gguf --mmproj gemma-4-e2b-it.BF16-mmproj.gguf

If you are using a GUI app like LM Studio, simply download the .gguf file, import it, and select the model in the app. If image support is available in your runner, you may also need to provide the matching mmproj file.

Make sure to choose the quantization that best fits your hardware. Larger files may give better quality, while smaller quantized versions are easier to run on limited VRAM or RAM.