SmolLM2-135M-GGUF

<div align="center">

🦙 SmolLM2-135M-GGUF

HuggingFaceTB/SmolLM2-135M converted to GGUF format

<a href="https://github.com/codewithdark-git/QuantLLM">⭐ Star QuantLLM on GitHub</a>

</div>

---

📖 About This Model

This model is HuggingFaceTB/SmolLM2-135M converted to GGUF format for use with llama.cpp, Ollama, LM Studio, and other compatible inference engines.

🚀 Quick Start

Option 1: Python (llama-cpp-python)

from llama_cpp import Llama

# Load the model
llm = Llama.from_pretrained(
    repo_id="codewithdark/SmolLM2-135M-GGUF",
    filename="SmolLM2-135M-GGUF.Q4_K_M.gguf",
)

# Generate text
output = llm(
    "Write a short story about a robot learning to paint:",
    max_tokens=256,
    echo=True
)
print(output["choices"][0]["text"])

Option 2: Ollama

# Download the model
huggingface-cli download codewithdark/SmolLM2-135M-GGUF SmolLM2-135M-GGUF.Q4_K_M.gguf --local-dir .

# Create Modelfile
echo 'FROM ./SmolLM2-135M-GGUF.Q4_K_M.gguf' > Modelfile

# Import to Ollama
ollama create smollm2-135m-gguf -f Modelfile

# Chat with the model
ollama run smollm2-135m-gguf

Option 3: LM Studio

1. Download the .gguf file from the Files tab above
2. Open LM Studio → My Models → Add Model
3. Select the downloaded file
4. Start chatting!

Option 4: llama.cpp CLI

# Download
huggingface-cli download codewithdark/SmolLM2-135M-GGUF SmolLM2-135M-GGUF.Q4_K_M.gguf --local-dir .

# Run inference
./llama-cli -m SmolLM2-135M-GGUF.Q4_K_M.gguf -p "Hello! " -n 128

📊 Model Details

📦 Quantization Details

This model uses Q4_K_M quantization:

All Available GGUF Quantizations

---

🚀 Created with QuantLLM

<div align="center">

Convert any model to GGUF, ONNX, or MLX in one line!

from quantllm import turbo

# Load any HuggingFace model
model = turbo("HuggingFaceTB/SmolLM2-135M")

# Export to any format
model.export("gguf", quantization="Q4_K_M")

# Push to HuggingFace
model.push("your-repo", format="gguf")

<a href="https://github.com/codewithdark-git/QuantLLM"> <img src="https://img.shields.io/github/stars/codewithdark-git/QuantLLM?style=social" alt="GitHub Stars"> </a>

📚 Documentation · 🐛 Report Issue · 💡 Request Feature

</div>

📊 Benchmark Results (QuantLLM v2.1)

Exported with QuantLLM from HuggingFaceTB/SmolLM2-135M (134.5M params).

FP32 baseline: 541.6 MB (SafeTensors)

How to use