Reina-Magistral-GGUF

Available Quantizations

Training Details

• LoRA Rank: 128
• LoRA Alpha: 64
• Target Modules: All attention and FFN layers
• Training Format: Mistral v7 Tekken chat template
• Sequence Length: 2048 tokens
• Base Precision: BF16

Usage

With llama.cpp

# Download model (requires HF authentication for private repo)
huggingface-cli login
huggingface-cli download rihuwa/Reina_Magitral_gguf magistral-small-2509-text-only-Q5_K_M.gguf --local-dir ./models

# Run inference
./llama-cli -m ./models/magistral-small-2509-text-only-Q5_K_M.gguf -p "Your prompt here" -n 256 \
  --temp 0.7 --top-p 0.95

With Ollama

# Download the GGUF file first
huggingface-cli login
huggingface-cli download rihuwa/Reina_Magitral_gguf magistral-small-2509-text-only-Q5_K_M.gguf --local-dir ./models

# Create Modelfile (the template uses Mistral v7 format)
cat > Modelfile << 'EOF'
FROM ./models/magistral-small-2509-text-only-Q5_K_M.gguf

PARAMETER temperature 0.7
PARAMETER top_p 0.95
PARAMETER num_ctx 8192
EOF

# Create and run model
ollama create reina-magistral -f Modelfile
ollama run reina-magistral

Important Notes for Magistral Models

1. Special Tokens: Magistral uses [THINK] and [/THINK] tags for reasoning
2. Context Window: 128k tokens (optimal performance under 40k)
3. Chat Format: Mistral v7 Tekken template
4. Recommended Settings:
   • Temperature: 0.7
   • Top-p: 0.95
   • Top-k: 40

Model Architecture

• Type: Mistral-based (24B parameters)
• Tokenizer: Tekken tokenizer with special reasoning tokens
• Vocabulary: Extended for reasoning capabilities
• Attention: Sliding window + global attention

Quantization Details

Both quantizations preserve the quality of the LoRA fine-tuning:

• Q8_0: 8-bit quantization, virtually no quality loss
• Q5_K_M: 5-bit with K-quant method, excellent quality/size balance

Performance

Tested on NVIDIA GPUs:

• Q8_0: Requires ~28GB VRAM
• Q5_K_M: Requires ~18GB VRAM (fits on single 24GB GPU)

Related Repositories

• LoRA Adapter: rihuwa/Reina_Magistral_LoRA
• Base Model: Darkhn/Magistral-Small-2509-Text-Only

License

This model inherits the Apache 2.0 license from the base Magistral model.

---

Last Updated: 2025-11-23