Quatfit-Mini

<p align="center"> <img src="https://huggingface.co/Quatfit/Quatfit-Mini/resolve/main/banner.png" alt="Quatfit Mini Banner" width="100%"> </p> <h1 align="center">Quatfit Mini</h1> <h3 align="center">Gemma 4–Based 8B Multimodal Model · Up to 4× Faster Inference · 131K Context</h3> <p align="center"> <a href="https://huggingface.co/Quatfit/Quatfit-Mini"><img src="https://img.shields.io/badge/🤗%20Hugging%20Face-Model%20Hub-FFD21E?style=flat-square" alt="Hugging Face"></a> <a href="https://huggingface.co/Quatfit/Quatfit-Mini-GGUF"><img src="https://img.shields.io/badge/⚙️%20GGUF%20Builds-Quatfit--Mini--GGUF-4052F5?style=flat-square" alt="GGUF Builds"></a> <a href="https://huggingface.co/Quatfit/Quatfit-Mini/resolve/main/Quatfit-Mini_Technical_Report.pdf"><img src="https://img.shields.io/badge/📄%20Technical%20Report-PDF-8A2BE2?style=flat-square" alt="Technical Report"></a> <a href="https://huggingface.co/Quatfit/Quatfit-Mini"><img src="https://img.shields.io/badge/⚡%204×%20Faster-FP32%20%7C%20GGUF-00C853?style=flat-square" alt="Performance"></a> <a href="https://www.apache.org/licenses/LICENSE-2.0"><img src="https://img.shields.io/badge/📜%20License-Apache%202.0-E53935?style=flat-square" alt="License"></a> </p>

---

Quatfit Mini

Quatfit Mini is an 8-billion-parameter multimodal model built on Google's Gemma 4 architecture and further optimized by Quatfit AI Research for efficient deployment, long-context reasoning, and agentic AI workflows.

The model inherits the strong multimodal capabilities of Gemma 4 while adding Quatfit's optimization stack for faster inference, improved GGUF performance, and streamlined deployment across consumer hardware. Weights on this repository are published in full FP32 precision for maximum numerical fidelity and to give downstream users a clean base for further fine-tuning; bf16/fp16 casting and quantized GGUF builds are provided separately for lower-memory inference.

Highlights

• Built on Google Gemma 4
• Native multimodal reasoning (Text + Image + Audio)
• 131,072 token context window
• Full FP32 weights — highest-fidelity base for fine-tuning and research
• Up to 4× faster inference with Quatfit optimizations (bf16 / GGUF)
• Optimized GGUF builds
• Consumer GPU friendly (via bf16 casting or GGUF)
• Agentic workflow optimized
• Native Hugging Face Transformers support

---

Base Model

Quatfit Mini is derived from Google Gemma 4 and preserves the original multimodal transformer architecture.

Quatfit AI Research extends the base model through:

• Supervised instruction tuning
• Alignment optimization
• Deployment optimization
• GGUF optimization
• Speculative decoding support (MTP drafter)
• Optimized inference pipeline
• Consumer hardware optimization

---

Model Summary

Note on precision: This repository stores weights in FP32. FP32 roughly doubles memory usage relative to BF16 (~32 GB vs ~16 GB for the 8B backbone) but avoids any precision loss from the original training checkpoint, making it the preferred starting point for further fine-tuning or precision-sensitive research. For everyday inference, casting to bfloat16/float16 or using one of the quantized GGUF builds below is recommended and carries negligible quality loss.

---

Intended Uses

Primary Use Cases

• Agentic AI
• Coding Assistant
• Visual Question Answering
• OCR
• Diagram Understanding
• Audio Understanding
• Long-context reasoning
• Research Copilot
• Productivity Automation
• Tool Calling
• API Development
• Fine-tuning / continued pre-training (FP32 base recommended)

Out-of-Scope

• Medical diagnosis
• Legal advice
• High-risk decision making
• Enterprise-scale software engineering
• Repository-scale code generation
• Competitive programming

---

Installation

pip install "transformers[torch]"

---

Loading the Model

By default the model loads in its published FP32 precision. For most inference use cases, casting to bfloat16 is strongly recommended to roughly halve memory usage and increase throughput with negligible quality impact.

import torch
from transformers import (
    AutoProcessor,
    AutoModelForImageTextToText,
)

# Native FP32 (highest fidelity, ~32 GB VRAM, recommended for fine-tuning)
model = AutoModelForImageTextToText.from_pretrained(
    "Quatfit/Quatfit-Mini",
    torch_dtype=torch.float32,
    device_map="auto"
)

# Recommended for inference: cast to BF16 (~16 GB VRAM)
model = AutoModelForImageTextToText.from_pretrained(
    "Quatfit/Quatfit-Mini",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

processor = AutoProcessor.from_pretrained(
    "Quatfit/Quatfit-Mini"
)

---

Text Generation

messages = [
    {
        "role": "user",
        "content": "Write a Python implementation of binary search."
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_tensors="pt"
)

outputs = model.generate(
    **inputs,
    max_new_tokens=512
)

print(processor.decode(outputs[0]))

---

Image Understanding

messages = [
    {
        "role": "user",
        "content": [
            {
                "type":"text",
                "text":"Describe this image."
            },
            {
                "type":"image",
                "image":"image.png"
            }
        ]
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_tensors="pt"
)

outputs = model.generate(
    **inputs,
    max_new_tokens=512
)

print(processor.decode(outputs[0]))

---

Long Context Example

text = open("document.txt").read()

messages = [
    {
        "role":"user",
        "content":f"Summarize:\n\n{text}"
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_tensors="pt"
)

outputs = model.generate(
    **inputs,
    max_new_tokens=1024
)

print(processor.decode(outputs[0]))

---

Performance

FP32 is the published, reference-quality precision; the speedups above are measured relative to it. BF16 casting and the quantized GGUF builds are derived from the same FP32 weights and carry negligible accuracy loss.

---

Benchmark Scores

Scores were measured on the FP32 reference weights; BF16 and Q4_K_M/Q5_K_M GGUF builds match these scores within noise.

---

Architecture

Quatfit Mini preserves the Google Gemma 4 multimodal architecture while introducing Quatfit-specific optimizations for inference and deployment.

Foundation

---

Text Decoder

---

Vision Encoder

• Gemma 4 Vision Transformer
• Patch Size 16×16
• Variable aspect ratio input handling
• Axial 2D-RoPE + 2D absolute position embeddings
• Native Visual Embeddings

---

Audio Encoder

• Gemma 4 Conformer (USM-style)
• Continuous representations (no vector quantization)
• Streaming Compatible
• Causal Chunk Attention

---

Quatfit Optimizations

Quatfit Mini extends Gemma 4 with deployment-focused optimizations including:

• Optimized GGUF conversion
• Faster llama.cpp inference
• Speculative decoding via an autoregressive MTP (multi-token-prediction) drafter
• Flash Attention 3
• Sliding Window Attention
• Grouped Query Attention + KV-cache sharing + key/value reuse
• Consumer GPU optimization
• Memory-efficient inference
• Quantization-aware deployment (GGUF, mobile int2/int4)

---

Training

Quatfit Mini is built upon the pretrained Google Gemma 4 model and further optimized by Quatfit AI Research through post-training techniques including:

• Supervised Fine-Tuning (SFT)
• Preference Alignment
• Instruction Tuning
• Multimodal Alignment
• Inference Optimization

The underlying foundation model retains the original Gemma 4 pretraining while Quatfit contributes additional post-training and deployment optimizations. The final checkpoint is published in FP32 to preserve full training precision for downstream fine-tuning.

---

GGUF Quantized Builds

For local inference with llama.cpp, Ollama, LM Studio, Jan, or Open WebUI, use the companion repository:

Quatfit/Quatfit-Mini-GGUF

It provides every legacy and K-quant format from F16 down to Q2_K, converted from these FP32 weights. Each text quant is paired with its own mmproj file, so vision and audio input work immediately after download — no separate multimodal conversion step required.

---

Cross Platform Support

See the GGUF repository for the full quant lineup, exact sizes, and paired mmproj files.

---

Hardware

Recommended

• For native FP32 (~32 GB VRAM): A100 (40/80 GB), H100, RTX 6000 Ada
• For BF16 (~16 GB VRAM): RTX 3090, RTX 4090, RTX A6000, A100
• For GGUF quantized builds (~5–9 GB VRAM): RTX 3060/3070 and above

Minimum

• GPU with ~6 GB VRAM (Q4_K_M / Q5_K_M GGUF)
• CPU inference through llama.cpp (GGUF)

The FP32 weights in this repository require significantly more VRAM (~32 GB) than the BF16 or GGUF derivatives. Most consumer GPUs (e.g. 24 GB cards) should load the model with torch_dtype=torch.bfloat16 or use a GGUF build rather than native FP32.

---

Responsible AI

Quatfit Mini may generate inaccurate, biased, or inappropriate outputs.

For production deployments:

• Verify critical information
• Apply RAG for factual grounding
• Use application-level safety filters
• Keep human oversight for high-risk domains

---

Citation

@article{quatfitmini2026,
    title={Quatfit Mini: A Gemma 4-Based Multimodal Model Optimized for Efficient Inference},
    author={Quatfit AI Research},
    year={2026}
}

---

License

Apache License 2.0. See LICENSE for details.

---

<p align="center"> <a href="https://huggingface.co/Quatfit/Quatfit-Mini">🤗 Hugging Face</a> • <a href="https://huggingface.co/Quatfit/Quatfit-Mini-GGUF">⚙️ GGUF Builds</a> • <a href="https://huggingface.co/Quatfit/Quatfit-Mini/resolve/main/Quatfit-Mini_Technical_Report.pdf">📄 Technical Report</a> </p>