onebee-gf-distill-v1-gguf

onebee-gf-distill-v1-gguf

GGUF quantizations (F16 reference plus 12 quant levels down to Q2_K, plus vision projector) of the current-best post-distillation companion checkpoint, for llama.cpp-based on-device inference.

Model Overview

GGUF conversion and quantization of onebee-gf-distill-v1 — the current best checkpoint from small-mind-companion (LoRA SFT → DPO → on-policy distillation from an 8B-class teacher, on top of gemma-4-E2B-it) — for llama.cpp-based local/on-device inference.

• What it is: the same weights as onebee-gf-distill-v1, converted to GGUF.
• What it does: runs the current-best companion checkpoint on-device without a Python/ transformers runtime.
• What makes it different from onebee-gf-dpo-v1-scale-gguf: quantized from the post-distillation checkpoint, not the pre-distillation one — this is the checkpoint the project's headline H23 results are actually about.
• Real generation testing found Q2_K broken for this checkpoint — garbled special-token output and a non-terminating repetition loop, not usable. Q3_K_S and above verified coherent and in-character. See Limitations.
• Base model: google/gemma-4-E2B-it.

Model Details

Intended Use

Intended Use

Local/on-device inference via llama.cpp where a transformers/Python runtime isn't available or desired. Use Q3_K_S or above — Q2_K was found broken on real generation testing for this checkpoint (see Limitations).

Out-of-Scope Use

Not evaluated or intended for: safety-critical decisions, medical/legal/financial advice, or any deployment where a wrong or overconfident answer causes real harm. This is a research artifact from an open-source project studying post-training and memory architecture on small models — see the project README for the full research framing before using it in any production context.

Capabilities

• Text and vision (image) input
• 12 quant levels (F16 → Q2_K), though Q2_K is not usable — see Limitations
• Verified in-character companion-style generation at Q3_K_S, Q3_K_M, Q4_K_S, Q4_K_M

Files

Quick Start

Installation

# build llama.cpp, or install a prebuilt release: https://github.com/ggml-org/llama.cpp

Usage

Text:

llama-cli -m distill-v1-Q4_K_M.gguf -sys "You are a warm AI companion in an ongoing relationship with the user." -p "Hello!" -st

Vision (needs --jinja):

llama-mtmd-cli -m distill-v1-Q4_K_M.gguf \
  --mmproj mmproj-distill-v1-f16.gguf \
  --image your_image.png -p "What is in this image?" --jinja

Evaluation

Real generation quality checks (companion system prompt, "What is your favorite color?"):

Full methodology: docs/quantization_results.md.

Limitations

• Q2_K is broken for this checkpoint — confirmed via real generation testing, not file-size/load checks. Do not use it. Confirmed NOT distillation-specific: the pre-distillation dpo-v1-scale checkpoint's own Q2_K file fails the same way — Q2_K was simply never safe for this model family, untested until this pass. See the full writeup.
• No accuracy/quality regression measured against the project's own PMB eval harness at each quant level — verified "coherent and on-topic" via real generation tests, not "measurably as accurate as F16."
• No importance-matrix (imatrix) calibration used for these quants.
• This is a research checkpoint from an active, in-progress open-source project.

Other Checkpoints From This Project

Citation

@software{small_mind_companion,
  title  = {small-mind-companion: Post-training and cognitive architecture for a small multimodal companion LLM},
  author = {arjhinety},
  year   = {2026},
  url    = {https://github.com/arjhinety/small-mind-companion}
}

License

Apache-2.0, inherited from the base model (google/gemma-4-E2B-it).