Qwen3-8B-Kintsugi-GGUF

Qwen3-8B-Kintsugi-GGUF

GGUF quantizations (Q4_K_M primary, Q5_K_M fallback) of ai-sherpa/Qwen3-8B-Kintsugi — a QLoRA fine-tuned for The Kintsugi Garden, a Hugging Face Build Small Hackathon submission that turns dreams, journal entries, and inner transitions into hedged, six-section symbolic reflections. This repo is what the live Space build-small-hackathon/Kintsugi-Garden loads in production via llama-cpp-python; the default file is Qwen3-8B-Kintsugi-Q4_K_M.gguf.

What this model is for

The Kintsugi Garden is a symbolic reflection surface, not a chat assistant. The fine-tune teaches the base Qwen3-8B model the editorial voice of the Garden's reading pipeline:

• Hedged language. Every claim is conditional — "may suggest", "could reflect", "often appears alongside" — never declarative.
• 42-symbol Jungian lexicon. The fine-tune is grounded in a curated set of archetypes (shadow, threshold, water, ascent, etc.) rather than freeform dream interpretation.
• Six-section output structure. Every reading has a fixed scaffold: Image · Symbols · Pattern · Tension · Invitation · Caveat. Format integrity is enforced by a downstream linter.
• Four-layer safety stack. The model is the third layer; a deterministic safety gate, a mundane-alias suppressor, the LLM's trained voice rules, and post-LLM sanitizers all collaborate to refuse crisis-routing, prediction, diagnosis, or advice.

It is the narrow model the Garden needs to ship under the Build Small constraint: small enough for free CPU inference on ZeroGPU, specific enough that 8B is genuinely sufficient.

What this model is NOT for

• Not therapy, diagnosis, prognosis, or advice. Crisis input is routed away from the LLM entirely by the safety gate; this model never sees it.
• Not prediction. The fine-tune is trained to refuse future-tense or outcome-claim framings.
• Not a general-purpose assistant. The QLoRA is narrow to the Garden's voice and will underperform on unrelated tasks (coding, math, summarization, chat).
• Not a substitute for the merged HF model. If you want PyTorch inference, transformers integration, or further fine-tuning, use ai-sherpa/Qwen3-8B-Kintsugi instead.

Files in this repo

Both quants pass the QA harness; Q4_K_M was selected as the default on the size/quality trade-off appropriate for free-tier Spaces.

How the Kintsugi Garden Space loads it

The Space's app.py reads two environment variables (with these defaults baked in since the TODO #14 ship commit 70ed23b):

KINTSUGI_LLAMA_REPO=ai-sherpa/Qwen3-8B-Kintsugi-GGUF
KINTSUGI_LLAMA_FILE=Qwen3-8B-Kintsugi-Q4_K_M.gguf

Source: app.py:81-82 in the GitHub repo. First boot downloads the GGUF into the HF cache; subsequent boots are instant. To swap to the Q5 fallback, override KINTSUGI_LLAMA_FILE in the Space's secrets — no code change required.

Inference example with llama-cpp-python

The Space loads this model in-process; the snippet below mirrors that path closely enough to reproduce a Garden-style reading locally.

from huggingface_hub import hf_hub_download
from llama_cpp import Llama

model_path = hf_hub_download(
    repo_id="ai-sherpa/Qwen3-8B-Kintsugi-GGUF",
    filename="Qwen3-8B-Kintsugi-Q4_K_M.gguf",
)

llm = Llama(
    model_path=model_path,
    n_ctx=4096,
    n_threads=8,
    verbose=False,
)

system_prompt = (
    "You are the Kintsugi Garden's symbolic reflection voice. "
    "Respond ONLY in the six-section format: Image, Symbols, "
    "Pattern, Tension, Invitation, Caveat. Use hedged language "
    "('may suggest', 'could reflect'). Never diagnose, predict, "
    "or advise. Ground symbols in the Jungian lexicon."
)

dream = (
    "I was walking through a flooded house. The water was warm "
    "and I wasn't afraid, but I couldn't find the staircase."
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": dream},
    ],
    temperature=0.7,
    max_tokens=512,
)

print(response["choices"][0]["message"]["content"])

The live Space wraps this in the four-layer safety stack and the format linter; calling the raw model directly skips those guards.

Quantization methodology

• Tool: llama.cpp at tag b9556.
• Pipeline:
  1. convert_hf_to_gguf.py on the merged ai-sherpa/Qwen3-8B-Kintsugi → fp16 intermediate GGUF.
  2. llama-quantize → Q4_K_M (primary).
  3. llama-quantize → Q5_K_M (fallback).
• Smoke test: each quant exercised through llama-cpp-python with a single representative dream prompt before push; both produced well-formed six-section output.
• Reproducibility: the full script is scripts/convert_to_gguf.py in the GitHub repo.

Source model and training

The merged Hugging Face-format weights live at ai-sherpa/Qwen3-8B-Kintsugi. See that card for the full training detail; in brief:

• Base: Qwen/Qwen3-8B
• Method: QLoRA, r = 16, α = 32, attention + MLP projection targets
• Epochs: 3
• Dataset: ~50 hand-curated examples spanning Jungian motifs, mundane entries, safety triggers, edge cases, and adversarial framings — assembled by scripts/build_training_data.py
• Hardware: single H100 on Modal
• Total training cost: ~$3

Evaluation

The fine-tune is gated by a QA acceptance harness that compares it against the un-tuned Qwen3-8B baseline on the same prompts. Verdict for this quant: PASS programmatically.

Full side-by-side comparison artifact: qa-results/compare-1f0699c2ca-vs-5eefd4a3dd-20260608T224802.md in the GitHub repo. Protocol: QA_PROTOCOL.md.

Hardware envelope

• HF Space (production): runs on the default free-tier ZeroGPU hardware in-process via llama-cpp-python. No GPU required for the GGUF path — the Q4_K_M file fits comfortably in the Space's RAM and inference completes within the user-facing latency budget.
• Apple Silicon (dev): local CPU inference at ~25–50 seconds per reading on an M-series MacBook with n_threads=8. This is the primary dev-loop hardware for the project.
• Linux CPU / GPU offload: standard llama-cpp-python n_gpu_layers parameter applies; not exercised by the Space but works locally.

License

Apache 2.0, inherited from the Qwen3-8B base. Same terms apply to both quants in this repo and to the merged source model.

Links

• Live Space: build-small-hackathon/Kintsugi-Garden
• Source code: github.com/AI-Sherpa/hfHack_KintsugiGargen
• Merged HF model: ai-sherpa/Qwen3-8B-Kintsugi
• Base model: Qwen/Qwen3-8B