Bible-Assistant-Qwen3.5-4B-v3.2-GGUF

Bible AI Assistant v3.2 — GGUF

GGUF quants of Ttimms/Bible-Assistant-Qwen3.5-4B-v3.2 — a Qwen3.5-4B continued-fine-tune for retrieval-grounded Bible Q&A, and the first checkpoint in the project's line to clear every acceptance gate: 80.3% verbatim verse recall, 98.9% citation, 1.9% hallucination, and the strongest result in its size class on a cross-encoder semantic metric against every other ~4B bible-tuned model tested. See the base repo for the full model card, version history, and the honest evaluation against larger models too.

Architecture

graph TD
    Base["Qwen/Qwen3.5-4B"]
    V31["v3.1 adapter (thematic-synthesis SFT)"]
    Cont["DMT-style continued fine-tune - RAFT-fixed thematic_qa + rehearsal, lr 5e-5"]
    Merge["merge adapter -> bf16"]
    Conv["convert_hf_to_gguf --no-mtp + llama-quantize"]
    ST["Bible-Assistant-Qwen3.5-4B-v3.2 (safetensors)"]
    GG["...-v3.2-GGUF (Q4_K_M / Q5_K_M / Q6_K / Q8_0 / F16)"]
    NV["...-v3.2-GGUF NVFP4 (advanced-gguf-quantizer, Blackwell-native)"]
    RAG["hybrid RAG: dense (nomic) + BM25 + RRF + bge-reranker-v2-m3"]
    LLM["llama.cpp / LM Studio / Ollama"]
    Base --> V31 --> Cont --> Merge --> ST
    Merge --> Conv --> GG
    Merge --> Conv --> NV
    ST --> RAG --> LLM
    GG --> LLM
    NV --> LLM

Download

Grab one file, not the whole repo.

NVFP4-GGUF (Blackwell-native)

bible-v3.2-4b-nvfp4.gguf is a mixed NVFP4 quant built with advanced-gguf-quantizer (deep-mode tensor search, allow_diagnostic sensitivity ranking, hard-gated on tail KLD) targeting native Blackwell FP4 tensor cores (BLACKWELL_NATIVE_FP4=1 confirmed on an RTX 5070 Ti / SM120 at eval time) — a genuinely different quantization technique than the k-quant ladder above, not just another point on the same size/quality curve.

Real, measured result — a mixed outcome, stated plainly rather than rounded to a single "better/worse": identical methodology, same bf16 reference, same calibration/eval corpus, both files run through llama-perplexity --kl-divergence back-to-back.

NVFP4 is smaller and has a lower mean-token perplexity ratio than Q4_K_M despite spending fewer bits per weight (4.66 vs 5.15 bpw) — but it has a higher mean KL-divergence against the bf16 reference, meaning its per-token probability distribution drifts further from the base model even where average next-token accuracy holds up. Passed this project's own coherence + hard tail-KLD gate (p99/p999) either way.

Which to pick: Q4_K_M if you want the closest per-token match to the unquantized model's behavior (the safer default for a RAG assistant that quotes verses verbatim). NVFP4 if size and mean perplexity matter more than tail-distribution fidelity, or you're specifically exercising Blackwell's native FP4 path.

Status (2026-09-17): this is the current NVFP4 candidate, not a final answer. A follow-up run (nvfp4_mxfp6 profile, promoting the most KLD-sensitive tensors to MXFP6_E2M3 instead of leaving the whole model at a uniform NVFP4 floor) is queued to test whether real additional bits closes the KLD gap to Q4_K_M without losing NVFP4's size/PPL win. This card will be updated with that result when it lands — check back before assuming this is the final NVFP4 quality bar for this checkpoint.

Run it in

• llama.cpp — llama-server -m <file>.gguf -ngl 99
• LM Studio (bundles a recent llama.cpp)
• koboldcpp
• Jan
• text-generation-webui
• Ollama — once its bundled llama.cpp includes this arch (see Requirements)

Requirements

Qwen3.5 is a hybrid architecture (qwen35 / Gated-DeltaNet + attention). You need a recent llama.cpp — a build that includes the qwen35 hybrid arch (commit 3173a56 or newer). Verified working with llama-cli/llama-server from a source build — coherent output, correctly quoted John 3:16, 175 tok/s on an RTX 5070 Ti.

• ✅ llama.cpp (current): llama-server -m bible-v3.2-4b-Q4_K_M.gguf -ngl 99
• ✅ LM Studio (recent versions bundle a current llama.cpp)
• ⚠️ Ollama 0.33.x: the bundled llama.cpp is too old for the qwen35 arch (check_tensor_dims: tensor 'blk.32.attn_norm.weight' not found). Use once Ollama updates its runtime, or run llama.cpp directly.

bible-v3.2-4b-nvfp4.gguf was built and evaluated end-to-end on an RTX 5070 Ti (Blackwell, SM120) using llama.cpp's native NVFP4 GEMM path. It has not been verified on non-Blackwell hardware in this session — if your llama.cpp build or GPU lacks native FP4 kernel support, prefer one of the k-quant files above.

Thinking mode

The Qwen3.5 chat template defaults to thinking on. This model was fine-tuned without <think> traces, so for a grounded RAG assistant you want it off:

• llama.cpp /v1/chat/completions: pass "chat_template_kwargs": {"enable_thinking": false}, or
• use a chat template that emits a closed empty <think>\n\n</think>\n\n after <|im_start|>assistant\n (the project's deployment/pc/Modelfile does this).

Intended use

Retrieval-augmented Bible Q&A — the model expects retrieved verses in a Context: block, then the question. It is not designed for context-free use, medical / legal / financial advice, counselling (it redirects those to a pastor / crisis line), or authoritative theological rulings.

License

Weights: Apache-2.0 (inherits from Qwen3.5-4B). Project code: MIT. Bible translations: public domain. See the base repo for full attribution and the version history that led to this checkpoint.