nilamind-gemma-3-4b-GGUF

NilaMind — therapy-tuned Gemma-3-4B (GGUF) · ⚠️ Research preview

⚠️ Research preview — NOT for clinical, production, or unsupervised use. This is an experimental, known-imperfect small model. It is not a medical device, not a therapist, and not a crisis service, and it has no built-in crisis-safety layer — it must sit behind a separate safety layer (see Limitations). Please read this whole card before using it.

A QLoRA fine-tune of google/gemma-3-4b-it, distilled toward the warm, brief voice of Nila — the on-device companion in NilaMind, a privacy-first mental-health app that runs entirely on the phone. This repo holds the Q4_K_M GGUF (~2.3 GB) that the app loads on-device via llama.cpp / llama-cpp-capacitor.

It is shared as a research artifact — so others can study the recipe, reproduce it, and learn from its failure modes — not as a deployable mental-health assistant.

⚠️ Limitations (read first)

• Repetitive / formulaic phrasing — the main practical limitation. Replies tend toward a sameness (similar openers and structure across turns). It's a small model and can be bland, repeat itself, or miss nuance.
• On "role-confusion" (corrected). An earlier single-turn eval suggested the model sometimes replied in the patient's voice (~25–33%), and it was briefly marked NO-GO. On investigation this was an evaluation-harness artifact, not a model defect: that eval prompted the model with [system, user] and no prior assistant turn — which the app never does. In the deployment prompt shape, where Nila's greeting is seeded as the first assistant turn, replies stay in character (re-verified with a 26-generation probe across single-turn, multi-turn, and minimal-reply conversations: 0 role-confusion). The training data was also checked — 0% of assistant turns are in the patient's voice. Always run this model with a seeded assistant greeting, the way the app does.
• No safety layer in the weights. In NilaMind, crisis detection is a separate, deterministic layer that runs around the model (keyword scan + a small on-device classifier — the app's "§9"). The model alone does no crisis handling. If you use these weights, you must supply your own safety layer.
• Small, English-only, not factual. A 4B model, Q4-quantised — fast on-device, but prone to confident errors. Do not use it for facts, diagnosis, or advice.
• Persona-dependent. It was trained with NilaMind's "Nila" persona as the system prompt; behaviour depends on a similar system prompt.
• Internal eval only. No clinical validation, no regulatory review.

Intended use

• Research into small, on-device, distilled mental-health support models and their failure modes.
• A reference for the on-device brain of the NilaMind app.
• Not for end-user mental-health support without a robust external safety layer, human oversight, and substantially more evaluation.

What it is

• Base: google/gemma-3-4b-it (Gemma-3, instruction-tuned).
• Method: QLoRA SFT — r=16, α=16, lr=1e-5, ≤3 epochs, bf16 (Gemma-3 requires bf16; fp16 produces inf grads), plus a small (~3%) safety/refusal mix to preserve refusals after fine-tuning (Qi et al. 2023, arXiv:2310.03693; Bianchi et al. 2023, arXiv:2309.07875).
• Distillation teachers — open models only: Qwen3.5-397B and gpt-oss-120b, whose licenses permit distillation. No Anthropic/Claude model was ever used as a teacher — Claude appears only as an evaluation baseline (see below), never in training.
• Data: ~1,600 teacher-generated multi-turn therapy dialogues in Nila's persona (client turns seeded from CounselChat, MIT) + a safety/refusal set. No real user data.
• Format: v2-4b-Q4_K_M.gguf (~2.3 GB), for llama.cpp / llama-cpp-capacitor; uses the Gemma-3 chat template.

Evaluation (internal, not clinical)

Evaluated MindEval-style across 5 clinical-communication axes, with Claude used only as the judge/patient baseline (evaluating against, never training on — per Anthropic's terms).

Usage (llama.cpp)

hf download sampathmannam/nilamind-gemma-3-4b-GGUF v2-4b-Q4_K_M.gguf --local-dir .

# Research only. Add your own crisis-safety layer before ANY real use.
llama-cli -m v2-4b-Q4_K_M.gguf --chat-template gemma -cnv \
  -p "You are Nila, a warm, brief, on-device mental-health companion. ..."

License & attribution

Built with Gemma. These weights are a derivative of Google's Gemma-3 and are governed by the Gemma Terms of Use and the Gemma Prohibited Use Policy — not a standard open-source license. By downloading or using them you agree to those terms (which prohibit harmful uses), and any redistribution must propagate them.

• Base: google/gemma-3-4b-it © Google, under the Gemma Terms.
• The NilaMind app code (separate repo) is Apache-2.0; these model weights are not — they are Gemma-licensed.

Links

• App, design & safety docs: https://github.com/sampathmannam/nilamind
• See the app's SAFETY.md for the crisis-safety posture this model is designed to sit behind.

---

Shared in the spirit of open research. If you build on it: keep a real safety layer, run it with a seeded assistant greeting (its deployment shape), and don't present it as a usable therapist.