sahayak-e2b-gguf

Sahayak-E2B — Q4_0 GGUF, runs on a phone's NPU with no network

Quantised GGUF build of Sahayak, an offline emergency-response assistant for disaster zones where cell towers and internet are down. It is a QLoRA fine-tune of google/gemma-4-E2B-it (merged to full weights), converted to GGUF and quantised to Q4_0 for on-device inference with llama.cpp on CPU, Adreno GPU (OpenCL), or Hexagon NPU (HTP).

This repo also ships a prebuilt Hexagon-v81 runtime, so an 8 Elite Gen 5 phone runs it on the NPU with no compiling.

• LoRA adapter + training details: kesav2k04/sahayak-e2b
• Project & full benchmark record: github.com/Kesav2k04/Sankat-Mochan
• Benchmarks (reproducible): docs/benchmarks/
• Evaluation report (one page): https://sahayak-e2b-benchmark.vercel.app/
• Context: Snapdragon Multiverse Hackathon — national finalist

Not a medical device. Interim first-aid guidance for when no clinician and no network are reachable. Read Limitations — there is a known anaphylaxis failure.

---

The file

Both unit figures describe the same file. Earlier docs said "3.11 GB", which was actually GiB.

Measured on-device performance

Single measured run, 2026-07-12, OnePlus 15 (CPH2745, SM8850, Snapdragon 8 Elite Gen 5, Hexagon v81, 15.5 GB RAM), llama.cpp ggml-hexagon, greedy decoding:

NPU offload verified, not assumed — verbose load logs show all 35 transformer layers assigned to HTP0, and the runtime aborts rather than silently falling back to CPU.

Against E4B: 2.2× the throughput and 1.69 GiB less on disk (~3.5 GB vs ~5.5 GB RAM to load). On a phone, that difference decides whether the model loads at all. Sahayak is second to stock E2B on raw throughput by 0.7 tok/s — the adapter is merged, so it adds no architectural cost.

⚠️ This is n = 1, with no thermal control, no time-to-first-token, and no energy measurement. An earlier "best quality-per-watt" claim has been withdrawn because no power was ever measured. Full caveats, including why this is the GGUF path and not the vendor QNN/QAIRT path: 02-ON-DEVICE-NPU-RUNTIME.md.

Tested silicon: Snapdragon 8 Elite Gen 5 (Hexagon v81) only. No benchmark exists for Snapdragon X Elite — please don't cite one.

Quality vs the base model

50 held-out prompts, never seen in training, base vs fine-tune, identical system prompt and greedy decoding. [R] = reproducible via verify_benchmarks.py · [H] = team-graded, not independently recomputable.

Safety [H]: on three adversarial prompts the base model broadcast raw GPS coordinates, relayed a false claim that would divert aid, and agreed to falsify casualty numbers. Sahayak refused all three.

Note the quantised build has not been separately re-evaluated — these figures come from the fp16 adapter. A quantisation ablation is on the roadmap.

---

Run with llama.cpp

# CPU / any platform
llama-cli -m sahayak-gemma-Q4_0.gguf -p "<start_of_turn>user
You are Sahayak, an offline emergency-response assistant. Be brief, calm, practical.

Someone's arm is bleeding heavily. First-aid steps?<end_of_turn>
<start_of_turn>model
" -n 512 --temp 0 -no-cnv

Run on the Snapdragon NPU (turnkey, no build) — 8 Elite Gen 5

This repo ships a prebuilt Hexagon-v81 runtime in npu-hexagon-v81/ (llama.cpp ggml-hexagon + libggml-htp-v81.so).

# On a PC with adb (phone in USB-debugging mode):
huggingface-cli download kesav2k04/sahayak-e2b-gguf --local-dir sahayak
adb push sahayak/sahayak-gemma-Q4_0.gguf sahayak/npu-hexagon-v81 /data/local/tmp/sh/
adb shell "chmod +x /data/local/tmp/sh/npu-hexagon-v81/bin/*; \
  cd /data/local/tmp/sh/npu-hexagon-v81 && sh run-npu.sh 'first-aid for a deep cut?'"

The published benchmark used -ngl 99 --device HTP0 --no-mmap --ctx-size 2048 -t 6 --temp 0. Because decoding is greedy, a correct reproduction returns byte-identical text.

Needs ~3.5 GB free RAM — reboot / close apps first, or --no-mmap will thrash. Other Snapdragon chips: rebuild llama.cpp with the arm64-android-snapdragon-release preset for your Hexagon version. See the llama.cpp Snapdragon docs and npu-hexagon-v81/NOTICE.md.

Prompt format

Standard Gemma chat template (<start_of_turn>user … <end_of_turn>\n<start_of_turn>model\n). Use the Sahayak system prompt and greedy decoding to reproduce evaluated behaviour.

The model emits a short [Start thinking] … [End thinking] block before its final answer — give it enough tokens (-n 512) to reach the answer, or strip the block for a clean display. The app sets enable_thinking = false; the benchmark runs above left it on, so their latency is pessimistic relative to app behaviour.

---

Limitations

1. ⚠️ Anaphylaxis fails in BOTH this model and the base model. Neither recognises throat-tightening plus wheezing after stings as anaphylaxis, and neither mentions an adrenaline auto-injector.
2. Non-English output is unreliable. Multilingual accuracy moved only 38% → 43%. The model sometimes degenerates into repetition or emits garbled packets with hallucinated fields in Marathi/Telugu/Bengali — roughly 3 training examples per non-English language. Do not deploy for non-English users yet.
3. Numeric reasoning can regress — it once allocated 36 of 18 available volunteers.
4. Noisy/garbled input is frequently misread by both models.
5. 4-bit quantisation can shift outputs, and this quantised build was not separately re-evaluated.
6. General capability was never re-tested (no MMLU / MedQA / IFEval), so catastrophic forgetting from narrow fine-tuning cannot be ruled out.
7. Accuracy grades are unblinded and team-assigned, with no second rater.

Full adversarial critique and the experiments that would close each gap: 03-LIMITS-AND-ROADMAP.md.

Intended use

Offline first-aid and coordination guidance in low-connectivity disaster settings, as a component of a human-supervised response system. Not a medical device; not a substitute for professional care; not for autonomous dispatch. It gives first-aid steps only and directs users to professional help.

Licence

Derivative of Google Gemma — governed by the Gemma Terms of Use. By using these weights you agree to the Gemma Prohibited Use Policy. Gemma is not an OSI-approved open-source licence. The bundled llama.cpp binaries are MIT (© ggml-org / llama.cpp contributors). The Sahayak Emergency Dataset v2 is Apache-2.0.

Citation

@software{sahayak_e2b_2026,
  title  = {Sahayak-E2B: an offline on-device disaster-response fine-tune of Gemma 4 E2B},
  author = {Jayakumar, Kesav},
  year   = {2026},
  url    = {https://github.com/Kesav2k04/Sankat-Mochan}
}

Sahayak is one component of Sankat-Mochan, a team project. It is a Gemma 4 E2B fine-tune, distinct from the Qwen3-4B model used elsewhere in that project for triage.