mhqa-itu-adapters

MHQA · ITU · Zindi Challenge — Generation Adapters

DariusTheGeek/mhqa-itu-adapters · the trained-weight component of the winning solution to the Multilingual Health Question Answering in Low-Resource African Languages Challenge (ITU), hosted on Zindi. Data by the HASH consortium (Hub for AI in Maternal, Sexual & Reproductive Health).

This repo hosts 3 LoRA adapters (the generation members). The full retrieval + generation + judge system, and one-command reproduction, live in the code repo: Multilingual-Health-QA-ITU-Zindi-Challenge on GitHub.

⚠️ Not medical advice. This is a research artifact built for a benchmark competition. It is not a medical device, is not for clinical use, and its outputs must not be used to diagnose, treat, or make health decisions. Sexual/reproductive-health content may be inaccurate, incomplete, or unsafe.

---

Model summary

The three adapters

raft_r1 was trained on top of the merged gen7454 model (RAFT round-1), not on the raw base.

---

Architecture & approach

The competition is scored 0.37·ROUGE-1 + 0.37·ROUGE-L + 0.26·judge, so ~74 % of the score is lexical overlap with a hidden gold answer. The test set splits into two regimes:

• Retrieval subsets (~60 %: Eng_Uga, Eng_Ken, Eng_Eth, Swa_Ken, Lug_Uga) — the gold answer is almost always a verbatim reuse of a Train+Val answer-bank entry, already in the candidate pool. The job is selecting the canonical twin, done by LightGBM selectors that use a fine-tuned cross-encoder as a feature (not an argmax chooser). These are trained inline and deterministic — not weight files.
• Generation subsets (~37 %: Aka_Gha, Eng_Gha) — golds are original/local, so answers are generated. The 3 LoRA adapters in this repo each produce a K36 sample pool; a cross-model ROUGE-MBR medoid picks the consensus answer.

A final LLM-judge pass (gemma-4-31B-it) reverts off-topic picks (27 cached decisions).

What is a weight vs. deterministic code:

---

Intended use

• In scope: reproducing / validating the competition result; research on retrieval-vs-generation for low-resource multilingual health QA; a baseline for the 8 named country×language subsets.
• Out of scope: any clinical, diagnostic, or patient-facing use; generating medical advice; deployment as a health chatbot; languages/domains outside the 8 competition subsets (behaviour is untested and unsafe).

Training data

All data is from the Zindi competition (HASH consortium sexual/reproductive-health Q&A across Uganda, Kenya, Ghana, Ethiopia). No external or hand-authored data; no preprocessing outside the code. Raw Train.csv / Val.csv are Zindi competition data and are not redistributed — download them from the competition page. The adapters are trained on this data but do not contain it.

Evaluation

The exact 0.728509 requires the hidden Test gold + Zindi's hosted judge. common/eval.py computes the exact 0.37/0.37/0.26 blend given a gold file (and per-row judge scores); reproduce.sh proves the shipped output is byte-identical to the scored sub_v40.

Known limitations

• Coverage: only the 8 competition subsets/languages; no guarantees elsewhere.
• ROUGE-optimized: the system is tuned to a lexical-overlap metric; high ROUGE ≠ clinical correctness.
• Answer-bank reliance: retrieval subsets depend heavily on verbatim reuse of Train/Val bank answers, so generalization to genuinely novel questions is weak.
• Amharic: ROUGE is effectively 0 for Ge'ez script under the metric; Amh_Eth passes through generation.
• Generation is stochastic: the shipped pool uses temperature>0 sampling (reproducible up to sampling; a greedy demo path is deterministic). Not a source of clinical reliability.
• Health-safety: outputs may be inaccurate/unsafe; see the disclaimer above.

How to use

The adapters are components of a pipeline — use the GitHub repo, which pulls these weights automatically:

git clone https://github.com/DariusTheGeek/Multilingual-Health-QA-ITU-Zindi-Challenge.git
cd Multilingual-Health-QA-ITU-Zindi-Challenge
bash env/install.sh
bash stages/0_setup/fetch_artifacts.sh              # ~250 MB data/features (Path A, CPU)
bash reproduce.sh                                    # -> sub_v40, BYTE-IDENTICAL, minutes, no GPU

# Run the actual trained model (GPU ~48 GB):
bash stages/0_setup/fetch_artifacts.sh --adapters    # pulls these 3 adapters (4.7 GB)
python stages/1_generation/demo_generate.py --adapter models/adapters/gen7454 --base google/gemma-4-31B-it --n 2

Standalone load (PEFT):

from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained("google/gemma-4-31B-it", torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "DariusTheGeek/mhqa-itu-adapters", subfolder="gen7454")

Dependencies

Python 3.11. CPU reproduction: lightgbm, pandas, pyarrow, regex, rouge_score, scikit-learn (requirements.txt). GPU generation: unsloth / vllm + transformers/peft (env/requirements-serve.txt, env/requirements-train.txt).

License

The adapters are LoRA derivatives of gated Google base models; base weights are referenced, not re-hosted. Use is subject to the base licenses — gemma-4-31B-it: Gemma Terms of Use and medgemma-27b-text-it: Health AI Developer Foundations terms. Accept each on the Hub before downloading the bases. Competition data is governed by the Zindi competition rules.

Citation

Solution to the ITU Multilingual Health Question Answering in Low-Resource African Languages challenge (Zindi / HASH consortium). Author: DariusTheGeek. Companion dataset repo: DariusTheGeek/mhqa-itu-artifacts.