msrh-zindi-magic

Magic — Zindi MSRH Multilingual Health Q&A (Top-1 Solution)

A LoRA-adapter ensemble that scored private LB = 0.730865 on the Zindi Multilingual Sexual & Reproductive Health Q&A challenge. Ships 19 LoRA adapters over 3 Qwen base models plus a per-row consensus ensemble builder that regenerates the submitted go.csv.

Model architecture

• Base models (3 backbones): Qwen3.5-27B, Qwen3.6-27B, Qwen3-32B.
• Adapter type: LoRA (peft), rank r=128, alpha=256, dropout 0.05, target_modules=all (7 modules: q/k/v/o/gate/up/down proj).
• Training: LlamaFactory + DeepSpeed ZeRO-3, bf16, AdamW, lr=2e-4, cosine schedule with warmup_ratio=0.05, 3 epochs, effective batch =64 (per_device=2 × grad_accum=4 × 8 GPUs).
• Retrieval: McGill-NLP/AfriE5-Large-instruct top-3 passages (Train+Val as candidate pool, per-subset filtering, self-mask on training queries).
• Few-shot demos: K ∈ {3, 4, 5, 7} same-subset AfriE5-nearest (Q, A) pairs prepended to each prompt.
• Prompt variants: v1 baseline + v8 anchored-extraction (shortened copy-verbatim instruction).
• Ensemble: per-row consensus pick across the 19 adapter predictions.
• Private LB: 0.730865 on the private test set (see go.csv).

Intended use

Answering maternal, sexual, and reproductive health questions posed in English and in four low-resource African languages: Akan (Aka_Gha), Amharic (Amh_Eth), Luganda (Lug_Uga) and Swahili (Swa_Ken) — together the 8 language×country subsets defined by the competition.

Primary intended users are:

• Research on retrieval-augmented multilingual medical Q&A.
• Reviewers reproducing the leaderboard result.

Out-of-scope: clinical decision-making, diagnosis, or any use case where factual correctness for a specific patient matters. The model has NOT been audited for medical safety.

Dependencies

Pinned versions and install instructions are in requirements/infer.txt (inference) and requirements/train.txt (training).

Hardware: reproduce on any 80GB GPU (H100 / A100). The launcher auto-detects visible GPUs and runs up to min(8, visible) predicts concurrently, so 1 GPU works (sequential, ~20-30h wall-clock) and 8 GPUs is the sweet spot (~2h wall-clock). No config changes needed.

Inference / reproduction (one command)

bash scripts/run_all.sh

This driver runs the full end-to-end recipe:

1. Loads each of the 19 LoRA adapters onto its base model via vLLM.
2. Generates predictions on the shipped test JSONLs (LF/data/).
3. Converts each generated_predictions.jsonl to a Zindi-format CSV.
4. Runs scripts/build_ensemble.py over the 19 CSVs to regenerate submission.csv.
5. MD5-verifies the regenerated CSV against the shipped go.csv.

Step-by-step (if you want to run individually):

# 1. Generate 19 per-adapter predictions (writes to predict_out/)
bash scripts/launch_all_predicts.sh

# 2. (JSONL → CSV conversion runs inline inside run_all.sh; no separate script)

# 3. Ensemble → final CSV (writes submission.csv + md5 check)
python scripts/build_ensemble.py

Full detail (env setup, LlamaFactory installation, retraining from scratch) is in README.md.

Known caveats & setup notes

Before running scripts/run_all.sh, be aware of the following (from an end-to-end audit of a fresh clone from this repo):

1. Base models are NOT included (license reasons). Reviewers must download the three Qwen backbones separately from Hugging Face and place them under hub/:

   Base model	HF link	Local path
   Qwen3.5-27B	https://huggingface.co/Qwen/Qwen3.5-27B	hub/Qwen3.5-27B/
   Qwen3.6-27B	https://huggingface.co/Qwen/Qwen3.6-27B	hub/Qwen3.6-27B/
   Qwen3-32B	https://huggingface.co/Qwen/Qwen3-32B	hub/Qwen3-32B/

   Example download:

   hf download Qwen/Qwen3.5-27B --local-dir hub/Qwen3.5-27B
   hf download Qwen/Qwen3.6-27B --local-dir hub/Qwen3.6-27B
   hf download Qwen/Qwen3-32B   --local-dir hub/Qwen3-32B
   

   If a repo ID 404s on your side, use a compatible mirror (e.g. an unsloth/ upload of the same weights).

2. base_model_name_or_path in every adapter_config.json points at /mnt/msrh/Magic_submission/hub/<base> — this is a submission-time fake path. Two options:

   • Extract this repo into /mnt/msrh/Magic_submission/ (may need sudo mkdir /mnt/msrh first) and populate hub/ there — no code changes.
   • Or edit base_model_name_or_path in each adapter config to point at your local snapshot / HF repo ID.

3. scripts/launch_all_predicts.sh auto-locates its workspace root from the script path (default: parent dir of scripts/). If you want to point at a different location, override the env var:

   ROOT=/my/extract/path bash scripts/launch_all_predicts.sh
   

4. First-run vLLM warm-up is slow — the FlashInfer GDN prefill kernel is JIT-compiled on the first launch (~1 min extra per GPU). vLLM also suggests --gdn-prefill-backend triton as an alternative if you want to skip JIT; not required for correctness.

5. Regenerated submission.csv matches go.csv byte-for-byte only on identical hardware / kernel / vLLM state. vLLM inference is not deterministic across hardware, driver versions, or torch.compile / FlashInfer cache states. On a fresh environment, expect ~60-70% of rows to match go.csv byte-for-byte; the remaining rows will be paraphrases of the same underlying answer. Functional LB equivalence (ROUGE metrics) is what actually matters for evaluation.

Citation

If you use this work, please cite the Zindi competition:

Zindi Africa. "Multilingual Health Question Answering in Low-Resource
African Languages Challenge", 2026. https://zindi.africa/competitions/
multilingual-health-question-answering-in-low-resource-african-languages-challenge

License

Apache-2.0 for the adapter weights and code in this repository. The base Qwen models carry their own licenses (see the corresponding HF repos).