multilingual-rag-lora-consensus

LanAnh — Multilingual SRH-QA Solution

LoRA-pool solution for the Zindi Multilingual Health-QA in Low-Resource African Languages challenge. Six LlamaFactory-fine-tuned 27 B / 32 B Qwen variants are combined by a Pairwise Agreement consensus to produce the final answer per row.

---

1. Solution Summary

Four-stage pipeline. Each stage is one executable script.

Trained weights & one-click inference

The six LoRA adapters are published on the Hugging Face Hub, and reproduce_inference.ipynb runs the whole inference chain (download → retrieve → demo_prep → 6× vLLM → consensus) end to end:

Each adapter loads on its stock base model (Qwen/Qwen3.5-27B, Qwen/Qwen3.6-27B, Qwen/Qwen3-32B).

Pool — six recipes

The selected ckpt column is the checkpoint each recipe's batch_infer loads (checkpoint-{selected_step}); with save_steps: 100 these exist for every recipe. The 27B models are GDN (linear-attention) Qwen3.5/3.6 and require the qwen3_5_nothink template + processor at train time; Qwen3-32B is a dense text model and uses the stock qwen template.

All six configs use 3 training epochs (matching the LlamaFactory default for this dataset size). The pool spans two reference-block sizes (the _n5 / _n7 suffix) crossed with three Qwen base models — enough cross-architecture decorrelation for the consensus step to recover per-row mistakes.

Every per-cand config — both the LanAnh-side metadata (id, n_shots, runtime, LoRA rank) and the LlamaFactory training block — lives in a single YAML at LlamaFactory/configs/lananh/<id>.yaml. The pool definition for Stage D lives beside them at LlamaFactory/configs/lananh/pool.yaml.

---

2. Installation

Three independent conda envs (LlamaFactory ↔ vLLM ship incompatible torch ABIs):

conda env create -f envs/llamafactory.yml     # Stage A + B
conda env create -f envs/vllm.yml             # Stage C
conda env create -f envs/consensus.yml        # Stage D (CPU only)

LlamaFactory ships inside this repo at LlamaFactory/ (cloned from hiyouga/LLaMA-Factory). Stage 2 installs it editable on first run:

cd LlamaFactory && pip install -e ".[deepspeed]"

All six training configs live at LlamaFactory/configs/lananh/q*_n*.yaml, together with LlamaFactory/configs/lananh/ds_z3_config.json (the DeepSpeed ZeRO-3 config referenced by every training YAML).

If you only want to rerun the consensus on existing candidate CSVs, only the consensus env is required.

Project Structure

Every script defaults to WORKSPACE=/srv/lananh. Place under that root:

/srv/lananh/
├── data/
│   ├── Train.csv  Val.csv  Test.csv  SampleSubmission.csv     # Zindi raw
│   └── retrieval/  n5/  n7/                                   # populated by Stage A
├── models/
│   ├── afrie5/                                                # McGill-NLP/AfriE5-Large-instruct
│   └── base/Qwen3.5-27B/  Qwen3.6-27B/  Qwen3-32B/            # HF snapshots
└── adapters/  predictions/  submission.csv                    # populated by Stages B–D

One-shot run

export WORKSPACE=/srv/lananh

conda activate lananh-lf
bash runners/stage1_prepare_data.sh    # ~25 min on a single H100
bash runners/stage2_train_all.sh       # ~30 h aggregated (six SFT runs, 3 epochs each)

conda activate lananh-vllm
bash runners/stage3_infer_all.sh       # ~45 min, 6 GPUs in parallel

conda activate lananh-consensus
bash runners/stage4_consensus.sh       # ~5 min CPU → submission.csv

---

3. Dataset

• Zindi raw — Train.csv (29 815), Val.csv (6 686), Test.csv (2 618), SampleSubmission.csv (2 618). Drop under data/.

• Retrieval pool = Train ⊕ Val (36 501 rows). For each row, the top relevant passages are pulled via AfriE5 cosine similarity, restricted to the same language subset.

• Prior exemplars = nearest training-pool questions (same subset, ID-exclusive). Each user turn is rendered with XML-style tags:

  You are a specialist assistant for maternal, sexual and reproductive health
  questions posed in African low-resource languages. Reply strictly in {LANG}.
  
  Use the wrapped <evidence> blocks below as the single source of truth: copy
  their exact wording whenever it addresses the question, and add no caveats,
  opinions or self-references.
  
  <evidence id="1">...</evidence>
  <evidence id="2">...</evidence>
  
  <priors>
    <exemplar id="1">
      <prior_q>...</prior_q>
      <prior_a>...</prior_a>
    </exemplar>
    ...
  </priors>
  
  <query>...</query>
  
  <response>
  

• Tokenisation — at train time each config declares its own LlamaFactory template (qwen3_5_nothink for the 27B GDN models, qwen for the 32B dense model); at inference time the model's own chat template is applied via tokenizer.apply_chat_template(..., enable_thinking=False), so inference is template-agnostic across the pool.

Environment gotchas (read before running B/C)

These bit us during bring-up; skipping them reproduces the exact failures:

• 27B needs ninja in the vLLM env — Qwen3.5/3.6 are GDN models whose linear-attention kernels are JIT-compiled at load; without ninja on PATH, vLLM Stage C aborts compiling the fla kernels.
• Guard torchaudio for training — LlamaFactory's mm_plugin.py does a bare import torchaudio; if the LlamaFactory env's torch ABI post-dates its torchaudio build the import raises undefined symbol. Wrap it in try/except (already patched in the bundled LlamaFactory/).
• 32B inference needs TP=2 — Qwen3-32B does not leave enough KV cache on a single 80 GB GPU at gpu_memory_utilization=0.85; run its two recipes with tensor-parallel across 2 GPUs (or raise util to ≥0.90).
• GDN per-device cap — keep per_device_train_batch_size ≤ 2; the fla l2norm_bwd Triton kernel OOMs at per-device 4 regardless of free VRAM.
• Effective batch size — configs target eff_bs ≈ 60–64 (per_device 2 × grad_accum × nproc); the shipped grad_accum values assume the GPU counts we trained on. Adjust grad_accum to hold eff_bs ≈ 64 for a different GPU count.

---

4. Consensus — Pairwise Agreement

After Stage C we have 6 candidate CSVs, one per config, each with the same 2 618 rows. For every Test row the consensus step looks across all six candidate answers and keeps the one that is most consistent with the rest of the pool — i.e. the candidate every other candidate "agrees with" the most. Standalone-LB outliers get filtered out unless they happen to coincide with the majority opinion.

CLI

conda activate lananh-consensus
python -m pipeline.consensus \
    --config LlamaFactory/configs/lananh/pool.yaml \
    --workspace /srv/lananh \
    --out /srv/lananh/submission.csv

The script prints the resulting MD5, the per-config pick distribution and the output path so you can attach it as the Zindi submission.

Notes on reproducibility

• vLLM is deterministic under temperature=0 for the same hardware, driver, and max_num_seqs — cross-machine bit-for-bit reproduction is not guaranteed; functional LB reproduction within ±0.001 is.
• The consensus step itself is fully deterministic (pure NumPy).
• Every config pins its own seed (distinct per recipe — an extra decorrelation axis for the consensus step); LlamaFactory honours it for the LoRA init.

---

References

---

Repository tree

lananh-submission/
├── README.md
├── submission.csv                  # final Zindi submission (regenerated by Stage D)
├── pyproject.toml
├── envs/                           # conda env yamls
│   ├── llamafactory.yml   vllm.yml   consensus.yml
├── LlamaFactory/                   # cloned hiyouga/LLaMA-Factory (`pip install -e .`)
│   └── configs/lananh/             # six per-cand merged YAMLs + ds_z3_config + pool.yaml
├── pipeline/                       # source code
│   ├── retrieve.py     demo_prep.py     train_lf.py
│   ├── batch_infer.py  consensus.py
└── runners/                        # stage1..stage4 .sh