AgriQA-Phi-MoE-1B

AgriQA-Phi-MoE-1B

Fine-tuned LoRA adapter on microsoft/Phi-tiny-MoE-instruct (~1B sparse mixture-of-experts) trained on the AgriSci-QA benchmark for agricultural scientific question answering.

<!-- ## Performance vs. larger dense models | Model | Params | ROUGE-L | Semantic Grounding | OOD Transfer | |---|---|---|---|---| | **Phi-tiny-MoE (this model)** | **~1B** | **0.6233** | **0.705** | **−0.95%** | | Phi-3.5-mini | 3.8B | 0.4919 | 0.808 | −16.2% | | Qwen3-4B | 4B | 0.4333 | 0.165 | +41.0% | | LLaMA-3.1-8B | 8B | 0.5226 | 0.795 | −30.7% | This 1B MoE model outperforms all 3.8B–8B dense baselines on in-distribution ROUGE-L and shows near-zero OOD degradation, demonstrating that sparse activation preserves base-model generalisation that dense LoRA erodes. ## Training Configuration - **Base model**: `microsoft/Phi-tiny-MoE-instruct` - **Method**: LoRA with rsLoRA scaling (r=64, lora_alpha=32, dropout=0.05) - **Target modules**: q_proj, k_proj, v_proj, o_proj, w1, w2, w3 - **Dataset**: 9,989 AgriSci-QA training pairs across 7 reasoning types (factual, multi-hop, causal, temporal, comparative, numerical, counterfactual) - **Hardware**: NVIDIA RTX A4500 (20 GB), single GPU ## Usage ```python from transformers import AutoTokenizer, AutoModelForCausalLM from peft import PeftModel base = AutoModelForCausalLM.from_pretrained( "microsoft/Phi-tiny-MoE-instruct", torch_dtype="auto", device_map="auto" ) model = PeftModel.from_pretrained(base, "sayande/AgriQA-Phi-MoE-1B") tokenizer = AutoTokenizer.from_pretrained("sayande/AgriQA-Phi-MoE-1B") prompt = "Why does drought stress reduce photosynthesis and grain yield?" inputs = tokenizer(prompt, return_tensors="pt").to(model.device) output = model.generate(**inputs, max_new_tokens=256, temperature=0.7) print(tokenizer.decode(output[0], skip_special_tokens=True)) ``` ## Citation ```bibtex @article{de2026agrisciqa, title={{AgriSci-QA}: A Knowledge-Grounded Benchmark for Agricultural Scientific Question Answering}, author={De, Sayan and Sanyal, Debarshi Kumar and Mukherjee, Imon}, journal={IEEE Transactions on Knowledge and Data Engineering}, year={2026} } ``` -->