qwen35-4b-plantdisease-cpt

Qwen3.5-4B-Base Continued Pretraining — Plant Disease Domain

Continued pretraining of Qwen/Qwen3.5-4B-Base on a curated corpus of plant disease and agricultural literature.

Training Data

Total: ~125M tokens across {TOTAL_DOCS} documents

Training Configuration

• Base model: Qwen3.5-4B-Base
• Architecture: Hybrid GatedDeltaNet + Attention (32 layers, 2560 dim)
• Precision: bfloat16
• Optimizer: 8-bit AdamW (lr=2e-5, cosine schedule, 5% warmup)
• Effective batch size: 4 (batch=1 × grad_accum=4)
• Sequence length: 2048
• Gradient checkpointing: enabled
• Hardware: NVIDIA A100 80GB

Evaluation

• Final validation PPL: {FINAL_VAL_PPL}
• Best validation PPL: {BEST_VAL_PPL} at step {BEST_STEP}

Domain Coverage

• Plant pathology (fungal, bacterial, viral diseases)
• Crop protection and IPM
• Pathogen biology and epidemiology
• Agricultural practices
• Biocontrol agents

Limitations

• Domain-adapted base model (NOT instruction-tuned)
• Bilingual (English majority + some Indonesian)
• Best used as foundation for further SFT
• Knowledge cutoff: training data sources

Citation

This model was created as part of the MioLineara plant disease research project.

---

Trained: 2026-05-09