mdco-chartqa-llava

MD-CO ChartQA — llava Reproducibility Bundle

Frozen reproduction of MD-CO (Multi-task Distillation for Chart QA + OCR) on llava-hf/llava-1.5-13b-hf over ChartQA (Masry et al. 2023).

Results (paper Table 1 convention, test split, relaxed accuracy %)

Convention: human = trained on train_human → eval test_human; augmented = trained on train_aug → eval test_aug; avg = (human + aug) / 2.

MD-CO knowledge-distillation tuning (human split)

*augmented column here = same human-trained checkpoint evaluated on test_aug (diagnostic).

Limitation: LLaVA-1.5 is near-chance on charts zero-shot, so it cannot absorb the teacher distribution — KD does not help and MFT remains the best strategy. This is a documented capacity-floor boundary of MD-CO.

Repo layout

• code/ — full source (train.py, evaluate.py, configs/, models/, trainers/, data/, scripts/)
• checkpoints/ — trained LoRA adapters: llava_dt_augmented, llava_dt_human, llava_ft_augmented, llava_ft_human, llava_mdco_augmented, llava_mdco_human, llava_mdco_human_T3, llava_mdco_human_ocrT3, llava_mft_augmented, llava_mft_human
• results/ — per-run eval summary + predictions JSON
• REPRODUCIBILITY.md — env + exact commands

Quick eval

pip install -r code/requirements.txt
python code/evaluate.py --model llava --strategy mdco --data both --split test \
  --checkpoint checkpoints/llava_mdco_human --run-name llava_mdco_human \
  --override code/configs/vessl.yaml

Citation

@article{go2026mdco,
  title={MD-CO: A Knowledge Distillation Framework for Sophisticated Understanding
          and Reasoning in Chart Question Answering},
  author={Go, Young-Min and Jung, Hae Sun and Uprety, Sudan Prasad and Park, Keon Chul},
  journal={International Journal on Document Analysis and Recognition},
  year={2026}
}