mistral-v0.3-tablebench

mistral-v0.3-tablebench

Replication of TableBenchLLM, trained from Mistral-7B-Instruct-v0.3 on the corresponding instruction-tuning corpus.

Released as part of the EACL 2026 Findings paper "What Really Matters for Table LLMs? A Meta-Evaluation of Model and Data Effects" (Deng et al., 2026). The paper instruction-tunes three 7B foundation models (Mistral-v0.3, OLMo, Phi-3) on four existing training corpora (TableLlama, TableLLM, TableBench, TableGPT) to disentangle the contributions of base model versus training data, finding that base model choice plays a more dominant role than the training data itself.

• 📄 Paper: aclanthology.org/2026.findings-eacl.195
• 💻 Code & eval scripts: github.com/dnaihao/table-sft-eacl-2026
• 🤗 All replicated models: collection

Training

Full hyperparameter sweep, ablations, and per-benchmark numbers are reported in the paper.

Evaluation

Per-{model, benchmark} eval scripts and parsed metrics are available at github.com/dnaihao/table-sft-eacl-2026/tree/main/eval/mistral-v0.3-tablebench. Raw model outputs (generated_predictions.jsonl) are released as the dataset dnaihao/table-sft-eval-predictions.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("dnaihao/mistral-v0.3-tablebench")
model = AutoModelForCausalLM.from_pretrained(
    "dnaihao/mistral-v0.3-tablebench",
    torch_dtype="auto",
    device_map="auto",
)

License

This model inherits the license of its base model (mistralai/Mistral-7B-Instruct-v0.3: apache-2.0).

Citation

@inproceedings{deng-etal-2026-really,
    title = "What Really Matters for Table {LLM}s? A Meta-Evaluation of Model and Data Effects",
    author = "Deng, Naihao  and Zhang, Sheng  and Zhu, Henghui  and Chang, Shuaichen  and Zhang, Jiani  and Li, Alexander Hanbo  and Hang, Chung-Wei  and Kobayashi, Hideo  and Hu, Yiqun  and Ng, Patrick",
    booktitle = "Findings of the Association for Computational Linguistics: EACL 2026",
    year = "2026",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.findings-eacl.195/",
    doi = "10.18653/v1/2026.findings-eacl.195"
}