mistral-7b-mio-arc-fp16

AIRE-ARC-Mistral-7B-SFT-MIO-FP16

A standalone FP16 fine-tune of mistralai/Mistral-7B-v0.1 for ARC-Challenge and related science multiple-choice reasoning. The model was trained in two stages:

1. completion-only QLoRA SFT on 27,294 public science-QA training examples;
2. MIO preference fine-tuning on 47,087 train-only pairs, where the rejected answers were the incorrect choices that the SFT model found most plausible.

The final model reached 76.54% acc_norm and 74.74% raw accuracy on the 1,172-example ARC-Challenge test split with 25-shot evaluation. Model selection was completed on the validation split before the test split was opened.

Important: this is a domain-specialized completion model, not a chat or general instruction model. It improves science and commonsense multiple-choice reasoning, but capability-retention audits found regressions on MMLU Humanities, MMLU Social Sciences, and WikiText perplexity.

Model summary

The repository contains the tokenizer and merged model weights. PEFT and bitsandbytes are not required for inference. merge_summary.json records the merge provenance used for the submitted checkpoint.

Intended use

Suitable uses include:

• evaluating science and commonsense multiple-choice reasoning;
• studying model-scored hard-negative selection;
• reproducing the reported ARC-Challenge evaluation;
• research on preference optimization for short-answer QA.

The model is not intended for safety-critical use, factual deployment without verification, general chat, or instruction following. No additional safety alignment was performed.

Prompt format

Training and evaluation use answer-text completions rather than answer-position labels:

Question: {question}
Answer:

The expected completion is the answer text, including its leading space. No chat template is used.

Quick start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "psymon/mistral-7b-mio-arc-fp16"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
if tokenizer.pad_token_id is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    dtype=torch.float16,
    device_map="auto",
)
model.eval()

prompt = "Question: What is a worldwide increase in temperature called?\nAnswer:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=24,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

completion = tokenizer.decode(
    output[0, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)
print(completion)

For multiple-choice evaluation, compare the conditional log-likelihood of each answer text. Do not convert the choices to A, B, C, and D unless the evaluation protocol is changed accordingly.

Training data

Only public training splits were used for optimization. ARC-Challenge validation was used for deduplication, failure analysis, and model selection. ARC-Challenge test was not used during training or model selection.

SFT corpus

The SFT prompt contains the question only; the target is the correct answer text. Choices were retained for data normalization and duplicate checks but were not included in the SFT prompt.

Preference corpus

The SFT model scored every answer choice for 15,698 train-only prompts from ARC-Challenge, ARC-Easy, OpenBookQA, and QASC. Each score was the answer-token log-likelihood sum divided by the answer's character length. For each question, up to three highest-scoring incorrect choices were paired with the correct answer:

prompt   = "Question: {question}\nAnswer:"
chosen   = " {correct answer text}"
rejected = " {model-selected incorrect answer text}"

This produced 47,087 preference pairs. Seven three-choice questions contributed two pairs each; the remaining questions contributed three pairs each. In 3,081 prompts, at least one incorrect choice received a higher selection score than the correct answer.

Training procedure

Stage 1: completion-only SFT

Loss was computed only on the answer completion; prompt tokens were excluded from the labels.

Stage 2: MIO preference fine-tuning

MIO (Mutual Information Optimization) contrasts a chosen and rejected completion against a frozen reference model. This run used the merged SFT checkpoint as the reference and initialized a new LoRA policy from the same weights. Completion log-probabilities were averaged over response tokens, including EOS.

The initial policy/reference log-ratios were exactly zero and the initial loss was 1.386294, matching ln(4). Reference inference was executed first under no_grad; its activations were released before the policy forward pass to fit training on a 40GB A100 without changing the objective.

The result should be interpreted as the effect of the complete procedure: SFT reference, model-scored hard negatives, mean completion log-probabilities, and MIO. This experiment did not isolate the causal contribution of each component.

Evaluation

Evaluation used EleutherAI's lm-evaluation-harness protocol with 25 few-shot examples and answer-choice likelihood normalization.

ARC-Challenge validation

From SFT to MIO, 51 normalized predictions changed from incorrect to correct and 9 changed from correct to incorrect, for a net gain of 42 correct answers on 299 validation examples.

One-time held-out test

The candidate model was locked before this evaluation. The test split was evaluated once, and no post-test training, hyperparameter tuning, or model reselection was performed.

Post-test overlap audit

The test split was compared with the training ledger only after final evaluation:

• exact Test/SFT ID overlap: 0;
• exact question plus complete choice-set overlap: 0;
• exact normalized question-stem overlap: 9 test rows;
• character similarity of at least 0.90: 18 test rows.

Removing the 18 near-overlap rows changed test acc_norm from 76.5358% to 76.4298% (-0.1060 percentage points). This is a sensitivity analysis, not proof that semantic contamination is absent; paraphrases and shared underlying facts may remain undetected.

Capability-retention audit

After model selection and the one-time ARC test, the standalone Hub checkpoint was downloaded again and compared with the base model. These results were not used for training or selection.

The MMLU regressions partly depend on answer-position labels: accuracy fell much more for answers at positions B and D than for A and C. This suggests that output-label calibration changed in addition to any knowledge loss. WikiText does not use answer labels and also regressed, so format calibration alone cannot explain all of the degradation.

Reproducing the ARC evaluation

pip install "lm-eval==0.4.12"

python -m lm_eval \
  --model hf \
  --model_args "pretrained=psymon/mistral-7b-mio-arc-fp16,dtype=float16" \
  --tasks arc_challenge \
  --num_fewshot 25 \
  --batch_size 8

The reported submission result used the official ARC-Challenge parquet files in an offline custom task, with hashes checked before evaluation. The submitted Colab notebook contains the exact data validation, model-selection lock, and one-time test procedure.

Limitations

• Domain specialization: optimized for short English science-QA completions.
• Not a chat model: no instruction-following or conversational alignment was added.
• Single training seed: training variance was not estimated across seeds.
• Public-corpus overlap risk: post-test string audits cannot exclude semantic overlap or paraphrases.
• Capability trade-offs: MMLU and WikiText regressions show that the model did not preserve all base capabilities.
• Incomplete coverage: code, mathematics, safety, long-form generation, summarization, and multilingual behavior were not evaluated.
• No safety guarantee: outputs may be incorrect, biased, or unsafe.

License and data terms

The parent model is released under Apache-2.0. The training sources have separate terms:

Because the SFT stage includes SciQ and the OpenBookQA card does not declare complete licensing information, this model card intentionally uses license: other rather than presenting the repository as unconditionally Apache-2.0. The repository is shared for research and coding-test evaluation. Users are responsible for reviewing the parent-model license and each source dataset's terms before redistribution or downstream use, especially commercial use. This section is informational and not legal advice.

References

• A. Q. Jiang et al., Mistral 7B, 2023.
• T. Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, 2023.
• X. Lv et al., The Hidden Link Between RLHF and Contrastive Learning, ICLR 2026.
• P. Clark et al., Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, 2018.