Llama-ML-Expert-DPO-1b

Llama-ML-Expert-DPO-1b

The final, preference-aligned model in the LLM-ArXiv-Domain-Expert pipeline. Built on top of danivpv/Llama-ML-Expert-Instruct-1b via Direct Preference Optimization (DPO), aligning outputs toward the authoritative, academic writing style of the source ArXiv ML papers.

Model Details

Training Data

danivpv/ml-arxiv-dpo — instruction/chosen/rejected triples, where chosen is a verbatim excerpt from the source ArXiv paper and rejected is a factually-grounded but LLM-paraphrased answer. The model is optimized to prefer authentic academic phrasing over generic LLM-speak — see the dataset card for the full rationale and its tradeoffs.

How to Use

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="danivpv/Llama-ML-Expert-DPO-1b",
    max_seq_length=2048,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

prompt = """### Instruction:
{your ML question here}

### Response:
"""
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

<!-- TODO: if you ran the evaluation pipeline (run-evaluation-pipeline), add a Results section comparing this DPO model against the SFT-only Instruct model — that comparison is the single most interesting artifact this repo can show a technical reviewer: does preference alignment measurably shift style/quality? -->

Relationship to Instruct-1b

This model is a second-stage fine-tune, not a replacement — it trades a small amount of generative flexibility for closer adherence to source-paper phrasing. If you need more open-ended generation, Llama-ML-Expert-Instruct-1b (SFT-only) may be the better starting point.

Limitations

• Same 1B-scale limitations as the base Instruct model: strong for domain-specialized short-form Q&A, not a general-purpose reasoning model.
• DPO training used 1 epoch at a small learning rate — a conservative alignment pass, not an exhaustive preference-optimization run.

Related

• SFT model: danivpv/Llama-ML-Expert-Instruct-1b
• Preference dataset: danivpv/ml-arxiv-dpo
• SFT dataset: danivpv/ml-arxiv-instruct
• Pipeline code: github.com/danivpv/LLM-ArXiv-Domain-Expert