Mistral_academic_style_tune

Mistral-7B Fine-Tuned for Academic Style (QLoRA)

This is a parameter-efficient fine-tuning of mistralai/Mistral-7B-v0.1 using QLoRA on 500K academic abstracts. It was built for the ECS 271 final project at UC Davis.

Intended Use

The model is designed to generate formal academic paragraphs given a paper title, useful for research drafts, educational AI tools, and academic-style assistants.

Training Details

• Base model: mistralai/Mistral-7B-v0.1
• Method: QLoRA (low-rank adapter)
• Prompt format: "Write an academic paragraph given the title: ..."
• Dataset: 500K arXiv abstracts
• Epochs: 1
• GPU: RTX 5070 Ti (~60 hours)

Limitations

• Generic or templated outputs
• No citation support
• Frequent hallucinations

Example

Prompt:
"Write an academic paragraph given the title: LoRA for In-Context Learning"

Output:
"We present LoRA (Loosely Regularized Adapters), a novel approach to fine-tune large language models for in-context learning tasks. Unlike traditional methods, LoRA updates only a small number of trainable parameters, achieving comparable performance while reducing training costs."

More Info

Main Project: Github