IELTSWritingTask2-Qwen2.5-7B-QLoRA

IELTS Writing Task 2 — Qwen2.5-7B QLoRA

A QLoRA adapter for Qwen/Qwen2.5-7B-Instruct, fine-tuned to generate complete IELTS Writing Task 2 essays from essay prompts.

The model is designed for essay generation, not IELTS scoring.

Input: an IELTS Writing Task 2 question
Output: one complete academic essay, typically around 280–330 words

This repository contains the LoRA adapter, not a fully merged 7B model. The Qwen2.5-7B-Instruct base model is loaded separately and the adapter is attached with PEFT.

---

Quick Start

1. Install dependencies

pip install -U transformers peft accelerate bitsandbytes sentencepiece

2. Load the model

import torch

from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
)
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen2.5-7B-Instruct"
ADAPTER_REPO = "vltruong01/IELTSWritingTask2-Qwen2.5-7B-QLoRA"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.float16,
)

tokenizer = AutoTokenizer.from_pretrained(
    ADAPTER_REPO,
    trust_remote_code=True,
)

if tokenizer.pad_token_id is None:
    tokenizer.pad_token = tokenizer.eos_token

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.float16,
    trust_remote_code=True,
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER_REPO,
)

model.eval()

3. Generate an essay

SYSTEM_PROMPT = (
    "You are an expert IELTS Writing Task 2 writer aiming for Band 8 or higher. "
    "Write a complete academic essay that answers every part of the task and maintains a clear position. "
    "Develop each main idea with specific explanation and a relevant example where appropriate. "
    "Prioritize grammatical accuracy, logical cohesion, and precise natural vocabulary over rare or forced words. "
    "Avoid headings, bullet points, repetition, and memorized templates."
)

question = '''
Some people believe that artificial intelligence will improve people's lives,
while others think it will create serious problems for society.

Discuss both views and give your own opinion.
'''

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {
        "role": "user",
        "content": (
            "IELTS Writing Task 2\n\n"
            f"{question.strip()}\n\n"
            "Write only the final essay, approximately 280–330 words."
        ),
    },
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(
    prompt,
    return_tensors="pt",
    add_special_tokens=False,
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=520,
        do_sample=False,
        repetition_penalty=1.03,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

new_tokens = outputs[0, inputs["input_ids"].shape[1]:]

essay = tokenizer.decode(
    new_tokens,
    skip_special_tokens=True,
).strip()

print(essay)
print("\nWord count:", len(essay.split()))

For a ready-to-run notebook, see:

IELTS_Qwen25_7B_Inference.ipynb

---

Model Details

The LoRA adapter targets:

q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

---

Training Data

The training notebook downloads the source dataset directly from Hugging Face:

chillies/ielts-writing-task2-essays

No pre-generated train, validation, or test dataset files are required.

The preprocessing pipeline runs in memory:

1. normalize questions and essays,
2. convert band-score fields to numeric values,
3. apply the quality filter,
4. remove exact duplicate essays,
5. group identical and near-duplicate questions,
6. create deterministic train / validation / test splits by question group.

Quality filter

Overall band >= 8.0
Task Achievement >= 7.0
Coherence & Cohesion >= 7.0
Lexical Resource >= 7.0
Grammatical Range & Accuracy >= 7.0
Essay length: 250–420 words

After filtering and exact essay de-duplication:

Average component scores:

---

Leakage-Aware Data Split

A simple random row split would be inappropriate because several essays can correspond to the same or nearly identical IELTS question.

To reduce question leakage, near-duplicate prompts are grouped before splitting.

The split is performed by question_group, not by individual essay row.

Because entire question groups stay together, the final essay-row proportions are not exactly 80/10/10.

The split is deterministic with a fixed random seed. The split files themselves are not stored in this repository.

---

Training Configuration

Base model: Qwen/Qwen2.5-7B-Instruct

QLoRA:
- 4-bit NF4
- double quantization
- FP16 compute

LoRA:
- rank: 32
- alpha: 64
- dropout: 0.05
- target modules: attention + MLP projections

Training:
- epochs: 3
- per-device train batch size: 1
- per-device validation batch size: 1
- gradient accumulation: 8
- effective batch size: 8
- learning rate: 1e-4
- scheduler: cosine
- warmup ratio: 0.08
- optimizer: paged_adamw_8bit
- gradient checkpointing: enabled
- maximum sequence length: 768

Only assistant-response tokens contribute to the language-model loss. System and user prompt tokens are masked.

The best checkpoint is selected using validation loss:

load_best_model_at_end=True
metric_for_best_model="eval_loss"
greater_is_better=False

---

Validation Results

The best observed validation loss was reached around step 80.

The best validation checkpoint is restored before the final LoRA adapter is saved.

Validation loss is a language-model objective and should not be interpreted as an IELTS band score.

---

Held-Out Evaluation

The held-out test split contains:

36 essays
14 unseen question groups
11 topics

For qualitative generation analysis, representative unseen prompts are selected from the held-out question groups rather than generating once for every reference essay row.

Existing generated examples are available under:

artifacts/test_results/

Recommended human evaluation dimensions:

• Task Response
• Coherence & Cohesion
• Lexical Resource
• Grammatical Range & Accuracy

Generated essays should not be assumed to have a specific IELTS band score solely because the training examples were high-scoring essays.

---

Reproduce the Training

The full workflow is provided in:

IELTSWritingtask2_Qwen25_7B.ipynb

Pipeline:

Hugging Face source dataset
        ↓
quality filtering
        ↓
exact essay de-duplication
        ↓
near-duplicate question grouping
        ↓
deterministic group-aware split
        ↓
QLoRA fine-tuning
        ↓
best-checkpoint selection
        ↓
final LoRA adapter

The dataset is loaded directly from Hugging Face during the run. Train / validation / test JSONL files are not required or stored in this repository.

Experiment settings are recorded in:

artifacts/experiment_config.json

---

Repository Structure

IELTSWritingTask2-Qwen2.5-7B-QLoRA/
├── README.md
├── IELTSWritingtask2_Qwen25_7B.ipynb
├── IELTS_Qwen25_7B_Inference.ipynb
├── adapter_config.json
├── adapter_model.safetensors
├── added_tokens.json
├── chat_template.jinja
├── merges.txt
├── special_tokens_map.json
├── tokenizer.json
├── tokenizer_config.json
├── vocab.json
└── artifacts/
    ├── experiment_config.json
    └── test_results/
        ├── qwen25_ielts_quality_v2_test_outputs.csv
        └── qwen25_ielts_quality_v2_test_outputs.jsonl

Which file should I use?

---

Expected Input

Use a normal IELTS Writing Task 2 question, for example:

Some people think that governments should invest more money in public transport,
while others believe that building more roads is more important.

Discuss both views and give your own opinion.

You do not need to manually add Qwen special tokens. Use tokenizer.apply_chat_template() as shown above.

---

Expected Output

The intended output is:

• one complete essay,
• no headings,
• no bullet points,
• a clear position,
• developed body paragraphs,
• approximately 280–330 words,
• formal and natural academic English.

---

Intended Use

Suitable uses include:

• generating IELTS Writing Task 2 practice essays,
• studying long-form generation,
• experimenting with QLoRA and PEFT,
• comparing base-model and fine-tuned writing behavior,
• educational and research experiments.

---

Limitations

• The model is trained on a relatively small filtered dataset.
• Dataset band labels may not perfectly match official examiner-assessed scores.
• Generated essays can contain grammatical, factual, or logical errors.
• Some generations may reuse common IELTS-style structures.
• Quality may vary by question type and topic.
• A fluent essay is not necessarily a Band 8+ essay.
• Validation loss is not directly equivalent to an IELTS band score.
• The model is not designed to score or correct user essays.

For serious IELTS assessment, human evaluation using the official Writing Task 2 criteria is recommended.

---

Disclaimer

This is an experimental educational and research model.

It is not affiliated with IELTS, Cambridge University Press & Assessment, the British Council, or IDP Education.

Generated outputs should not be interpreted as official IELTS assessment or guaranteed band-score performance.