qwen3-4b-p300-text-correction-lora

Qwen3-4B P300 Text Correction LoRA

Two PEFT LoRA checkpoints fine-tuned from Qwen/Qwen3-4B-Instruct-2507 for restoring short Russian phrases affected by synthetic character-level substitution, adjacent duplication, and deletion noise.

The artificial corruption imitates character errors that may occur in decoded P300-speller text.

Checkpoints

Checkpoint 780 had lower clean and mixed validation micro CER and fewer unnecessary clean modifications. Checkpoint 1560 is included for reproducibility, not because it performed better.

Quick start

The tokenizer and chat template must be loaded from the pinned base model. To use checkpoint 1560, set adapter_kwargs to the commented alternative below.

from pathlib import Path

import torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-4B-Instruct-2507"
base_revision = "cdbee75f17c01a7cc42f958dc650907174af0554"
adapter_id = "redche7/qwen3-4b-p300-text-correction-lora"

tokenizer = AutoTokenizer.from_pretrained(base_id, revision=base_revision)
base_model = AutoModelForCausalLM.from_pretrained(
    base_id,
    revision=base_revision,
    dtype=torch.bfloat16,
    device_map="auto",
)

adapter_kwargs = {}
# adapter_kwargs = {"subfolder": "checkpoint-1560"}
model = PeftModel.from_pretrained(base_model, adapter_id, **adapter_kwargs)
model.eval()

prompt_path = hf_hub_download(adapter_id, "prompt.txt")
system_prompt = Path(prompt_path).read_text(encoding="utf-8")
messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "ПРИВЕТ, КАК ДИЛА?"},
]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

with torch.inference_mode():
    generated = model.generate(**inputs, do_sample=False, max_new_tokens=160)

prompt_length = inputs["input_ids"].shape[1]
correction = tokenizer.decode(
    generated[0, prompt_length:],
    skip_special_tokens=True,
).strip()
print(correction)

Training

The adapters were trained with QLoRA on 22,698 deterministic prompt-completion records. Source labels and audio were not used as targets, and no training rows are distributed in this model repository.

Key settings were NF4 QLoRA with BF16 compute, LoRA r=16, alpha=32, dropout 0.05, maximum sequence length 384, effective batch size 16, and learning rate 1e-4. The full machine-independent summary is in training_config.yaml.

Evaluation

Both checkpoints were compared on the same 300 clean and 300 synthetically corrupted validation requests:

The selected checkpoint 780 was then compared with the frozen BF16 base model:

The external set contains 1,600 human Opusparcus-derived phrases and 400 synthetic phrases. The clean control uses the original phrases, while the mixed condition applies the same artificial character-corruption protocol.

On the external clean control, unnecessary modifications decreased from 24.35% to 5.85%. These results apply only to the frozen artificial-corruption protocol, prompt, deterministic decoding, Russian text sources, and evaluated phrase lengths.

License

The training sources use different licenses, so the repository is marked license: other rather than Apache-2.0. No commercial-use clearance or claim of cross-license compatibility is provided. See LICENSE.md for the applicable CC BY-NC-SA 4.0 and OpenRAIL terms.

Exact adapter hashes, sizes, checkpoint mapping, and verification status are in release_manifest.json.