ara-extract-v1

ara-extract-v1 — LoRA adapter for Qwen2.5-0.5B-Instruct

LoRA adapter trained on trilingual (Uzbek · Russian · English) government-document extraction and classification tasks. Designed to teach a small base model to emit structured JSON for contract fields, document type, language ID, named entities, and one-sentence summaries.

This adapter is the small-model half of the ARA fine-tuning pair. The QLoRA-on-7B counterpart lives at bilalsaidumarov/ara-extract-7b-qlora.

TL;DR

The headline number is JSON validity 0 → 83%: the adapter taught the base model the structured output format, which is the practical reason to fine-tune a 0.5B model for this workload at all. Exact match is held back by the held-out tail being heavy on language-ID examples — see Limitations below.

Intended use

Direct use — inside the ARA document-intelligence platform as the LLM backend for structured extraction over Uzbek / Russian / English government documents (contracts, invoices, memos, reports, letters). The 0.5B variant is the fast / cheap path for low-VRAM environments and CPU-only demos.

Prompt format (matches training):

### Instruction:
{instruction}

### Input:
{input}

### Response:

How to use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

BASE = "Qwen/Qwen2.5-0.5B-Instruct"
ADAPTER = "bilalsaidumarov/ara-extract-v1"

tokenizer = AutoTokenizer.from_pretrained(BASE)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
base = AutoModelForCausalLM.from_pretrained(BASE)
model = PeftModel.from_pretrained(base, ADAPTER)
model.eval()

prompt = (
    "### Instruction:\nExtract amount, signing date, and counterparty from the "
    "contract excerpt. Return a JSON object with keys amount, date, counterparty.\n\n"
    "### Input:\nAGREEMENT №ARA-2026-014 dated 14.03.2026 between Ministry of "
    "Economy and Finance and Acme Logistics LLC. Total contract value: 1,200,000 UZS.\n\n"
    "### Response:\n"
)
enc = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
    out = model.generate(**enc, max_new_tokens=128, do_sample=False,
                         pad_token_id=tokenizer.pad_token_id)
print(tokenizer.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))

Serving with vLLM

vllm serve Qwen/Qwen2.5-0.5B-Instruct \
    --enable-lora \
    --lora-modules ara-extract-v1=bilalsaidumarov/ara-extract-v1 \
    --max-loras 4

Training

Dataset

95 supervised examples across 15 task families, trilingual (uz · ru · en):

Format — one JSON object per line: {"instruction": ..., "input": ..., "output": ...}.

Evaluation

Held-out 20% (19 examples), greedy decoding, max_new_tokens=128.

Same eval harness on the 7B QLoRA sibling:

Limitations

• Small dataset (95 examples). Enough to demonstrate the technique and shift the JSON-validity rate decisively, not enough for production quality across all 15 task families. Language ID and translation in particular need more examples.
• Exact-match is the weakest metric because the held-out tail is heavy on language-ID rows, which the 0.5B base struggles with after only 3 epochs.
• Multilingual coverage is uneven. Uzbek has the least training data of the three; outputs in Uzbek are noisier than in Russian or English.
• No hyperparameter sweep. r=16, alpha=32, lr=2e-4, 3 epochs are defaults from the LoRA paper — not tuned.
• Inherits base-model risks (bias, hallucination) — use temperature ≤ 0.2 and validate JSON before downstream use.

License

Apache-2.0 (matches base model Qwen/Qwen2.5-0.5B-Instruct).

Citation

If this adapter is useful in your work, cite the ARA project:

@misc{ara2026,
  title  = {ARA — AI Resource Assistant (document-intelligence platform)},
  author = {Bilol Saidumarov},
  year   = {2026},
  url    = {https://github.com/sb-bilal-dev-2/ara}
}

Framework versions

• PEFT 0.19.1
• Transformers ≥ 4.46
• PyTorch ≥ 2.5