asure-12b-tr-attr-grpo

Asure-12B + GRPO — Turkish e-commerce attribute extraction

LoRA adapter for Trendyol/Trendyol-LLM-Asure-12B, trained with GRPO (RLVR) on a custom Turkish e-commerce attribute-extraction reward function.

Headline result

On an out-of-distribution test set of 578 Trendyol products drawn from 18 categories (swimwear, underwear, sportswear, scarves, hats, slippers, maternity, bedding, towels, bathrobes, curtains, belts, wallets, men's bags) that were never seen during GRPO training, this adapter brings the base 12B Asure model above the frontier-LLM bar with statistical significance:

Δ vs Claude on OOD: +0.026 macro F1, paired-bootstrap 95% CI [+0.011, +0.041], P(no improvement) = 0.001 — significant at 99.9% confidence.

The trained adapter wins on all 4 fields OOD (not just Desen). Confidence intervals are computed via paired bootstrap (N=2000 resamples, same indices across models so per-prompt difficulty cancels — see eval/bootstrap_ci.py).

In-distribution test (200 rows, held-out from training)

For the original held-out test split:

In-dist Δ vs Claude: +0.014 macro F1, 95% CI [−0.008, +0.036], P=0.11 — directionally above Claude, but the 200-row sample is too small to reach 95% significance. The OOD result above is the more rigorous test.

The biggest single contributor to the gain is the Desen (pattern) field: recall jumped from 0.41 → 0.71 because the policy fully internalized the catalog-default rule (Desen → "Düz" when no explicit pattern keyword is in the title — the modal value at ~65% in-dist and ~75% OOD).

Intended use

Given a Turkish product title (and optionally a brand), produce a strict JSON object with up to four catalog attributes:

• Renk (color) — canonical colors, e.g. "Beyaz", "Siyah", "Lacivert", ...
• Materyal (material) — catalog adjective form, e.g. "Pamuklu" (not "Pamuk"), "Denim" (not "Jean"), ...
• Kalıp (fit) — short English form without "Fit" suffix, e.g. "Slim" (not "Slim Fit"), "Regular", "Oversize", ...
• Desen (pattern) — defaults to "Düz" (plain) when no pattern keyword is in the title; otherwise "Çizgili", "Çiçekli", "Baskılı", ...

Use null (literal JSON null, not the string "null") when an attribute can't be inferred.

Example output

Input: Marka: Mavi\nÜrün adı: Erkek Slim Fit %100 Pamuk Basic Tişört Beyaz

Output:

{"Renk": "Beyaz", "Materyal": "Pamuklu", "Kalıp": "Slim", "Desen": "Düz"}

Usage

This is a PEFT/LoRA adapter — you must load it on top of the base model.

With vLLM (recommended for serving)

vllm serve Trendyol/Trendyol-LLM-Asure-12B \
    --enable-lora \
    --lora-modules grpo=KocKaankk/asure-12b-tr-attr-grpo \
    --max-lora-rank 16 \
    --served-model-name asure-12b-grpo \
    --dtype bfloat16 --max-model-len 4096

Then call the OpenAI-compatible endpoint with model="asure-12b-grpo". See the prompt template below for the exact system+user shape we trained on.

With transformers + PEFT (research / single-call)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Trendyol/Trendyol-LLM-Asure-12B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "KocKaankk/asure-12b-tr-attr-grpo")
tok = AutoTokenizer.from_pretrained("KocKaankk/asure-12b-tr-attr-grpo")

# See "Prompt template" below for SYSTEM and USER strings.
messages = [
    {"role": "system", "content": SYSTEM},
    *FEW_SHOTS,
    {"role": "user", "content": USER.format(title="Erkek Slim Fit %100 Pamuk Basic Tişört Beyaz", brand_line="Marka: Mavi\n")},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
out = model.generate(
    **tok(prompt, return_tensors="pt").to(model.device),
    max_new_tokens=128, do_sample=False,
)
print(tok.decode(out[0], skip_special_tokens=True))

Prompt template (v3, the one we trained on)

The model is sensitive to the prompt format. The system text sets per-field canonical vocabulary and the Desen → "Düz" default rule; the user template is a tight Marka:/Ürün adı:/JSON: shape with three Turkish few-shot examples between system and user.

The complete system prompt is ~280 tokens — too long to paste here. The exact text used in training is in eval/prompts.py (V3 constant). If serving in production, copy that file verbatim — there is no other supported prompt for this adapter.

Training

Held-out checkpoint sweep

We saved checkpoints every 50 steps and evaluated each on the held-out test set:

Distribution shape (best-of-8 at T=0.8 on the held-out test set)

Distribution narrowed by ~31% with no ceiling sacrifice — textbook successful RLVR signature.

Limitations

• Domain-specific. Trained on a 1,278-title slice of 36 Trendyol categories. Out-of-distribution categories (luxury, perishables, electronics with very different attribute schemas) have not been evaluated.
• 4 fields only. Trendyol pages have 8-12 attributes per product; this model is good only at the 4 most common ones.
• Capability narrowing. RL on a narrow reward function reduces the model's general-Turkish capabilities. This is fine for a specialized attribute-extraction model and a known cost of RLVR; do not use this adapter for free-form Turkish chat.
• Prompt-locked. The model only works well with the v3 prompt above. Other prompts (especially zero-shot) will significantly underperform.
• JSON output is regex-parsed downstream. The model produces well-formed JSON in 200/200 test cases, but a robust serving pipeline should still validate.
• Statistical robustness. The in-distribution +0.014 over Claude (on 200 rows) is not statistically significant (95% CI [−0.008, +0.036], P=0.11). The out-of-distribution +0.026 (on 578 rows) is significant (95% CI [+0.011, +0.041], P=0.001). When citing this model, prefer the OOD number — it's both larger and more rigorous. See notes/v0.6.2-ood-eval.md in the source repo for full methodology.

Reproducibility

Full code, dataset scrapers, training script, eval pipeline, and Terraform infra: github.com/KocKaankk/tr-rl (repo link will be live once published).

The training run that produced this adapter is documented end-to-end in notes/v0.6-grpo.md of that repo.

Citation

If you use this adapter in research:

@misc{karakas2026asuregrpo,
  author       = {Karakaş, Kaan},
  title        = {Asure-12B + GRPO: Turkish e-commerce attribute extraction},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/KocKaankk/asure-12b-tr-attr-grpo}},
}

License

CC BY-NC 4.0 — inherited from the base model (Trendyol/Trendyol-LLM-Asure-12B).