Qwen3.8-27B-Turkish-CPT-LoRA-Beta

Qwen3.8-27B-Turkish-CPT-LoRA-1M

A continued-pretrained Low-Rank Adaptation (LoRA) adapter for Qwen3.8-27B, trained on high-quality filtered Turkish web text from the Kumru corpus.

Model Summary

• Base Model: Qwen/Qwen3.8-27B (Text-only causal LM backbone)
• Adapter Type: PEFT LoRA (Rank 16, Alpha 16)
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Training Tokens: 1,000,000 tokens (512 token packed context)
• Training Precision: 4-bit NF4 Base + BF16 Compute / Gradients
• Parameters: 79,691,776 trainable parameters (304 MiB safetensors)

Evaluation & Benchmarks

Usage with Transformers & PEFT

import torch
from transformers import AutoTokenizer, BitsAndBytesConfig, Qwen3_5ForCausalLM
from peft import PeftModel

base_model_id = "Qwen/Qwen3.8-27B"
adapter_id = "UgurInanc12/Qwen3.8-27B-Turkish-CPT-LoRA-1M"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = Qwen3_5ForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=quantization_config,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

prompt = "Türkiye'de yapay zeka araştırmalarının geleceği hakkında bir paragraf yaz."
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation & Attribution

• Training Corpus: VNGRS-AI (vngrs-ai/vngrs-web-corpus)
• Base Model: Qwen Team (Qwen/Qwen3.8-27B)
• Author: Ugur Inanc