Qwen3.5-4B-TR-Finance-LoRA

Qwen3.5-4B-TR-Finance-LoRA

This repository contains the final LoRA / PEFT adapter produced by sequentially adapting Qwen/Qwen3.5-4B with:

1. Turkish + Finance DAPT / Continual Pre-training
2. Turkish + Finance Supervised Fine-Tuning

The same LoRA adapter was continued from the DAPT stage into the SFT stage. Therefore, this adapter encodes the cumulative DAPT + SFT updates.

For direct benchmark / leaderboard submission, the corresponding merged standalone model repository is recommended:

emretmrk/Qwen3.5-4B-TR-Finance

Türkçe Özet

Bu repo, Qwen/Qwen3.5-4B modeli üzerinde önce DAPT, ardından SFT ile eğitilmiş final LoRA adapter dosyalarını içerir.

Bu adapter:

• Türkçe dil adaptasyonu,
• Türkçe finans domain adaptasyonu,
• Türkçe instruction-following

amaçlarıyla eğitilmiştir.

Benchmark sistemleri doğrudan model adresi istediğinde merged model repo'sunun kullanılması önerilir. Bu adapter repo'su ise yeniden üretilebilirlik, düşük depolama maliyeti, PEFT inference ve ek fine-tuning için uygundur.

Adapter Details

Qwen3.5-4B is a multimodal upstream model. This adapter was trained using text-only datasets; visual capabilities were not explicitly optimized or evaluated.

Training

Stage 1 — DAPT / Continual Pre-training

Datasets

• ituperceptron/turkish-financial-sentiment-256k
• umarigan/turkish_corpus

Mixture

• 60% Turkish-finance text
• 40% general Turkish text

The mixture was constructed on a sample/row basis.

DAPT hyperparameters

Stage 2 — SFT

Datasets

• umarigan/turkiye_finance_qa
• Dbmaxwell/turkish-finance-instruction-dataset
• malhajar/alpaca-gpt4-tr

Data strategy

• Finance datasets used after filtering
• Finance training examples oversampled approximately 3×
• Approximately 12k filtered alpaca-gpt4-tr examples
• Evaluation split performed before finance oversampling
• Samples exceeding the configured sequence-length budget were filtered

SFT hyperparameters

Load the Adapter

import torch
from transformers import AutoProcessor, AutoModelForMultimodalLM
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen3.5-4B"
ADAPTER_ID = "emretmrk/Qwen3.5-4B-TR-Finance-LoRA"

processor = AutoProcessor.from_pretrained(BASE_MODEL)

base_model = AutoModelForMultimodalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype="auto",
    device_map="auto",
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER_ID,
    is_trainable=False,
)

model.eval()

Text-only generation example

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Merkez bankası politika faizini artırdığında ekonomide hangi kanallar etkilenebilir?"
            }
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
    )

answer = processor.decode(
    outputs[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)

print(answer)

Continue Fine-Tuning

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER_ID,
    is_trainable=True,
)

Merge the Adapter

merged_model = model.merge_and_unload()
merged_model.save_pretrained(
    "./Qwen3.5-4B-TR-Finance",
    safe_serialization=True,
)
processor.save_pretrained("./Qwen3.5-4B-TR-Finance")

Evaluation

Intended Use

• Reproducing the final Turkish-finance model
• PEFT inference
• Continued fine-tuning
• Research on DAPT → SFT pipelines
• Storage-efficient model distribution
• Comparing adapter and merged-model behavior

Limitations

• Requires the compatible Qwen/Qwen3.5-4B base model.
• Financial answers may be incorrect, incomplete, or outdated.
• The model is not a substitute for professional financial, investment, legal, or tax advice.
• Training data may contain inherited noise or bias.
• The adapter was trained with text-only data; multimodal behavior was not explicitly optimized or evaluated.

License and Dataset Terms

The upstream Qwen/Qwen3.5-4B model is released under the Apache License 2.0.

The adapter was trained using third-party datasets. Users should separately review the dataset cards and any applicable usage terms. Not all upstream dataset cards expose the same level of licensing detail, and this repository's license metadata does not override any applicable dataset terms.

Related Model

Merged standalone model:

emretmrk/Qwen3.5-4B-TR-Finance

Acknowledgements

Built with:

• Qwen3.5-4B
• Transformers
• TRL
• PEFT
• Weights & Biases