Qwen3.5-4B-TR-Finance-DAPT

Qwen3.5-4B-TR-Finance-DAPT

Qwen3.5-4B-TR-Finance-DAPT is a Turkish and Turkish-finance adapted version of Qwen/Qwen3.5-4B.

The model was adapted in two sequential stages:

1. Domain-Adaptive / Continual Pre-training (DAPT)
2. Supervised Fine-Tuning (SFT)

Training focused on improving Turkish language capability, Turkish financial-domain knowledge, and Turkish instruction following.

This repository contains the standalone merged model weights obtained after merging the final LoRA adapter into the upstream model. It is the recommended repository for direct inference and benchmark / leaderboard submission.

Note: Training in this project used text-only Turkish and finance data. The upstream Qwen3.5-4B model is multimodal, but visual capabilities were not explicitly trained or evaluated in this adaptation.

Türkçe Özet

Bu model, Qwen/Qwen3.5-4B modelinin Türkçe dil yeteneğini ve özellikle Türkçe finans alanındaki bilgisini geliştirmek amacıyla iki aşamada eğitilmiştir:

• DAPT / Continual Pre-training: Türkçe finans ve genel Türkçe ham metinleri
• SFT: Türkçe finans soru-cevap / instruction verileri ve genel Türkçe instruction verileri

Bu repo, final LoRA adapter'ın ana modele merge edilmiş standalone model sürümüdür. Benchmark ve doğrudan inference için bu repo önerilir.

Model Details

Training

Stage 1 — DAPT / Continual Pre-training

Datasets

• ituperceptron/turkish-financial-sentiment-256k
• umarigan/turkish_corpus

Mixture

• 60% Turkish-finance text
• 40% general Turkish text

The 60/40 mixture was constructed on a sample/row basis, not by exact token count.

DAPT hyperparameters

Training Stack

• Hugging Face Transformers
• TRL
• PEFT
• Accelerate
• Weights & Biases
• TensorBoard

Usage

The upstream Qwen3.5-4B architecture is multimodal. The exact loading API should follow the config.json stored in this repository.

import torch

from transformers import AutoProcessor, AutoModelForMultimodalLM

 

MODEL_ID = "emretmrk/Qwen3.5-4B-TR-Finance-DAPT"

 

processor = AutoProcessor.from_pretrained(MODEL_ID)

 

model = AutoModelForMultimodalLM.from_pretrained(

    MODEL_ID,

    torch_dtype="auto",

    device_map="auto",

)

 

messages = [

    {

        "role": "user",

        "content": [

            {

                "type": "text",

                "text": "Türkiye'de enflasyon ile politika faizi arasındaki ilişkiyi açıkla."

            }

        ],

    }

]

 

inputs = processor.apply_chat_template(

    messages,

    add_generation_prompt=True,

    tokenize=True,

    return_dict=True,

    return_tensors="pt",

).to(model.device)

 

with torch.no_grad():

    outputs = model.generate(

        **inputs,

        max_new_tokens=256,

        do_sample=False,

    )

 

answer = processor.decode(

    outputs[0][inputs["input_ids"].shape[-1]:],

    skip_special_tokens=True,

)

 

print(answer)

Benchmark / Leaderboard Usage

For benchmark systems that request a Hugging Face model identifier, use this merged model repository:

 

emretmrk/Qwen3.5-4B-TR-Finance-DAPT

 

The separate LoRA adapter repository is intended primarily for reproducibility, continued training, and PEFT-based loading.

Evaluation

Formal benchmark evaluation is performed separately from training.

Recommended comparison:

1. Qwen/Qwen3.5-4B
2. DAPT checkpoint

Intended Use

• Turkish text generation
• Turkish instruction following
• Turkish finance-domain question answering
• Turkish financial terminology and domain-language tasks
• Research on continual pre-training, DAPT and LoRA
• Benchmarking Turkish language capability

Limitations

• The model may generate incorrect or hallucinated information.
• Financial knowledge can become outdated.
• The model should not be treated as financial, investment, legal, or tax advice.
• Training data may contain noise, inaccuracies, or biases inherited from upstream datasets.
• The adaptation focused on text-only Turkish and finance data; multimodal capabilities were not explicitly optimized or evaluated.
• Benchmark gains should not be assumed until independently measured.

License and Dataset Terms

The upstream Qwen/Qwen3.5-4B model is released under the Apache License 2.0.

This model was trained using third-party datasets. Their individual dataset cards and usage terms should be reviewed separately. Not every upstream dataset card exposes the same level of licensing detail. This repository's license metadata does not override any applicable terms associated with training data.

Acknowledgements

This work builds on:

• Qwen3.5-4B
• Hugging Face Transformers
• TRL
• PEFT