Qwen3-4B-Stage1

Qwen3-4B — Stage 1 (General Foundation)

A 4B parameter model fine-tuned on ~308k mixed samples covering general chat, instruction following, math, coding, and factual QA. This is Stage 1 of a multi-stage training pipeline built on top of Qwen3-4B.

Model Details

• Base Model: unsloth/Qwen3-4B
• Parameters: 4B
• Architecture: Qwen3 (dense, pure text)
• Training: LoRA fine-tune → merged
• License: Apache 2.0
• Language: English (multilingual base)

Files

Training Details

Dataset Mix (~308k samples)

Total: ~308k samples, 2 epochs

Hyperparameters

Hardware

Trained on AMD Instinct MI300X (192GB VRAM), ROCm 6.2.4, Unsloth 2026.3.3, PyTorch 2.7.1+rocm6.2.4. Runtime: ~16.3 hours.

Usage

Ollama (GGUF)

ollama run hf.co/lqfdjbf32n/Qwen3-4B-Stage1:Q4_K_M

llama.cpp

llama-cli -m qwen4b-s1-Q4_K_M.gguf \
    -p "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nYour question here<|im_end|>\n<|im_start|>assistant\n" \
    -n 512

Python (LoRA adapter)

from unsloth import FastLanguageModel
from peft import PeftModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    "unsloth/Qwen3-4B",
    max_seq_length=4096,
    dtype=torch.bfloat16,
    load_in_4bit=False,
)
model = PeftModel.from_pretrained(model, "lqfdjbf32n/Qwen3-4B-Stage1")
model = model.merge_and_unload()

Limitations

• Stage 1 only — general foundation, not reasoning-specialized
• English primary (multilingual via base model)
• Not suitable for production without validation
• Complex multi-step reasoning may still fail

Part of a Series