Qwen3-4B-TL-JSON

Qwen3-4B-TL-JSON

Let me start by saying this is a test, not an attempt to make the best LLM ever at 4B, as much as i'd like to.

This is V0.2 of finetuning Qwen3-4B with line by line translations in json format with variable line amounts per example. With the goal of improving its ability to handle this format.

Finetuned for 5168 examples, plan to expand to 30000 examples and a larger rank, while reducing the learning rate, and I also plan on using examples of the shisa v2 dataset to finetune its japanese understanding at some point,

Example of ideal input

Translate to English
{
    "Line1": "欲しいものはあるかい？ 良いアイテムがあれば高値で買い取るよ.",
    "Line2": "行動ポイントを消費して討伐依頼を出しますか？",
    "Line3": "悪魔のしっぽを手に入れた.",
    "Line4": "行動ポイントを消費して討伐依頼を出しますか？",
    "Line5": "かしこさのたね、ふしぎなきのみを手に入れた.",
    "Line6": "行動ポイントを消費して討伐依頼を出しますか？",
    "Line7": "うさぎのしっぽを手に入れた. 魔獣の皮を手に入れた.",
    "Line8": "行動ポイントを消費して討伐依頼を出しますか？",
    "Line9": "竜のウロコを手に入れた."
}

Predicted Output

thinking temp 0.7
{
    "Line1": "Is there anything you're looking for? If you have good items, I'll buy them at a high price.",
    "Line2": "Do you want to use action points to submit a capture request?",
    "Line3": "Acquired the tail of a demon.",
    "Line4": "Do you want to use action points to submit a capture request?",
    "Line5": "Acquired the seed of obedience and the mysterious pill.",
    "Line6": "Do you want to use action points to submit a capture request?",
    "Line7": "Gained the tail of a rabbit and the skin of a monster.",
    "Line8": "Do you want to use action points to submit a capture request?",
    "Line9": "Gained the scales of a dragon."
}

no thinking temp 0.3
{
"Line1": "Do you want something? If there are good items, I'll buy them at a high price.",
"Line2": "Consume action points to issue a capture request?",
"Line3": "Acquired the tail of a demon.",
"Line4": "Consume action points to issue a capture request?",
"Line5": "Acquired the mysterious ki no mi (magic potion).",
"Line6": "Consume action points to issue a capture request?",
"Line7": "Acquired the rabbit's tail. Acquired the skin of a monster.",
"Line8": "Consume action points to issue a capture request?",
"Line9": "Acquired the scales of a dragon."
}

In personal testing with this particular build.

I recommend using temperature 0.3, and then the normal sampler params for this model

When benchmarking how 'accurate' it is to maintaining this format, i got greater than 95% success when json parsing 512 samples generated from this model.

Trained using Unsloth on Windows!

Rank:512 Alpha:512 BS:4 EBS:16-32

stage1 LR:1e-7 WS:100 MS:600

stage2 LR:8e-8 WS:450 MS:500

stage3 LR:1e-7 WS:50 MS:192