Downloads · 30 days
8
5% of all-time downloads
Rakushaking/Qwen4b-SFT-d9-merged-after-dpo-toml-xml-yaml-dpo
Qwen4b-SFT-d9-merged-after-dpo-toml-xml-yaml-dpo is a text generation model from Rakushaking. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This model is a fine-tuned version of Rakushaking/Qwen4b-SFT-d9-merged-after-dpo-d2 using Direct Preference Optimization (DPO) via the Unsloth library.
Downloads · 30 days
8
5% of all-time downloads
All-time downloads
168
Public
Parameters
4B
8.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors8 GB · 100%
From the Hugging Face model README
This model is a fine-tuned version of Rakushaking/Qwen4b-SFT-d9-merged-after-dpo-d2 using Direct Preference Optimization (DPO) via the Unsloth library.
This repository contains the full-merged 16-bit weights. No adapter loading is required.
This model has been optimized using DPO to align its responses with preferred outputs, focusing on:
[section] / [[array]] style over inline table { key = val } style (500 pairs)Rejected outputs were programmatically generated (not LLM-generated) to ensure consistent quality:
toml.loads() then re-serialized as inline tables& left unescaped / wrapped in markdown codeblocks| Format | Baseline (pre-train) | After SFT | After DPO Round 2 (this model) |
|---|---|---|---|
| JSON | 100% | 100% | 100% |
| CSV | 100% | 100% | 100% |
| YAML | 97% | 94% | 94% |
| XML | 80% | 95% | 90% |
| TOML | 72% | 48% | 64% |
| Total | 92.0% | 89.3% | 91.3% |
While the overall parse success rate is comparable (BL 92.0% vs Final 91.3%), this model achieves higher evaluation scores than the baseline due to the following qualitative improvements:
1. XML: 80% → 90% (+10pt)
SFT provided extensive training on correct XML structures (tag nesting, repeated elements for arrays). DPO further reinforced well-formed XML by explicitly rejecting outputs with mismatched tags, unescaped & characters, and codeblock wrapping.
2. Output cleanliness
| Issue | Baseline | This model |
|---|---|---|
Markdown codeblock wrapping (```json ... ```) | 8 tasks | 4 tasks |
| Explanation text mixed in ("Here's the converted...") | 3 tasks | 0 tasks |
| Language mixing (Chinese characters in English output) | 1 task | 0 tasks |
| Factual errors (e.g. Rosetta Stone wrong era/discoverer) | Multiple | 0 (switched to fictional data) |
3. Chain-of-Thought reasoning quality The baseline produced raw output without reasoning. This model generates structured CoT before each output:
This improves the structural accuracy of the generated data, which is reflected in evaluation scores even when parse rates are similar.
4. Factual error avoidance The baseline attempted to generate real-world data (e.g. Rosetta Stone) and produced numerous factual errors (wrong era, wrong discoverer, wrong dimensions). This model generates fictional/synthetic data instead, avoiding factual inaccuracies entirely.
| Format | Baseline | This model | Reason |
|---|---|---|---|
| YAML | 97% | 94% (-3pt) | Minor indentation issues in complex nested structures |
| TOML | 72% | 64% (-8pt) | BL used inline tables (shallow structures happened to parse); this model uses [section] style which is more correct but fails on complex nesting |
Note: The TOML decrease in parse rate does not reflect a decrease in output quality. The baseline's 72% relied on inline tables { key = val } which happened to parse for shallow structures but would fail for deeply nested data. This model intentionally uses [section] / [[array]] style which is the proper TOML convention.
Since this is a merged model, you can use it directly with transformers.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "Rakushaking/Qwen4b-SFT-d9-merged-after-dpo-toml-xml-yaml-dpo"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.float16,
device_map="auto"
)
prompt = "Your question here"
inputs = tokenizer.apply_chat_template(
[{"role": "user", "content": prompt}],
tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to("cuda")
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0]))