Downloads · 30 days
24
7% of all-time downloads
allura-forge/qwen3.5-9b-seed2.0-claude-dist-v2-qlora-ckpts
qwen3.5-9b-seed2.0-claude-dist-v2-qlora-ckpts is a text generation model from allura-forge. Use it when you need the model to write or continue text. It is set up for peft.
This model is a fine-tuned version of Qwen/Qwen3.5-9B.
Downloads · 30 days
24
7% of all-time downloads
All-time downloads
348
Public
Repo size
2.5 GB
Likes
0
Public
Click a slice to open those files.
.safetensors3.3 GB · 98%
From the Hugging Face model README
This model is a fine-tuned version of Qwen/Qwen3.5-9B.
W&B run: https://wandb.ai/cooawoo-personal/huggingface/runs/jelbo6gf
| Parameter | Value |
|---|---|
| Learning rate | 0.0002 |
| LR scheduler | SchedulerType.CONSTANT |
| Per-device batch size | 2 |
| Gradient accumulation | 4 |
| Effective batch size | 8 |
| Epochs | 2 |
| Max sequence length | 4096 |
| Optimizer | OptimizerNames.ADAMW_TORCH |
| Weight decay | 0.01 |
| Warmup ratio | 0.05 |
| Max gradient norm | 1.0 |
| Precision | bf16 |
| Loss type | nll |
| Chunked cross-entropy | yes |
| Parameter | Value |
|---|---|
| Rank (r) | 64 |
| Alpha | 512 |
| Target modules | attn.proj, down_proj, gate_proj, in_proj_a, in_proj_b, in_proj_qkv, in_proj_z, k_proj, linear_fc1, linear_fc2, o_proj, out_proj, q_proj, qkv, up_proj, v_proj |
| Quantization | 4-bit (nf4) |
| Dataset | Samples | Total tokens | Trainable tokens |
|---|---|---|---|
| allura-forge/doubao-seed2.0-claude-distill-v1-qwen3.5-format | 3,644 | 7,187,856 | 6,625,087 |
model_name_or_path: Qwen/Qwen3.5-9B
bf16: true
gradient_checkpointing: true
gradient_checkpointing_kwargs:
use_reentrant: false
use_liger: true
use_cce: true
max_length: 4096
learning_rate: 0.0002
warmup_ratio: 0.05
weight_decay: 0.01
lr_scheduler_type: constant
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
optim: adamw_torch
max_grad_norm: 1.0
use_peft: true
load_in_4bit: true
lora_r: 64
lora_alpha: 512
lora_dropout: 0.0
use_rslora: false
logging_steps: 1
disable_tqdm: false
save_strategy: steps
save_steps: 500
save_total_limit: null
report_to: wandb
output_dir: output-fizz-v2
data_config: data.yaml
prepared_dataset: prepared
num_train_epochs: 2
saves_per_epoch: 1
run_name: qwen35-9b-qlora-fizz
</details>
<details>
<summary>Data config</summary>
datasets:
- path: allura-forge/doubao-seed2.0-claude-distill-v1-qwen3.5-format
type: conversational
truncation_strategy: drop
shuffle_datasets: true
shuffle_combined: true
shuffle_seed: 42
eval_split: 0.0
split_seed: 42
assistant_only_loss: true
</details>