Downloads · 30 days
10
8% of all-time downloads
cjkasbdkjnlakb/qwen3-4b-agent-0926-quancan
qwen3-4b-agent-0926-quancan is a text generation model from cjkasbdkjnlakb. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
10
8% of all-time downloads
All-time downloads
122
Public
Parameters
503M
17.7 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors17.6 GB · 100%
From the Hugging Face model README
axolotl version: 0.12.2
# Automatically upload checkpoint and final model to HF
# hub_model_id: username/custom_model_name
# 是否以 8-bit 精度加载模型
load_in_8bit: false
# 是否以 4-bit 精度加载模型(与QLoRA绑定, 强制使用)
load_in_4bit: false
# 是否严格匹配模型结构,关闭表示可加载少部分差异结构(如以适配 adapter)
# strict: false
base_model: Qwen/Qwen3-4B-Instruct-2507
# 数据集设置
chat_template: qwen3
datasets:
- path: /workspace/train_dir_0926-02/goal_data.json # 只使用新增的1500条数据
type: chat_template # chat_template(自定义格式) alpaca
roles_to_train: ["assistant"]
field_messages: messages # 标识的字段
message_property_mappings: # message_property_mappings={'role':'role', 'content':'content'})
role: role
content: content
dataset_prepared_path:
val_set_size: 0.1 # 8卡训练可以适当减少验证集比例,提高训练数据利用率
output_dir: /workspace/train_dir_0926-02/checkpoints
sequence_len: 7000 # 模型所能处理的最大上下文长度(默认2048)
pad_to_sequence_len: true
# context_parallel_size: 2 # 长序列拆分至多个GPU(强制要求 mirco_batch_size: 1)
sample_packing: false # Function call数据通常较短,可以启用packing提高效率
eval_sample_packing: false # 评估时拼接多个样本
# 训练超参数 - 针对function call优化
lora_r: 32 # function call需要更强的表达能力,提高rank
lora_alpha: 64 # 设为 2*r,保持合理的缩放
lora_dropout: 0.1 # 8卡训练数据分布更均匀,可以适当降低dropout
# 批处理设置 - 充分利用8卡H100
micro_batch_size: 8 # H100显存大,可以增加batch size
gradient_accumulation_steps: 2 # 8卡并行,可以减少累积步数
auto_find_batch_size: false # 允许Axolotl不断调整batch_size ⚠️Zero-3不适用
num_epochs: 3 # function call数据质量高,可以多训练几轮
# 优化器设置
optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 8e-5 # 在已有模型基础上微调function call,稍微提高学习率
# 精度设置
bf16: auto
tf32: true
# 日志和评估
logging_steps: 30 # 8卡训练速度快,可以适当减少日志频率
flash_attention: true
warmup_ratio: 0.1 # 已有基础模型,减少warmup
evals_per_epoch: 1 # 适中的评估频次
saves_per_epoch: 1 # 适中的保存频次
weight_decay: 0.01 # 8卡训练更稳定,减少正则化
# FSDP配置 - 针对8卡H100优化
fsdp:
- full_shard
- auto_wrap
fsdp_config:
fsdp_limit_all_gathers: true
fsdp_sync_module_states: true
fsdp_offload_params: false # H100显存充足,无需offload
fsdp_use_orig_params: false
fsdp_cpu_ram_efficient_loading: true
fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
fsdp_transformer_layer_cls_to_wrap: Qwen3DecoderLayer
fsdp_state_dict_type: FULL_STATE_DICT
fsdp_sharding_strategy: FULL_SHARD
fsdp_activation_checkpointing: true # 使用FSDP的activation checkpointing
</details><br>
This model is a fine-tuned version of Qwen/Qwen3-4B-Instruct-2507 on the /workspace/train_dir_0926-02/goal_data.json dataset. It achieves the following results on the evaluation set:
More information needed
More information needed
More information needed
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss | Mem Active(gib) | Mem Allocated(gib) | Mem Reserved(gib) |
|---|---|---|---|---|---|---|
| No log | 0 | 0 | 1.1221 | 83.21 | 83.01 | 84.11 |
| 0.2295 | 1.0 | 54 | 0.0622 | 121.88 | 121.69 | 136.37 |
| 0.0512 | 2.0 | 108 | 0.0403 | 121.88 | 121.69 | 136.37 |