Qwen3.5-lora-sft-v5-1-64k

Qwen3.5-lora-sft-v5-1-64k

This repository contains a LoRA adapter for Qwen/Qwen3.5-397B-A17B, trained with LLaMA-Factory on the amdpilot_v5_1 SFT dataset.

This is an adapter-only release. You need the base model Qwen/Qwen3.5-397B-A17B to use it.

Key training settings

• Fine-tuning method: LoRA
• LoRA rank / alpha: 32 / 64
• Context window: 65536
• Packing: true
• Neat packing: false
• Precision: bf16
• Distributed setup: 8x AMD MI355X
• Epochs: 10

Final metrics

• Final train loss: 0.0630452295144399
• Final eval loss: 0.133148193359375
• Train runtime: 47396.7738s (13.17h)

Eval trajectory

Dataset coverage note

On the current amdpilot_v5_1 training split, 65536 tokens cover about 82/89 samples (92.13%). This is substantially better coverage than the earlier 32768 setting.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "Qwen/Qwen3.5-397B-A17B"
adapter_id = "JinnP/Qwen3.5-lora-sft-v5-1-64k"

tokenizer = AutoTokenizer.from_pretrained(adapter_id, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(base_model_id, trust_remote_code=True)
model = PeftModel.from_pretrained(base_model, adapter_id)

Files

• adapter_model.safetensors: LoRA adapter weights
• adapter_config.json: PEFT adapter config
• tokenizer.json / tokenizer_config.json / chat_template.jinja: tokenizer assets
• all_results.json / eval_results.json / train_results.json: training metrics