qwen3.5-9b-clinical-lora-v2-think

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>

<details><summary>See axolotl config</summary>

axolotl version: 0.17.0.dev0

# 1. Base Model Configuration
base_model: Qwen/Qwen3.5-9B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

# 2. Precision Settings (LoRA only, NO Quantization)
load_in_8bit: false
load_in_4bit: false
strict: false

# 3. LoRA Adapter Configuration
adapter: lora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true

# 4. Dataset Configuration
datasets:
  - path: data/processed/diagnosis.train.jsonl
    type: chat_template
    chat_template: qwen3_5

test_datasets:
  - path: data/processed/diagnosis.val.jsonl
    type: chat_template
    chat_template: qwen3_5
    split: train

val_set_size: 0
# Cache the tokenized dataset so re-runs skip preprocessing.
dataset_prepared_path: last_run_prepared

# 5. Training Hyperparameters
sequence_len: 4096
# sample_packing triggers a bug in axolotl's qwen3_5 monkeypatch (get_cu_seqlens
# calls .view() on a non-contiguous position_ids tensor from packing -> RuntimeError)
# -> disabled, same workaround as the gemma config.
sample_packing: false
eval_sample_packing: false
pad_to_sequence_len: false

# SFT Hyperparameters
gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 3
# paged_adamw_8bit halves optimizer-state VRAM vs 32bit, negligible LoRA quality impact.
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 1e-4
# ~560 total steps (1494 examples / eff. batch 8 * 3 epochs); 0.1 ratio ~= 56 warmup steps.
warmup_ratio: 0.1

# 6. Hardware and Performance Optimization
bf16: auto
fp16: false
tf32: true   # free matmul speedup on Ampere+ (RTX PRO 6000 / A6000), no quality cost.
flash_attention: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false

# Liger fused kernels: big VRAM win on the large-vocab LM head + faster RMSNorm/GLU.
# Compatible with sample_packing: false and sdp/flash attention (doesn't touch attention).
plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rms_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: true

# Dataloader throughput.
dataloader_num_workers: 2
dataloader_pin_memory: true
dataloader_prefetch_factor: 8

# 7. Logging and Output
output_dir: results/qwen3.5-9b-lora-out
logging_steps: 10
save_steps: 100
eval_steps: 100

</details><br>

results/qwen3.5-9b-lora-out

This model is a fine-tuned version of Qwen/Qwen3.5-9B on the data/processed/diagnosis.train.jsonl dataset. It achieves the following results on the evaluation set:

• Loss: 0.5346
• Ppl: 1.7068
• Memory/max Active (gib): 34.5
• Memory/max Allocated (gib): 34.5
• Memory/device Reserved (gib): 38.37

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 53
• training_steps: 539

Training results

Framework versions

• PEFT 0.19.1
• Transformers 5.10.2
• Pytorch 2.10.0+cu128
• Datasets 4.8.5
• Tokenizers 0.22.2