Gemma-4-31B-glimmer-rp-v0.3

gemma4-31b-it-glimmer-rp-r16a32-v3

This model was fine-tuned using SFT.

Training procedure

Hyperparameters

LoRA configuration

Dataset statistics

<details> <summary>Training config</summary>

model_name_or_path: gemma-4-31B-glimmer-rp-v0.2-merged
data_config: data_v3.yaml
prepared_dataset: prepared_packed_fixed
output_dir: gemma4-31b-it-glimmer-rp-r16a32-v3
chat_template_path: chat_template_with_channel_v4_train.jinja
attn_implementation: flex_attention
bf16: true
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
use_cce: true
chunked_mlp: true
chunked_mlp_chunks: 16
dataloader_num_workers: 2
dataloader_pin_memory: true
model_parallel: true
max_memory:
  0: 16GiB
  1: 24GiB
max_length: 6144
per_device_train_batch_size: 1
gradient_accumulation_steps: 4
pad_to_multiple_of: 128
use_peft: true
load_in_4bit: true
bnb_4bit_quant_type: nf4
lora_r: 16
lora_alpha: 32
lora_dropout: 0.0
use_rslora: false
lora_target_modules: .*language_model\.layers\.\d+\.(self_attn\.(q|k|v|o)_proj|mlp\.(gate|up|down)_proj)$
learning_rate: 5.0e-06
lr_scheduler_type: constant_with_warmup
warmup_ratio: 0.05
weight_decay: 0.0
max_grad_norm: 1.0
optim: paged_adamw_8bit
num_train_epochs: 1
saves_per_epoch: 2
save_total_limit: 4
rolling_save_steps: 30
rolling_save_total_limit: 1
assistant_only_loss: true
full_mask_reasoning: true
logging_steps: 1
disable_tqdm: false
report_to: wandb
run_name: g4-31b-it-glimmer-rp-v3

</details> <details> <summary>Data config</summary>

datasets:
- path: instruct.jsonl
  type: chat
  truncation_strategy: split
- path: rp_generation_final.jsonl
  type: chat
  truncation_strategy: split
shuffle_datasets: true
shuffle_combined: true
shuffle_seed: 42
eval_split: 0
split_seed: 42
assistant_only_loss: true

</details>

Framework versions

• PEFT 0.18.1
• Loft: 0.1.0
• Transformers: 5.5.4
• Pytorch: 2.6.0+cu124
• Datasets: 4.6.1
• Tokenizers: 0.22.2