Downloads · 30 days
7
14% of all-time downloads
axolotl-ai-co/Gemma3-12B_qat
Gemma3-12B_qat is a image-text-to-text model from axolotl-ai-co. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
7
14% of all-time downloads
All-time downloads
50
Public
Parameters
12.2B
144 GB on disk
Likes
0
Public
Click a slice to open those files.
.bin82.3 GB · 61%
From the Hugging Face model README
axolotl version: 0.13.0.dev0
base_model: google/gemma-3-12b-it
# Automatically upload checkpoint and final model to HF
# hub_model_id: username/custom_model_name
load_in_8bit: false
load_in_4bit: false
strict: false
plugins:
- axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_layer_norm: true
liger_fused_linear_cross_entropy: true
dataset_prepared_path: /home/ubuntu/cluster-axolotl/alpaca_prepared_gemma12b
seed: 42
chat_template: gemma3
datasets:
- path: tatsu-lab/alpaca
type: alpaca
output_dir: ./outputs/qat_out_gemma/
sequence_len: 8096
sample_packing: true
flash_attention: true
qat:
activation_dtype: nvfp4
weight_dtype: nvfp4
group_size: 16 # only group_size of 16 is supported with nvfp4
wandb_project: qat_b200
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
gradient_accumulation_steps: 1
micro_batch_size: 16
num_epochs: 1
optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 4e-5
bf16: true
tf32: true
resume_from_checkpoint:
logging_steps: 1
evals_per_epoch: 1
saves_per_epoch: 1
warmup_ratio: 0.1
weight_decay: 0.0
fsdp_version: 2
fsdp_config:
offload_params: false
cpu_ram_efficient_loading: true
auto_wrap_policy: TRANSFORMER_BASED_WRAP
transformer_layer_cls_to_wrap: Gemma3DecoderLayer
state_dict_type: FULL_STATE_DICT
sharding_strategy: FULL_SHARD
reshard_after_forward: true
activation_checkpointing: true
special_tokens:
# save_first_step: true # uncomment this to validate checkpoint saving works with your config
</details><br>
This model was trained from scratch on the tatsu-lab/alpaca dataset.
More information needed
More information needed
More information needed
The following hyperparameters were used during training: