Downloads · 30 days
11
6% of all-time downloads
axolotl-ai-co/kd-llama-1b-evolkit-distill-kd-ratio-0_4
kd-llama-1b-evolkit-distill-kd-ratio-0_4 is a text generation model from axolotl-ai-co. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama3.2.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
11
6% of all-time downloads
All-time downloads
199
Public
Parameters
1.2B
2.5 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors2.5 GB · 99%
From the Hugging Face model README
axolotl version: 0.6.0
base_model: meta-llama/Llama-3.2-1B
tokenizer_config: meta-llama/Llama-3.2-3B
# Automatically upload checkpoint and final model to HF
# hub_model_id: axolotl-ai-co/kd-llama-1b-evolkit-distill-ratio-0_4
plugins:
- axolotl.integrations.kd.KDPlugin
- axolotl.integrations.liger.LigerPlugin
liger_rms_norm: true
liger_glu_activation: true
torch_compile: true
strict: false
chat_template: llama3
kd_trainer: true
kd_ce_alpha: 0.6
kd_alpha: 0.4
kd_temperature: 1.0
dataloader_prefetch_factor: 256
dataloader_num_workers: 4
dataloader_pin_memory: true
gc_steps: -1 # gc at the end of each epoch
datasets:
- field_messages: messages_combined
message_field_content: content
message_field_role: role
logprobs_field: llm_text_generation_vllm_logprobs
path: winglian/evolkit-logprobs-pipeline-75k-v2
type: axolotl.integrations.kd.chat_template
split: train
temperature: 1.0
dataset_prepared_path: last_run_prepared
val_set_size: 0.0
output_dir: ./outputs/out-1b-kd-more-saves
sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true
wandb_project: lobprob-kd-evolkit
wandb_entity: axolotl-ai
wandb_watch:
wandb_name:
wandb_log_model:
gradient_accumulation_steps: 1
micro_batch_size: 4
num_epochs: 3
optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 3e-5
save_safetensors: true
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: true
gradient_checkpointing: true
gradient_checkpointing_kwargs:
use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 100
evals_per_epoch: 2
eval_table_size:
saves_per_epoch: 20
debug:
# deepspeed: deepspeed_configs/zero1.json
weight_decay: 0.0
special_tokens:
pad_token: <|finetune_right_pad_id|>
eos_token: <|eot_id|>
</details><br>
This model is a fine-tuned version of meta-llama/Llama-3.2-1B on the winglian/evolkit-logprobs-pipeline-75k-v2 dataset.
More information needed
More information needed
More information needed
The following hyperparameters were used during training: