Downloads · 30 days
0
0% of all-time downloads
rayonlabs/tiny-random-LlamaForCausalLM-Magpie-Reasoning-v0_1-150K-02fa1c1f-9647-4e4f-8cb2-27ce7de9079c
tiny-random-LlamaForCausalLM-Magpie-Reasoning-v0_1-150K-02fa1c1f-9647-4e4f-8cb2-27ce7de9079c is a machine learning model from rayonlabs. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
0
0% of all-time downloads
All-time downloads
11
Public
Repo size
4.5 MB
Likes
0
Public
Click a slice to open those files.
.json3.6 MB · 85%
From the Hugging Face model README
axolotl version: 0.4.1
adapter: lora
base_model: trl-internal-testing/tiny-random-LlamaForCausalLM
bf16: true
chat_template: llama3
dataset_prepared_path: null
datasets:
- data_files:
- 57fd039527663fc0_train_data.json
ds_type: json
format: custom
path: /workspace/input_data/57fd039527663fc0_train_data.json
type:
field_input: knowledge
field_instruction: intent
field_output: response
format: '{instruction} {input}'
no_input_format: '{instruction}'
system_format: '{system}'
system_prompt: ''
debug: null
device_map:
? ''
: 0,1,2,3,4,5,6,7
early_stopping_patience: 2
eval_max_new_tokens: 128
eval_steps: 400
eval_table_size: null
flash_attention: true
gradient_accumulation_steps: 4
gradient_checkpointing: true
group_by_length: false
hub_model_id: Alphatao/daa4a8b2-4cda-4184-a2f2-686fc8389361
hub_repo: null
hub_strategy: null
hub_token: null
learning_rate: 0.0002
load_best_model_at_end: true
load_in_4bit: false
load_in_8bit: false
local_rank: null
logging_steps: 1
lora_alpha: 32
lora_dropout: 0.05
lora_fan_in_fan_out: null
lora_model_dir: null
lora_r: 16
lora_target_linear: true
lora_target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
- down_proj
- up_proj
lr_scheduler: cosine
max_grad_norm: 1.0
max_steps: 90538
micro_batch_size: 2
mlflow_experiment_name: /tmp/57fd039527663fc0_train_data.json
model_type: AutoModelForCausalLM
num_epochs: 2
optimizer: adamw_bnb_8bit
output_dir: miner_id_24
pad_to_sequence_len: true
resume_from_checkpoint: null
s2_attention: null
sample_packing: false
save_steps: 400
sequence_len: 2048
strict: false
tf32: true
tokenizer_type: AutoTokenizer
train_on_inputs: false
trust_remote_code: true
val_set_size: 0.03351206434316354
wandb_entity: null
wandb_mode: online
wandb_name: 02fa1c1f-9647-4e4f-8cb2-27ce7de9079c
wandb_project: Gradients-On-Demand
wandb_run: your_name
wandb_runid: 02fa1c1f-9647-4e4f-8cb2-27ce7de9079c
warmup_steps: 10
weight_decay: 0.0
xformers_attention: null
</details><br>
This model is a fine-tuned version of trl-internal-testing/tiny-random-LlamaForCausalLM on the None dataset. It achieves the following results on the evaluation set:
More information needed
More information needed
More information needed
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 10.3767 | 0.0001 | 1 | 10.3779 |
| 10.3427 | 0.0222 | 400 | 10.3398 |
| 10.3362 | 0.0444 | 800 | 10.3324 |
| 10.3379 | 0.0666 | 1200 | 10.3285 |
| 10.3285 | 0.0888 | 1600 | 10.3264 |
| 10.3319 | 0.1110 | 2000 | 10.3245 |
| 10.3274 | 0.1331 | 2400 | 10.3233 |
| 10.3247 | 0.1553 | 2800 | 10.3220 |
| 10.3313 | 0.1775 | 3200 | 10.3210 |
| 10.3211 | 0.1997 | 3600 | 10.3200 |
| 10.3256 | 0.2219 | 4000 | 10.3193 |
| 10.3283 | 0.2441 | 4400 | 10.3186 |
| 10.3188 | 0.2663 | 4800 | 10.3180 |
| 10.3278 | 0.2885 | 5200 | 10.3174 |
| 10.318 | 0.3107 | 5600 | 10.3169 |
| 10.3179 | 0.3329 | 6000 | 10.3165 |
| 10.3149 | 0.3551 | 6400 | 10.3159 |
| 10.3193 | 0.3773 | 6800 | 10.3153 |
| 10.3174 | 0.3994 | 7200 | 10.3149 |
| 10.3163 | 0.4216 | 7600 | 10.3147 |
| 10.3198 | 0.4438 | 8000 | 10.3144 |
| 10.3077 | 0.4660 | 8400 | 10.3138 |
| 10.3277 | 0.4882 | 8800 | 10.3135 |
| 10.3161 | 0.5104 | 9200 | 10.3133 |
| 10.309 | 0.5326 | 9600 | 10.3129 |
| 10.3324 | 0.5548 | 10000 | 10.3127 |
| 10.3138 | 0.5770 | 10400 | 10.3126 |
| 10.3176 | 0.5992 | 10800 | 10.3124 |
| 10.317 | 0.6214 | 11200 | 10.3122 |
| 10.3153 | 0.6436 | 11600 | 10.3120 |
| 10.3422 | 0.6657 | 12000 | 10.3117 |
| 10.3201 | 0.6879 | 12400 | 10.3116 |
| 10.3169 | 0.7101 | 12800 | 10.3110 |
| 10.305 | 0.7323 | 13200 | 10.3111 |
| 10.3171 | 0.7545 | 13600 | 10.3109 |
| 10.3278 | 0.7767 | 14000 | 10.3105 |
| 10.316 | 0.7989 | 14400 | 10.3104 |
| 10.322 | 0.8211 | 14800 | 10.3102 |
| 10.3154 | 0.8433 | 15200 | 10.3100 |
| 10.3126 | 0.8655 | 15600 | 10.3099 |
| 10.3163 | 0.8877 | 16000 | 10.3097 |
| 10.3061 | 0.9098 | 16400 | 10.3097 |
| 10.3199 | 0.9320 | 16800 | 10.3095 |
| 10.3191 | 0.9542 | 17200 | 10.3093 |
| 10.314 | 0.9764 | 17600 | 10.3092 |
| 10.3038 | 0.9986 | 18000 | 10.3090 |
| 10.3158 | 1.0208 | 18400 | 10.3089 |
| 10.3154 | 1.0430 | 18800 | 10.3088 |
| 10.3062 | 1.0652 | 19200 | 10.3088 |
| 10.3124 | 1.0874 | 19600 | 10.3086 |
| 10.3169 | 1.1096 | 20000 | 10.3085 |
| 10.3177 | 1.1318 | 20400 | 10.3085 |
| 10.3152 | 1.1540 | 20800 | 10.3084 |
| 10.3131 | 1.1761 | 21200 | 10.3084 |
| 10.3157 | 1.1983 | 21600 | 10.3082 |
| 10.3236 | 1.2205 | 22000 | 10.3081 |
| 10.3229 | 1.2427 | 22400 | 10.3081 |
| 10.3201 | 1.2649 | 22800 | 10.3080 |
| 10.3149 | 1.2871 | 23200 | 10.3080 |
| 10.3048 | 1.3093 | 23600 | 10.3078 |
| 10.3163 | 1.3315 | 24000 | 10.3078 |
| 10.3126 | 1.3537 | 24400 | 10.3078 |
| 10.3241 | 1.3759 | 24800 | 10.3077 |
| 10.3167 | 1.3981 | 25200 | 10.3077 |
| 10.301 | 1.4202 | 25600 | 10.3077 |
| 10.3015 | 1.4424 | 26000 | 10.3076 |
| 10.318 | 1.4646 | 26400 | 10.3076 |
| 10.3118 | 1.4868 | 26800 | 10.3076 |
| 10.3149 | 1.5090 | 27200 | 10.3075 |
| 10.319 | 1.5312 | 27600 | 10.3074 |
| 10.3021 | 1.5534 | 28000 | 10.3075 |
| 10.3213 | 1.5756 | 28400 | 10.3074 |
| 10.3223 | 1.5978 | 28800 | 10.3074 |
| 10.3263 | 1.6200 | 29200 | 10.3074 |
| 10.3171 | 1.6422 | 29600 | 10.3074 |
| 10.3188 | 1.6644 | 30000 | 10.3074 |
| 10.3089 | 1.6865 | 30400 | 10.3074 |
| 10.3154 | 1.7087 | 30800 | 10.3074 |