Downloads · 30 days
17
7% of all-time downloads
joseagmz/olmo-7B-Tinybook-epochs-1-lr-0002
olmo-7B-Tinybook-epochs-1-lr-0002 is a text generation model from joseagmz. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
17
7% of all-time downloads
All-time downloads
259
Public
Repo size
124 GB
Likes
0
Public
Click a slice to open those files.
.pt96.4 GB · 78%
From the Hugging Face model README
axolotl version: 0.4.0
base_model: allenai/OLMo-7B
tokenizer_type: AutoTokenizer
model_type: AutoModelForCausalLM
trust_remote_code: true
load_in_8bit: false
load_in_4bit: false
strict: false
datasets:
- path: utrgvseniorproject/Tinybook
type: completion
dataset_prepared_path: /home/josegomez15/med-llm/last_run_prepared
val_set_size: 0.05
output_dir: ./ollama-7B-Tinybook-epochs-1-lr-0002
sequence_len: 4096
sample_packing: false
pad_to_sequence_len: true
wandb_project: olmo-7B-Tinybook
wandb_entity: utrgvmedai
wandb_watch:
wandb_name: olmo-7B-Tinybook-epochs-1-lr-0002
wandb_log_model:
gradient_accumulation_steps: 1
micro_batch_size: 1
num_epochs: 1
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002
train_on_inputs: True # make sure you have this on True
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: false #olmo doesn't support
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
flash_attn_cross_entropy: false
flash_attn_rms_norm: true
flash_attn_fuse_qkv: false
flash_attn_fuse_mlp: true
warmup_steps: 100
evals_per_epoch: 4
eval_table_size:
eval_sample_packing:
saves_per_epoch: 1
debug:
deepspeed: /home/josegomez15/axolotl/deepspeed_configs/zero2.json
weight_decay: 0.1
fsdp:
fsdp_config:
special_tokens:
</details><br>
This model is a fine-tuned version of allenai/OLMo-7B on the None dataset. It achieves the following results on the evaluation set:
More information needed
More information needed
More information needed
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 4.3047 | 0.33 | 1 | 2.4062 |
| 4.0859 | 0.67 | 2 | 2.3906 |
| 3.9805 | 1.0 | 3 | 2.3906 |