tFINE-300m-instruct-L2

tFINE-base-300m-instruct-L2

This is an "instruct" model fine-tuned originally from pszemraj/tFINE-base-300m in the following phases:

1. two epochs on supernatural instructions
2. instruct tuning on 2M "easy"/L1 instructions
3. instruct tuning on 1M "harder"/L2 instructions

Usage example

from transformers import pipeline

pipe = pipeline(
    "text2text-generation",
    model="pszemraj/tFINE-300m-instruct-L2",
)
prompt = "write a python script to download a file from a url and save as a local file using requests. explain how it works"
res = pipe(
    prompt,
    num_beams=4,
    early_stopping=True,
    max_new_tokens=384,
    no_repeat_ngram_size=7,
)
print(res[0]["generated_text"])

Quick eval

Quick eval for: pszemraj/tFINE-base-300m-instruct-L2

hf (pretrained=pszemraj/tFINE-base-300m-instruct-L2,trust_remote_code=True,dtype=bfloat16,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 4
• eval_batch_size: 8
• seed: 17868
• gradient_accumulation_steps: 16
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1.0