tFINE-680m-e32-d16-infinity_instruct-L2

tFINE-680m-e32-d16-infinity_instruct-L2

this is an instruction-tuned version of a pretrained t5 with GQA.

Model description

This model is a fine-tuned version of BEE-spoke-data/tFINE-680m-e32-d16-infinity_instruct-L1 on the pszemraj/infinity-instruct-7m-T2T_en dataset (config deduped-L2).

It achieves the following results on the evaluation set:

• Loss: 1.3139
• Num Input Tokens Seen: 361724696

usage

prerequisite: you need to have t5-gqa fork of transformers installed, and accelerate.

from transformers import pipeline

pipe = pipeline(
    "text2text-generation",
    model="BEE-spoke-data/tFINE-680m-e32-d16-infinity_instruct-L2",
    device_map="auto",
)
prompt = "Write me a python fn that demonstrates an advanced sorting algorithm"
res = pipe(
    prompt, max_new_tokens=384, num_beams=4, early_stopping=True, repetition_penalty=1.1
)
print(res[0]["generated_text"])

Quick eval

Quick eval for: BEE-spoke-data/tFINE-680m-e32-d16-infinity_instruct-L2

hf (pretrained=BEE-spoke-data/tFINE-680m-e32-d16-infinity_instruct-L2,trust_remote_code=True,dtype=bfloat16,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2.5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 17868
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 32
• total_train_batch_size: 256
• total_eval_batch_size: 8
• optimizer: Use paged_ademamix_32bit and the args are: No additional optimizer arguments
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.02
• num_epochs: 1.0

Training results