liger-general-medium-v1

Model Card

We provide a low-rank adapter for an instruction-tuned 12B-parameter GPT3-style language model.

Prompting

For instructions that do not require extraneous inputs, the recommended prompt is:

Below is an instruction that describes a task. Write a response that appropriately completes the request.


### Instruction:

< -- instruction goes here --- >

### Response:

For instructions that do require extraneous inputs, the recommended prompt is:

Below is an instruction that describes a task. Write a response that appropriately completes the request.


### Instruction:

< -- instruction goes here -- >

### Input:

< -- extraneous input goes here -- >

### Response:

Since the model performs causal language modeling, the model's response to the prompt is the text completing the sequence beginning with the prompt.

Instruction-Tuning

This model was instruction-tuned on an internally-curated Alpaca-style dataset.

• Epochs: 3
• Batch size: 128
• Cutoff length: 2048
• Learning rate: 3.2e-5, cosine decay
• LoRA r: 8
• LoRA alpha: 16
• LoRA dropout: 0.0
• LoRA target modules: 'query_key_value', 'dense_h_to_4h', 'dense_4h_to_h', 'dense'
• License: The instruction-tuning data is subject to the Creative Commons 4.0 license.

Base Model

This model was instruction-tuned from a 12B variant from the Pythia family.

• Repository: EleutherAI/pythia-12b
• Paper: arxiv:2304.01373
• License: The base model is subject to the Apache 2.0 license.
• Model type: Transformer-based Language Model

Licensing Information

We release this adapter under the Creative Commons NonCommercial (CC BY-NC 4.0) license.