Llama-2-7b-Bogpit

LLaMa-2-7b The Pit Project/Bogpit.

Model Details

• Backbone Model: LLaMA-2
• Language(s): English
• Library: HuggingFace Transformers
• License: Use of this model is governed by the Meta license. In order to download the model weights and tokenizer, please visit the website and accept their License before downloading the model weights.

Datasets Details

• Scraped posts of a particular subject within an image board.
• The dataset was heavily augmented with various types of filtering to improve coherency and relevency to the origin and our goals.
• For our Bogpit model, it contains 361,050 entries.

Prompt Template

The model was not trained in an instructional or chat-style format, please ensure your inference program does not attempt to inject anything more than your sole input when inferencing, simply type whatever comes to mind and the model will attempt to complete it.

Hardware and Software

• Hardware: We utilized 5.5 Nvidia RTX 4090 hours for training our model.
• Training Factors: We created this model using Axolotl

Training details

• The rank and alpha we used was 128 and 16.
• Our learning rate was 2e-4 with 10 warmups steps with a cosine scheduler for 3 epoch.
• Our batch size was 5 microbatch
• Sample packing was used.

Limitations

It is strongly recommend to not deploy this model into a real-world environment unless its behavior is well-understood and explicit and strict limitations on the scope, impact, and duration of the deployment are enforced.