MN-12B-Celeste-V1.9-LimaRP-LoRA

Model Card for Model ID

A LoRA model trained on nothingiisreal/MN-12B-Celeste-V1.9 with Chat-Error/limarp dataset

<!-- Provide a quick summary of what the model is/does. -->

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

• Developed by: supaahiro
• Model type: Decoder only
• Language(s): English
• License: CC BY-NC-SA v4
• Finetuned from model: nothingiisreal/MN-12B-Celeste-V1.9

Uses

Direct Use

This model was uses the ChatML format:

<|im_start|>system\n<|im_end|>\n<|im_start|>user\n%instruction%: %input_text%<|im_end|>\n<|im_start|>assistant\n%output_text%<|im_end|>

Bias, Risks, and Limitations

NSFW, long context, wrapup bias and narration.

Training Data

5000 examples mixed from LIMAERP and LIMA

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Training Hyperparameters

3 epochs with 3e-4 lr, 4096 ctx token, QLoRA

Speeds, Sizes, Times [optional]

It takes about 18 hours to train this model

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: A40 with 9CPUs 48gb ram
• Hours used: 18
• Cloud Provider: Runpod
• Compute Region: US
• Carbon Emitted: 0.5KG

Credits

This model is based on a fine-tuned version of nothingiisreal/MN-12B-Celeste-V1.9 by Chat-Error/limarp using dataset.