Downloads · 30 days
8
11% of all-time downloads
Zoyd/flammenai_Mahou-1.2a-llama3-8B-2_2bpw_exl2
flammenai_Mahou-1.2a-llama3-8B-2_2bpw_exl2 is a text generation model from Zoyd. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama3.
Exllamav2 quant (exl2 / 2.2 bpw) made with ExLlamaV2 v0.0.21
Downloads · 30 days
8
11% of all-time downloads
All-time downloads
75
Public
Repo size
3.4 GB
Likes
0
Public
Click a slice to open those files.
.safetensors3.4 GB · 100%
From the Hugging Face model README
Exllamav2 quant (exl2 / 2.2 bpw) made with ExLlamaV2 v0.0.21
Other EXL2 quants:
| Quant | Model Size | lm_head |
|---|---|---|
| <center>2.2</center> | <center>3250 MB</center> | <center>6</center> |
| <center>2.5</center> | <center>3479 MB</center> | <center>6</center> |
| <center>3.0</center> | <center>3893 MB</center> | <center>6</center> |
| <center>3.5</center> | <center>4311 MB</center> | <center>6</center> |
| <center>3.75</center> | <center>4518 MB</center> | <center>6</center> |
| <center>4.0</center> | <center>4727 MB</center> | <center>6</center> |
| <center>4.25</center> | <center>4935 MB</center> | <center>6</center> |
| <center>5.0</center> | <center>5557 MB</center> | <center>6</center> |
| <center>6.0</center> | <center>6496 MB</center> | <center>8</center> |
| <center>6.5</center> | <center>6902 MB</center> | <center>8</center> |
| <center>8.0</center> | <center>8131 MB</center> | <center>8</center> |

Mahou is our attempt to build a production-ready conversational/roleplay LLM.
Future versions will be released iteratively and finetuned from flammen.ai conversational data.
This model has been trained to use ChatML format.
<|im_start|>system
{{system}}<|im_end|>
<|im_start|>{{char}}
{{message}}<|im_end|>
<|im_start|>{{user}}
{{message}}<|im_end|>
*asterisks**leans against wall cooly* so like, i just casted a super strong spell at magician academy today, not gonna lie, felt badass.
["<", "|", "<|", "\n"]Finetuned using an A100 on Google Colab.
Fine-tune a Mistral-7b model with Direct Preference Optimization - Maxime Labonne
LoRA, model, and training settings:
# LoRA configuration
peft_config = LoraConfig(
r=16,
lora_alpha=16,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules=['k_proj', 'gate_proj', 'v_proj', 'up_proj', 'q_proj', 'o_proj', 'down_proj']
)
# Model to fine-tune
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.bfloat16,
load_in_4bit=True
)
model.config.use_cache = False
# Reference model
ref_model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.bfloat16,
load_in_4bit=True
)
# Training arguments
training_args = TrainingArguments(
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
gradient_checkpointing=True,
learning_rate=5e-5,
lr_scheduler_type="cosine",
max_steps=2000,
save_strategy="no",
logging_steps=1,
output_dir=new_model,
optim="paged_adamw_32bit",
warmup_steps=100,
bf16=True,
report_to="wandb",
)
# Create DPO trainer
dpo_trainer = DPOTrainer(
model,
ref_model,
args=training_args,
train_dataset=dataset,
tokenizer=tokenizer,
peft_config=peft_config,
beta=0.1,
force_use_ref_model=True
)
# Fine-tune model with DPO
dpo_trainer.train()