Mistral-Nemo-Instruct-bellman-12b

Model Card for Bellman

This version of bellman is finetuned from Mistral-Nemo-Instruct-2407. It's a rank 128 qlora trained for about 1 epoch. It's finetuned for prompt question answering, based on a dataset created from Swedish wikipedia, with a lot of Sweden-centric questions. New from previous versions is questions from a translated code-feedback dataset, as well as a number of stories.

Consider this a work in progress as I adjust the training for this new model size. Will provide a few updates to the model.

Quality seems to be dependent on "initial seed". Sometimes it's just amazing, and sometimes quite bad. If you're having trouble getting good results, try lowering temperature.

For GGUFs, please look to: <br> https://huggingface.co/mradermacher/Mistral-Nemo-Instruct-bellman-12b-GGUF and <br> https://huggingface.co/mradermacher/Mistral-Nemo-Instruct-bellman-12b-i1-GGUF

Model Details

Rank: 128

Trained with Unsloth on a 3090

Differences from base model:

The base model is pretty good at Swedish already, but my 'vibe check' says this finetune is slightly improved. There are less weird wordings.

Bellman is trained on fairly short answers and tends to be less verbose.

Output example (Mistral-Nemo-Instruct-bellman-12b.i1-Q4_K_M.gguf):

User: "Hej!"

Bellman: "Hej! Vad kan jag hjälpa dig med?"

User: "Berätta lite om Sverige"

Bellman: "Sverige är ett land i norra Europa med en rik historia. Det består av 21 län och har en yta på drygt 450,000 kvadratkilometer. Stockholm är Sveriges huvudstad."

User: "När grundades det?"

Bellman: "Det är inte exakt känt när Sverige grundades. Det var först under 1300-talet att Sverige började ge sig uttryck i internationella affärer."

User: "Vad hände 1523?"

Bellman: "År 1523 blev Gustav Vasa kung över Sverige och inledde reformationen i landet. Han var också den förste monarken att bära titeln "Sveriges Konung"."

User: "Hur länge levde han?"

Bellman: "Gustav Vasa levde från år 1496 till 1560."

Training Parameters

per_device_train_batch_size = 2,<br> gradient_accumulation_steps = 64,<br> num_train_epochs=3,<br> warmup_steps = 5,<br> learning_rate = 1e-4,<br> logging_steps = 15,<br> optim = "adamw_8bit",<br> weight_decay = 0.01,<br> lr_scheduler_type = "linear",<br> seed = 3407,<br> per_device_eval_batch_size = 2,<br> evaluation_strategy="steps",<br> eval_accumulation_steps = 64,<br> eval_steps = 15,<br> eval_delay = 0,<br> save_strategy="steps",<br> save_steps=50,<br>

Model Description

• Developed by: Me
• Funded by: Me
• Model type: Instruct
• Language(s) (NLP): Swedish
• License: Apache 2 License
• Finetuned from model: Mistral-Nemo-Instruct-2407

Model Card Contact

[email protected]