Orca2-13B-selfmerge-39B

vmajor/Orca2-13B-selfmerge-39B

Orca2-13B-selfmerge-39B is a text generation model from vmajor. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as ms-pl.

This model is a result of merging three Orca2-13B models with itself using 'mergekit-legacy'. Merge parameters were passthrough for one and --weight 0.5 --density 0.5 for the second merge.

Downloads · 30 days

0% of all-time downloads

All-time downloads

39.8K

Public

Parameters

13B

26 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors26 GB · 100%

At a glance

Task: Text Generation
Library: transformers
License: ms-pl
Model type: llama
Access: Public
Created: Dec 1, 2023
Updated: Dec 19, 2023
SHA: 51c02c8b

Try a prompt

Model card

From the Hugging Face model README

This model is a result of merging three Orca2-13B models with itself using 'mergekit-legacy'. Merge parameters were passthrough for one and --weight 0.5 --density 0.5 for the second merge.

This merged model showed marginal improvement in perplexity scores:

The perplexity for Orca-2-13b is: 7.595028877258301 The perplexity for orca2-26B-self-merge is: 7.550178050994873 The perplexity for orca2-39B-self-merge is: NC

The following table summarizes the model performance across a range of benchmarks:

Model	Average	ARC	HellaSwag	MMLU	TruthfulQA	Winogrande	GSM8K
microsoft/Orca-2-13b	58.64	60.67	79.81	60.37	56.41	76.64	17.97
vmajor/Orca2-13B-selfmerge-26B	62.24	60.84	79.84	60.32	56.38	76.87	39.2
vmajor/Orca2-13B-selfmerge-39B	62.24	60.84	79.84	60.32	56.38	76.87	39.2

Interestingly the GSM8K performance more than doubled with the first self merge. Second self merge resulting in the 39B model did not produce any further gains.

license: ms-pl

Task: Text Generation
Library: transformers
Type: llama
License: ms-pl
Created: Dec 1, 2023
Updated: Dec 19, 2023