Lamarck-14B-v0.3

merge

Lamarck-14B is the product of a multi-stage merge which emphasizes arcee-ai/Virtuoso-Small in early and finishing layers, and midway features strong emphasis on reasoning, and ends balanced somewhat towards Virtuoso again.

For GGUFs, mradermacher/Lamarck-14B-v0.3-i1-GGUF has you covered. Thank you @mradermacher!

The merge strategy of Lamarck 0.3 can be summarized as:

• Two model_stocks commence specialized branches for reasoning and prose quality.
• For refinement on both model_stocks, DELLA merges re-emphasize selected ancestors.
• For smooth instruction following, a SLERP merges Virtuoso with a DELLA merge of the two branches, where reason vs. prose quality are balanced.
• For finalization and normalization, a TIES merge.

Thanks go to:

• @arcee-ai's team for the ever-capable mergekit, and the exceptional Virtuoso Small model.
• @CultriX for the helpful examples of memory-efficient sliced merges and evolutionary merging. Their contribution of tinyevals on version 0.1 of Lamarck did much to validate the hypotheses of the DELLA->SLERP gradient process used here.
• The authors behind the capable models that appear in the model_stock.

Models Merged

Top influences: These ancestors are base models and present in the model_stocks, but are heavily re-emphasized in the DELLA and SLERP merges.

• arcee-ai/Virtuoso-Small - A brand new model from Arcee, refined from the notable cross-architecture Llama-to-Qwen distillation arcee-ai/SuperNova-Medius. The first two layers are nearly exclusively from Virtuoso. It has proven to be a well-rounded performer, and contributes a noticeable boost to the model's prose quality.

• CultriX/SeQwence-14B-EvolMerge - A top contender on reasoning benchmarks.

Reason: Virtuoso Small is the strongest influence on starting and ending layers, but there are other contributions between:

• CultriX/Qwen2.5-14B-Wernicke - A top performer for Arc and GPQA, Wernicke is re-emphasized in small but highly-ranked portions of the model.

• VAGOsolutions/SauerkrautLM-v2-14b-DPO - This model's influence is understated, but aids BBH and coding capability.

Prose: While the prose module is gently applied, its impact is noticeable on Lamarck 0.3's prose quality, and a DELLA merge re-emphasizes the contributions of two models particularly:

• sthenno-com/miscii-14b-1028

• underwoods/medius-erebus-magnum-14b

Model stock: Two model_stock merges, specialized for specific aspects of performance, are used to mildly influence a large range of the model.

• sometimesanotion/lamarck-14b-reason-model_stock

• sometimesanotion/lamarck-14b-prose-model_stock - This brings in a little influence from EVA-UNIT-01/EVA-Qwen2.5-14B-v0.2, oxyapi/oxy-1-small, and allura-org/TQ2.5-14B-Sugarquill-v1.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here