Mixtral-Uphill-Crush-Crest-v0.4

Mixtral-Crush-Crest-Uphill-v0.4

This is a merge of pre-trained language models created using mergekit.

TL;DR

This model is intended for RP and ERP, but it's capable at storywriting. It has phrasing that's gloriously divergent from most models that have synthetic training.

Pacing is acceptable and You-isms are minimal to moderate depending on the card.

It's more instruction-following than your average RP-capable model.

Tests were done primarily with sampler strategy:

• t=1.21
• min.p=0.025
• sampler order=Temp Last (6,0,1,3,2,4,5)
• no miro
• no 'top' samplers
• smoothing, rep.pen, dry, d.temp disabled
• oh, and I've had better results setting moeexperts to like 5 or 6. 6 seemed to be the sweet spot.

Testing was done against Q6_K and Q5_K_M static quants. No importance matrix was used.

Highly recommended is the anti-slop, anti-rp cliche token banlist. This pushes the model even further in the direction of creativity.

Yes, we are aware that Mixtral is considered passe.

In the 20+ months since we started doing Mixtral work, several new base models have emerged and they're almost all hpt garbage for RP of any kind. There's too much synthetic training data in all the other models (especially Mistral-Small and Llama3)

Merge Details

Merge Method

This model was merged using the DARE TIES merge method using mistralai/Mixtral-8x7B-v0.1 as a base.

Models Merged

The following models were included in the merge:

• antisoc-qa-assoc/uphill-instruct-crest-0.1-e2
• antisoc-qa-assoc/uphill-instruct-crush-0.2-e1
• mistralai/Mixtral-8x7B-Instruct-v0.1
• antisoc-qa-assoc/Mixtral-8x7B-Yes-Instruct-LimaRP

Configuration

The following YAML configuration was used to produce this model:

# Faint tecnnique, try 4
# # try3 ran okay but sounded pretty... model-y
#
# First impression: HOLY FUCKING SHIT
#
# Note: nearly two years later I'm trying to bring Mixtral
# back from the dead. There are multiple reasons:
# 1. Mistral-Small is kind of crap and smells like slop.
#    Hell, even the comprehension felt weak but maybe that's
#    just how I tried to sample it.
# 2. Llama3 hasn't been interesting and is definitely crammed
#    with slop.
# 3. Mixtral is probably the least synthetic-trained sounding
#    of all the OG models. Even when I tried the Quen shit
#    it seemed to be just openai. Mixtral is still sloppy.
#
# So, the pieces that are ours are uphill: non-instruct lora
# being applied to the instruct rawdog without an intermediate
# step.
#
# Obviously we're using pure elemental antisoc loras, hush's shit
# but not her merge because the merges aren't "uphill", as in, 
# a lora made with "mixtral non-instruct" applied straight to
# the instruct with loraize.
#
# The notion, which came to me in the middle of the night, is
# to have the hush loras be only barely present layer-wise but
# weighted heavily. Likewise with LimaRP, send uphill from 
# doctor-shotgun's qlora straight into mixtral-instruct
#
# My hypothesis is that we should get really fucking close to
# pure-ass mixtral-instruct in terms of attention, but that
# we're weighting really hard not to write like it. I have no
# idea if that's how it works--I'm a fucking caveman.
#
# What I'm given to understand, and I'm way out of my depth,
# is that the antisoc layers won't have blotched the instruct
# as badly as they usually do, but when they're triggered they
# are dominant. It's entirely possible I've got no idea what
# I'm saying.
# 
models:
  - model: mistralai/Mixtral-8x7B-Instruct-v0.1
    parameters:
      density: 0.9
      weight: 0.4
  # Pure uphill crush e1. I like how e2 sounds but I don't think it's as coherent
  - model: antisoc-qa-assoc/uphill-instruct-crush-0.2-e1
    parameters:
      density: 0.5
      weight: 0.6
  # This is actually an uphill lima but I didn't name it that way.
  - model: antisoc-qa-assoc/Mixtral-8x7B-Yes-Instruct-LimaRP
    parameters:
      # Still just a breath of layers from the thing
      density: 0.125
      # I am gimping its weight compared to hush tunes because limarp has too
      # much ai-slop and amateur-smut cliche slop. Honestly, if there were
      # something better than limarp I'd try to train it myself but I don't
      # know if there is.
      weight: 0.1
  # Pure uphill crest e2 (we don't have any other epochs, hush never shared)
  - model: antisoc-qa-assoc/uphill-instruct-crest-0.1-e2
    parameters:
      density: 0.5
      weight: 0.6
# della sucked ass so dare_ties it is
merge_method: dare_ties
# I know all of these look like instruct but the lora
# is actually not so we go to the base base
base_model: mistralai/Mixtral-8x7B-v0.1
parameters:
  normalize: true
  int8_mask: true
dtype: bfloat16