TrckstrExtDimenBrainDamageNO_USE_RefernceOnly

Trickster Theta V2

PLEASE NOTE This version has an extra dimension, sadly. Due to the union base tokenizer +7 tokens in Trickster's tokenizer. I didn't realise it was through every single layer and that just changing the config would do nothing. This merge be broke. It won't produce output for you. It's here for my reference.

I wanted Hermes 2 Theta's qualities to be more marked, so I took it through a second merge. I have yet to find out how it worked. Will post an update as soon as I get that done.

merged_output

This is a merge of pre-trained language models created using mergekit.

Merge Details

The SCE method I am using for all my SCE merges isn't "pure" SCE. It's an SCE-style hybrid merge.

I used merge_method: sce (so the mergekit math uses the SCE normalization and cosine weighting), but then I overrode its democracy by specifying filters and manual weights per subsystem—attn, mlp, lm_head.

That means:

I kept SCE’s normalization, top-k filtering, and prescale behaviour (so everything stays "stable and smooth" cough in theory).

But I layered on selective weighting, steering which model dominates which parts of the architecture.

In other words, I hijacked SCE’s math engine to run a directed SCE merge—a hybrid between SCE and TIES / linear weighting.

It runs like SCE - for example this one took 7 hours to merge.

Merge Method

This model was merged using the SCE merge method using Babsie/Trickster-Theta-4-70B as a base.

Models Merged

The following models were included in the merge: *NousResearch/Hermes-2-Theta-Llama-3-70B (RoPE config adapted in my own repo, preprepped)

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: Trickster-Theta-4-70B
    name: base
  - model: NousResearch-Hermes-2-Theta-Llama-3-70B
    name: loki
merge_method: sce
base_model: Trickster-Theta-4-70B
parameters:
  select_topk: 0.70
  prescale: true
  normalize: true
weights:
  - filter: ".*(attn|attention).*"
    models: {base: 0.6, loki: 0.4}
  - filter: ".*(mlp|ffn).*"
    models: {base: 0.4, loki: 0.6}
  - filter: ".*(lm_head|output).*"
    models: {base: 0.2, loki: 0.8}
dtype: float32
out_dtype: bfloat16
tokenizer:
  source: union
  target: base