RPBizkit-v6-12B

RicardoEstep/RPBizkit-v6-12B

"Experimental Own Mix" using... "Karcher-Mean & DARE TIES merges with Mergekit", + "A custom Python Script".

Don't ask me how i could merge it with only "16GBs of RAM", all models are mixed equaly and all their data have same importance. This is a mix of some known "RP Uncensored" models. I'm not responsible of the use of this.

I recommend to use this version, if you don't want to use the "lastest one".

I did it all for the nookie, so you can take that cookie, and stick it up your...

¡This merge is based in SIX parts!

The "v5" version has a bug named "The Beige Effect". It's basically that the finished mixed model is "polished" and has not "any Voice", becoming very uncreative without iniciative. This is a way to solve this problem... "As is", i took the "original models", from the "model merges" i used on my other "last versions", and join them.

Part One: "Intelligence Core".

This part will use Karcher-Mean to create a new model called "output1".

Models Merged:

• DreadPoor/Krix-12B-Model_Stock as Base.
• romaingrx/red-teamer-mistral-nemo
• Delta-Vector/Ohashi-NeMo-12B
• TheDrummer/UnslopNemo-12B-v4.1
• HumanLLMs/Human-Like-Mistral-Nemo-Instruct-2407
• yamatazen/FusionEngine-12B-Lorablated
• shisa-ai/shisa-v2-mistral-nemo-12b
• Elizezen/Himeyuri-v0.1-12B

Configuration Used:

models:
  - model: DreadPoor/Krix-12B-Model_Stock
  - model: TheDrummer/UnslopNemo-12B-v4.1
  - model: yamatazen/FusionEngine-12B-Lorablated
  - model: Delta-Vector/Ohashi-NeMo-12B
  - model: HumanLLMs/Human-Like-Mistral-Nemo-Instruct-2407
  - model: G:\romaingrx_red-teamer-mistral-nemo   # Had to use it locally.
  - model: Elizezen/Himeyuri-v0.1-12B
  - model: shisa-ai/shisa-v2-mistral-nemo-12b
merge_method: karcher
base_model: DreadPoor/Krix-12B-Model_Stock
parameters:
  normalize: true
  karcher_steps: 96   # Extremely Clean, Stable, High‑quality Merge.
  tolerance: 1e-8    # Extremely smoothness and Minimal artifacts.
dtype: bfloat16
tokenizer:
  source: union

Part Two: "Narrative Core".

This part will use DARE TIES to create a new model called "output2".

Models Merged:

• ArliAI/Mistral-Nemo-12B-ArliAI-RPMax-v1.2 as Base.
• SicariusSicariiStuff/Impish_Bloodmoon_12B
• allura-org/MN-12b-RP-Ink
• FallenMerick/MN-Violet-Lotus-12B
• MuXodious/Wayfarer-2-12B-absolute-heresy

Configuration Used:

models:
  - model: ArliAI/Mistral-Nemo-12B-ArliAI-RPMax-v1.2
    parameters:
      weight: 0.4
      density: 1.0
  - model: allura-org/MN-12b-RP-Ink
    parameters:
      weight: 0.3
      density: 0.95
  - model: MuXodious/Wayfarer-2-12B-absolute-heresy
    parameters:
      weight: 0.2
      density: 0.9
  - model: FallenMerick/MN-Violet-Lotus-12B
    parameters:
      weight: 0.07
      density: 0.85
  - model: SicariusSicariiStuff/Impish_Bloodmoon_12B
    parameters:
      weight: 0.03
      density: 0.8
merge_method: dare_ties
base_model: ArliAI/Mistral-Nemo-12B-ArliAI-RPMax-v1.2
parameters:
  normalize: true
  int8_mask: false
dtype: bfloat16
tokenizer:
  source: union

Part Three: "Dark Style Core".

This part will use DARE TIES to create a new model called "output3".

Models Merged:

• DavidAU/MN-GRAND-Gutenberg-Lyra4-Lyra-12B-DARKNESS as Base.
• allura-org/Bigger-Body-12b
• ChaoticNeutrals/Nera_Noctis-12B
• ReadyArt/Forgotten-Safeword-12B-v4.0
• Lambent/Gilded-Arsenic-12B

Configuration Used:

models:
  - model: DavidAU/MN-GRAND-Gutenberg-Lyra4-Lyra-12B-DARKNESS
    parameters:
      weight: 0.35
      density: 1.0
  - model: ChaoticNeutrals/Nera_Noctis-12B
    parameters:
      weight: 0.25
      density: 0.95
  - model: Lambent/Gilded-Arsenic-12B
    parameters:
      weight: 0.2
      density: 0.9
  - model: ReadyArt/Forgotten-Safeword-12B-v4.0
    parameters:
      weight: 0.15
      density: 1.0
  - model: allura-org/Bigger-Body-12b
    parameters:
      weight: 0.05
      density: 0.7
merge_method: dare_ties
base_model: DavidAU/MN-GRAND-Gutenberg-Lyra4-Lyra-12B-DARKNESS
parameters:
  normalize: true
  int8_mask: false
dtype: bfloat16
tokenizer:
  source: union

Part Four: "Mixing the Cores".

This part will use DARE TIES to create a new model called "output4". I used my last model Tokenizer as it's clean based on "Mistral".

Models Merged:

• .\output3 as Base.
• .\output2
• .\output1

Configuration Used:

models:
  - model: .\output1  #Intelligence
    parameters:
      weight: 0.3
  - model: .\output2  #Narrative
    parameters:
      weight: 0.32
  - model: .\output3  #Dark
    parameters:
      weight: 0.38
merge_method: dare_ties
base_model: .\output3  #Dark
parameters:
  normalize: true
  int8_mask: false
dtype: bfloat16
tokenizer:
  source: RicardoEstep/RPBizkit-v5-12B-Lorablated

Part Five: "Light Re-Abiliteration" (just for stability).

I modified the "v5 script", for a better "Hybridithation LoRas", will create a new model called "output5".

LoRa used:

• nbeerbower/Mistral-Nemo-12B-abliterated-LORA

Script Used:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# -------- Configuration --------
base_model_path = "./output4"
tokenizer_path = "./output4"
lora_path = "nbeerbower/Mistral-Nemo-12B-abliterated-LORA"
output_path = "./output5"

# Hybrid scaling
ATTENTION_SCALE = 0.5
MLP_SCALE = 0.25

EXPECTED_VOCAB_SIZE = 131072

# --------------------------
print("Loading base model...")

model = AutoModelForCausalLM.from_pretrained(
    base_model_path,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True
)

model.config.use_cache = False

# --- Fix Embeddings ---
print("Checking embeddings...")

current_vocab_size = model.get_input_embeddings().weight.shape[0]

if current_vocab_size != EXPECTED_VOCAB_SIZE:
    print(f"Resizing embeddings from {current_vocab_size} → {EXPECTED_VOCAB_SIZE}")
    model.resize_token_embeddings(EXPECTED_VOCAB_SIZE)

model.tie_weights()

# --- APPLY LoRA MANUALLY ---
print("Loading LoRA...")

model = PeftModel.from_pretrained(
    model,
    lora_path,
    adapter_name="default"
)

# --- Hybrid Scaling ---
print("Applying hybrid scaling...")

adapter_name = "default"

attention_layers = ["q_proj", "k_proj", "v_proj", "o_proj"]
mlp_layers = ["up_proj", "down_proj", "gate_proj"]

for name, module in model.named_modules():

    # Only modify real LoRA layers
    if hasattr(module, "lora_A") and hasattr(module, "lora_B"):

        if any(layer in name for layer in attention_layers):

            if hasattr(module, "scaling"):
                module.scaling[adapter_name] = ATTENTION_SCALE

        elif any(layer in name for layer in mlp_layers):

            if hasattr(module, "scaling"):
                module.scaling[adapter_name] = MLP_SCALE

# --- Merge LoRA ---
print("Merging LoRA into base weights...")

model = model.merge_and_unload(progressbar=True)

# --- Load Tokenizer ---
print("Loading tokenizer...")

tokenizer = AutoTokenizer.from_pretrained(
    tokenizer_path,
    trust_remote_code=True
)

# --- Save Final Model ---
print("Saving final model...")

model.save_pretrained(
    output_path,
    safe_serialization=True
)

tokenizer.save_pretrained(output_path)

print("Hybrid LoRA merge complete!")

Part Six: "Light Re-UnSlop" (just for stability).

This part will use DARE TIES to create a new model called "output6".

Models Merged:

• .\output5
• TheDrummer/UnslopNemo-12B-v4.1

Configuration Used:

models:
  - model: .\output5  #LoRA Applied
    parameters:
      weight: 0.94
  - model: TheDrummer/UnslopNemo-12B-v4.1
    parameters:
      weight: 0.07
      density: 0.9
merge_method: dare_ties
base_model: .\output5
parameters:
  normalize: true
  int8_mask: true
dtype: bfloat16
tokenizer:
  source: base

Final Notes;

This version has "Clean Tokenizer & Embedding Sizes (131072)" based on "Mistral". But, the model will drift if any "Chat Template" based on "ChatML" or "Mistral" is used. The recommended chat template to use is "Alpaca (with "RAW" inputs)". The configuration files are already "Tweaked", to don't use any "Chat Template".

While it's supposed to support a "128K (131072)" context size. The recommended max context size is "8K (8192)".

Rapid Benchmarking - "64-Shot + 5318008 Seed" Evaluation.

These are the results of running "lm-evaluation-harness", using the "Q8_0 Model", with an "8K Context Window". Executed on a free "Google Collab - CUDA-accelerated backend (T4)" instance.

• Quantized Model, Here RicardoEstep/RPBizkit-v6-12B-GGUF.