falcon-h1-7b-instruct-x-h1r-slerp

Falcon-H1-7B-Instruct × H1R-7B SLERP Merge

To my knowledge, the first published SLERP merge of two Mamba-2 hybrid transformer models.

This is an experimental merge of tiiuae/Falcon-H1-7B-Instruct and tiiuae/Falcon-H1R-7B using Spherical Linear Interpolation (SLERP). The goal was to combine Instruct's general capability and chat fluency with H1R's exceptional reasoning performance (83.1% AIME 2025, 72.1% MMLU-Pro).

Result: The merge produces a functional, coherent model, but benchmark scores are degraded compared to both parents. This repo documents the process, findings, and failure modes as a reference for the community.

Why This Matters

Prior to this experiment, the community assumption was that weight-space merging (SLERP/TIES/DARE) would not work on Mamba-2 hybrid architectures because the state-space model parameters might not interpolate meaningfully the way attention weights do. This experiment demonstrates that:

1. SLERP on Mamba-2 hybrids produces a functional model — it loads, generates coherent text, and solves problems correctly at a non-trivial rate
2. However, the merge degrades performance below both parents — suggesting that SSM parameter interpolation introduces interference that simple SLERP cannot resolve
3. Specific failure modes emerge — repetition loops and degenerate output patterns that don't appear in either parent model

Technical Findings

Architecture Details (Previously Undocumented)

Falcon-H1 7B uses a parallel hybrid design — attention and Mamba-2 heads run in parallel within each block, not in alternating layers. Key details discovered during this work:

• 44 layers (not 32 as commonly assumed from the default config)
• 751 total parameters across both models, confirmed identical structure
• Vocab size mismatch: Instruct has 130,049 tokens, H1R has 130,048 (one extra special token from instruction tuning)

Parameter Name Mapping

The actual weight tensor name patterns in Falcon-H1 7B (useful for anyone doing merges, LoRA targeting, or architecture analysis):

Note: Many guides and configs reference filter: ssm or filter: mlp for these models. Those match nothing. The correct filters are mamba and feed_forward respectively.

Merge Configuration

slices:
  - sources:
      - model: tiiuae/Falcon-H1-7B-Instruct
        layer_range: [0, 44]
      - model: tiiuae/Falcon-H1R-7B
        layer_range: [0, 44]
merge_method: slerp
base_model: tiiuae/Falcon-H1-7B-Instruct
parameters:
  t:
    - filter: self_attn
      value: [0.3, 0.5, 0.7, 0.7, 0.5, 0.3]  # V-shaped: Instruct at edges, H1R in middle
    - filter: feed_forward
      value: [0.4, 0.6, 0.8, 0.8, 0.6, 0.4]  # Slightly more H1R bias
    - filter: mamba
      value: 0.75  # Heavy H1R bias for reasoning SSM params
    - filter: layernorm
      value: 0.5   # Even split
    - value: 0.5    # Default fallback
dtype: bfloat16

The merge was performed with a custom SLERP script (mergekit did not support the falcon_h1 architecture at time of experiment). The script processes safetensor shards sequentially to keep memory usage under 8GB.

Benchmark Results

20 math/reasoning problems, temperature 0, greedy decoding, 4-bit NF4 quantization on RTX 3060 12GB:

Observed Failure Modes

Repetition loops: The merged model falls into degenerate repetition patterns not present in either parent:

"Alright! Let's tackle this math problem together. Alright! Let's tackle this 
math problem together. Alright! Let's tackle this math problem together."

Blank-fill hallucination: The model generates template-style blanks instead of answers:

"Answer: ___________"

Competing output distributions: The two models' chat styles create conflicting attractors — the model oscillates between Instruct's structured formatting and H1R's reasoning-trace style.

These patterns suggest that while the tensor-level interpolation is mathematically valid, the learned output distributions of the two models interfere destructively at certain layers. The Mamba SSM parameters may encode sequential state dynamics that don't interpolate as cleanly as attention projection matrices.

Qualitative Observations

Despite the benchmark degradation, the merged model shows genuine trait transfer:

• Adopts H1R's \boxed{} LaTeX notation for math answers (not present in Instruct)
• Produces more direct code output than either parent (Instruct writes docs, H1R plans — merged model writes code)
• Blends Instruct's empathetic chat style with H1R's analytical depth

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    "iAmBoosted/falcon-h1-7b-instruct-x-h1r-slerp",
    device_map="auto",
    dtype=torch.bfloat16,
    quantization_config=bnb_config,
)
tokenizer = AutoTokenizer.from_pretrained("iAmBoosted/falcon-h1-7b-instruct-x-h1r-slerp")

inputs = tokenizer("Solve step by step: What is 17^2 - 13^2?", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=200, temperature=0.7, do_sample=True, repetition_penalty=1.1)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Tip: Use repetition_penalty=1.1 or higher to mitigate the repetition loop issue.

Hardware

• Merge performed on CPU (AMD Ryzen 7 7800X3D) inside Docker container
• Inference tested on RTX 3060 12GB in 4-bit NF4 quantization
• ~12-13 tok/s without Mamba CUDA kernels (mamba-ssm not installed)
• Container: nvcr.io/nvidia/pytorch:24.12-py3

Future Directions

• TIES/DARE merge may handle parameter interference better than SLERP by pruning conflicting changes before merging
• Lower SLERP ratios (0.3-0.5 for Mamba instead of 0.75) might reduce interference
• QLoRA adapters on H1R directly may be more effective than merging — the model's weaknesses are behavioral (output formatting, over-reasoning on simple tasks) rather than capability gaps

License

This model inherits the TII Falcon LLM License from Falcon-H1R-7B. The base Falcon-H1-7B-Instruct is Apache 2.0, but the merged output is governed by the more restrictive license.

Citation

If you reference this work:

@misc{falcon-h1-slerp-merge-2026,
  title={First SLERP Merge of Mamba-2 Hybrid LLMs: Falcon-H1-7B-Instruct × H1R-7B},
  author={iAmBoosted},
  year={2026},
  url={https://huggingface.co/iAmBoosted/falcon-h1-7b-instruct-x-h1r-slerp}
}