Downloads · 30 days
296
100% of all-time downloads
Split-Untied-31B
Blazed-Forge/Split-Untied-31B
Split-Untied-31B is a machine learning model from Blazed-Forge. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
<style .sx-shell { position: relative; padding: 10px; border-radius: 14px; background: linear-gradient(140deg, fb718522 0%, transparent 26%, transparent 74%, 7dd3fc22 100%), 0f070a; box-shadow: 0 26px 80px rgba(0,0,0,…
All-time downloads
296
Public
Parameters
32.7B
65.4 GB on disk
Likes
24
Public
Repo makeup
Click a slice to open those files.
.safetensors65.4 GB · 100%
At a glance
- License
- apache-2.0
- Model type
- gemma4
- Access
- Public
- Created
- Sep 6, 2026
- Updated
- Sep 6, 2026
- SHA
- 15ba4d9a
Base models
Model card
From the Hugging Face model README
<style>
.sx-shell {
position: relative; padding: 10px; border-radius: 14px;
background: linear-gradient(140deg, #fb718522 0%, transparent 26%, transparent 74%, #7dd3fc22 100%), #0f070a;
box-shadow: 0 26px 80px rgba(0,0,0,0.72);
}
.sx-main {
position: relative; overflow: hidden; border-radius: 10px; padding: 44px 40px 36px;
background-color: #0b0508;
background-image:
radial-gradient(ellipse 70% 55% at 4% -6%, #fb718526 0%, transparent 60%),
radial-gradient(ellipse 65% 50% at 96% 104%, #7dd3fc22 0%, transparent 60%),
radial-gradient(ellipse 90% 60% at 50% 50%, rgba(255,255,255,0.018) 0%, transparent 70%);
color: #dcc9cf;
font-family: 'Avenir Next','Segoe UI',Roboto,Helvetica,Arial,sans-serif;
line-height: 1.72;
}
.sx-main::before {
content: ""; position: absolute; inset: 0; pointer-events: none; z-index: 1;
background: linear-gradient(180deg, #fb718514 0%, transparent 12%, transparent 88%, #7dd3fc12 100%);
}
/* ---------- hero ---------- */
.sx-hero { text-align: center; position: relative; z-index: 5; }
.sx-title {
font-family: 'Iowan Old Style','Palatino Linotype',Georgia,serif;
font-size: 3.5rem; font-weight: 400; letter-spacing: 13px; margin: 28px 0 16px;
background: linear-gradient(100deg, #fb7185 0%, #fdeaee 52%, #7dd3fc 100%);
-webkit-background-clip: text; background-clip: text;
-webkit-text-fill-color: transparent; color: #fdeaee;
}
.sx-badge {
display: inline-block; color: #fb7185; border: 1px solid #fb718544;
background: #fb718512; text-transform: uppercase; letter-spacing: 4px;
font-size: 0.7rem; font-weight: 600; padding: 7px 20px; border-radius: 999px;
}
.sx-tag { color: #9a8189; font-style: italic; font-size: 1.12rem; margin-top: 16px;
font-family: 'Iowan Old Style','Palatino Linotype',Georgia,serif; }
/* ---------- separators: pill on alternating sides ---------- */
.sx-sep { display: flex; align-items: center; gap: 18px; margin: 46px 0 8px; position: relative; z-index: 5; }
.sx-sep.right { flex-direction: row-reverse; }
.sx-sep b {
font-family: 'Iowan Old Style','Palatino Linotype',Georgia,serif;
font-weight: 400; letter-spacing: 5px; font-size: 0.9rem; text-transform: uppercase;
color: #eef3f9; white-space: nowrap; padding: 6px 20px; border-radius: 999px;
background: rgba(26,11,16,0.80); border: 1px solid #33161f;
}
.sx-sep b em { font-style: normal; color: #fb7185; margin-right: 9px; }
.sx-sep.right b em { color: #7dd3fc; }
.sx-sep i { flex: 1; height: 1px; border: 0; }
.sx-sep i { background: linear-gradient(90deg, transparent, #fb718555 60%, #fb718588); }
.sx-sep.right i { background: linear-gradient(90deg, #7dd3fc88, #7dd3fc55 40%, transparent); }
.sx-lead { font-size: 0.95rem; color: #9a8189; margin: 14px 2px 2px; position: relative; z-index: 5; }
/* ---------- chips ---------- */
.sx-chips { display: grid; grid-template-columns: repeat(auto-fit, minmax(150px,1fr)); gap: 12px; margin: 26px 0 4px; position: relative; z-index: 5; }
.sx-chip {
padding: 15px 14px; text-align: center; border-radius: 10px;
background: rgba(26,11,16,0.80); border: 1px solid #33161f;
font-size: 0.78rem; letter-spacing: 2px; font-weight: 600; color: #eef3f9;
transition: transform .2s ease, box-shadow .2s ease, border-color .2s ease;
}
.sx-chip:nth-child(odd) { border-bottom: 2px solid #fb718566; }
.sx-chip:nth-child(even) { border-bottom: 2px solid #7dd3fc66; }
.sx-chip:hover { transform: translateY(-3px); border-color: #fb718588; box-shadow: 0 10px 26px rgba(0,0,0,0.5); }
/* ---------- cards: asymmetric, side-anchored ---------- */
.sx-card {
position: relative; z-index: 5; margin: 22px 0; padding: 26px 30px;
background: linear-gradient(145deg, rgba(30,13,19,0.90), rgba(11,5,8,0.96));
border: 1px solid #33161f;
box-shadow: 0 12px 34px rgba(0,0,0,0.5);
}
.sx-card.L { border-left: 5px solid #fb7185; border-radius: 4px 20px 20px 4px; margin-right: 26px; }
.sx-card.R { border-right: 5px solid #7dd3fc; border-radius: 20px 4px 4px 20px; margin-left: 26px; }
.sx-card.C { border-top: 3px solid #fb718588; border-radius: 16px; }
.sx-card h3 {
font-family: 'Iowan Old Style','Palatino Linotype',Georgia,serif;
color: #eef3f9; margin: 0 0 12px; letter-spacing: 2px; font-weight: 400; font-size: 1.14rem;
}
.sx-card p { margin: 0 0 11px; font-size: 0.93rem; }
.sx-card p:last-child { margin-bottom: 0; }
.sx-rule { border: 0; height: 1px; margin: 15px 0;
background: linear-gradient(90deg, #fb718555, #33161f 55%, transparent); }
.sx-card.R .sx-rule { background: linear-gradient(90deg, transparent, #33161f 45%, #7dd3fc55); }
/* phase head: medallion + title, mirrored on R cards */
.sx-ph { display: flex; align-items: center; gap: 20px; margin-bottom: 14px; }
.sx-card.R .sx-ph { flex-direction: row-reverse; text-align: right; }
.sx-med {
flex: 0 0 76px; width: 76px; height: 76px; border-radius: 50%;
display: flex; align-items: center; justify-content: center;
font-family: 'Iowan Old Style','Palatino Linotype',Georgia,serif;
font-size: 1.7rem; color: #eef3f9;
background: radial-gradient(circle at 32% 28%, #fb718533, transparent 62%), rgba(26,11,16,0.80);
border: 1px solid #fb718566; box-shadow: 0 0 26px #fb718533, inset 0 0 18px rgba(0,0,0,0.5);
}
.sx-card.R .sx-med { background: radial-gradient(circle at 32% 28%, #7dd3fc33, transparent 62%), rgba(26,11,16,0.80);
border-color: #7dd3fc66; box-shadow: 0 0 26px #7dd3fc33, inset 0 0 18px rgba(0,0,0,0.5); }
.sx-ph h3 { margin: 0; }
.sx-ph small { display: block; color: #9a8189; letter-spacing: 3px; font-size: 0.66rem; text-transform: uppercase; margin-top: 3px; }
/* ---------- donor rows: CSS grid, so both blocks share one column ruler ---------- */
.sx-grid {
display: grid; grid-template-columns: minmax(0,0.85fr) minmax(0,1.15fr) minmax(0,1.6fr);
gap: 1px; background: #33161f; border: 1px solid #33161f; border-radius: 10px; overflow: hidden;
margin: 16px 0 4px; position: relative; z-index: 5; font-size: 0.87rem;
}
.sx-grid > div { background: #0e0609; padding: 11px 14px; color: #c8b6bc; min-width: 0; overflow-wrap: anywhere; }
.sx-grid > .h { background: #170a10; color: #fb7185; letter-spacing: 2px; font-size: 0.66rem; text-transform: uppercase; font-weight: 600; }
.sx-card.R .sx-grid > .h { color: #7dd3fc; }
.sx-grid > div b { color: #eef3f9; }
/* ---------- pick ---------- */
.sx-pick { display: grid; grid-template-columns: repeat(auto-fit, minmax(260px,1fr)); gap: 18px; margin: 20px 0; position: relative; z-index: 5; }
.sx-opt { padding: 22px 24px; border: 1px solid #33161f; background: rgba(26,11,16,0.80); }
.sx-opt.a { border-radius: 4px 20px 20px 4px; border-left: 5px solid #fb7185; }
.sx-opt.b { border-radius: 20px 4px 4px 20px; border-right: 5px solid #7dd3fc; }
.sx-opt .n { display: block; font-family: 'Iowan Old Style','Palatino Linotype',Georgia,serif; font-size: 1.12rem; letter-spacing: 2px; color: #eef3f9; }
.sx-opt .n a { color: #eef3f9 !important; text-decoration: none !important; }
.sx-opt .k { display: block; font-size: 0.64rem; letter-spacing: 3px; text-transform: uppercase; margin: 7px 0 12px; }
.sx-opt .d { font-size: 0.9rem; color: #9a8189; }
.sx-opt.on { box-shadow: 0 0 36px #fb718533; background: linear-gradient(145deg, #fb718514, rgba(26,11,16,0.80)); }
.sx-opt.b.on { box-shadow: 0 0 36px #7dd3fc33; background: linear-gradient(145deg, #7dd3fc14, rgba(26,11,16,0.80)); }
/* ---------- misc ---------- */
.sx-btns { display: flex; gap: 12px; flex-wrap: wrap; margin: 14px 0 2px; position: relative; z-index: 5; }
.sx-btn { display: inline-block; padding: 10px 22px; border-radius: 999px; border: 1px solid #fb718544; color: #fb7185 !important; text-decoration: none !important; letter-spacing: 2px; font-size: 0.76rem; background: rgba(26,11,16,0.80); transition: transform .2s ease; }
.sx-btn:hover { transform: translateY(-2px); box-shadow: 0 8px 22px #fb718533; }
.sx-btn.s { border-color: #7dd3fc44; color: #7dd3fc !important; }
.sx-kv { display: grid; grid-template-columns: minmax(0,1fr) minmax(0,1fr); gap: 1px; background: #33161f; border: 1px solid #33161f; border-radius: 10px; overflow: hidden; margin: 16px 0; position: relative; z-index: 5; }
.sx-kv > div { background: #0e0609; padding: 10px 15px; font-size: 0.87rem; color: #c8b6bc; }
.sx-kv > .v { color: #eef3f9; font-weight: 600; }
.sx-kv > .h { background: #170a10; color: #fb7185; letter-spacing: 2px; font-size: 0.66rem; text-transform: uppercase; }
.sx-det { margin: 4px 0 2px; }
.sx-det summary { cursor: pointer; color: #fb7185; letter-spacing: 2px; padding: 12px 0; font-size: 0.84rem; }
.sx-card.R .sx-det summary { color: #7dd3fc; }
.sx-cred ul { margin: 0; padding-left: 18px; }
.sx-cred li { margin-bottom: 9px; font-size: 0.9rem; }
.sx-foot { text-align: center; letter-spacing: 14px; margin-top: 40px; opacity: 0.45; position: relative; z-index: 5;
background: linear-gradient(90deg, #fb7185, #7dd3fc); -webkit-background-clip: text; background-clip: text; -webkit-text-fill-color: transparent; }
</style>
<div class="sx-shell">
<div class="sx-main">
<div style="font-size: 0.9rem; color: #94a3b8; margin-bottom: 15px; text-align: center;">by Nimbz</div>
<div class="sx-hero">
<video src="https://huggingface.co/Blazed-Forge/Split-Untied-31B/resolve/main/split-31b-header.mp4" poster="https://huggingface.co/Blazed-Forge/Split-Untied-31B/resolve/main/split-31b-header.png" width="100%" autoplay loop muted playsinline style="display:block;border-radius:10px;border:1px solid #fb718544;box-shadow:0 0 70px #fb71852e;margin:0 auto 4px;">
<img src="https://huggingface.co/Blazed-Forge/Split-Untied-31B/resolve/main/split-31b-header.png" width="100%" style="display:block;border-radius:10px;border:1px solid #fb718544;box-shadow:0 0 70px #fb71852e;margin:0 auto 4px;">
</video>
<h1 class="sx-title">SPLIT UNTIED</h1>
<div class="sx-badge">31B · Gemma 4 · StyleSwap Merge</div>
<div class="sx-tag">“Dismiss, and grow.”</div>
</div>
<div class="sx-chips">
<div class="sx-chip">FIVE MODELS</div><div class="sx-chip">THREE PHASES</div><div class="sx-chip">UNTIED HEAD</div><div class="sx-chip">EXPERIMENTAL</div><div class="sx-chip">DARK LEAN</div><div class="sx-chip">ROLEPLAY MERGE</div>
</div>
<div class="sx-sep"><i></i><b><em>◖◗</em>🔵 or 🔴</b><i></i></div>
<div class="sx-lead">Basically the same, they share a brain, yet differ on their speaking vein.</div>
<div class="sx-pick"><div class="sx-opt a"><span class="n"><a href="https://huggingface.co/Nimbz/Split-31B">SPLIT-31B</a></span><span class="k" style="color:#7dd3fc;">◖ the other head</span><span class="d">Pick this one if you want <b style="color:#eef3f9;">instruction adherence</b> for things like trackers or header and footer blocks. Expect a more structured output, and staying on track for long sessions.</span></div><div class="sx-opt b on"><span class="n">SPLIT-UNTIED-31B</span><span class="k" style="color:#fb7185;">◗ you are here</span><span class="d">Pick this one if you want <b style="color:#eef3f9;">more creativity, a darker lean and more lewdness</b>, but only if you're ready to trade a little prompt adherence for it.</span></div></div>
<div class="sx-sep right"><i></i><b><em>⬛</em>GGUF quants</b><i></i></div>
<div class="sx-lead">Thank you to anyone providing quants - mradermacher was diligent!</div>
<div class="sx-btns">
<a class="sx-btn" href="https://huggingface.co/mradermacher/Split-Untied-31B-GGUF">STATIC</a>
<a class="sx-btn s" href="https://huggingface.co/mradermacher/Split-Untied-31B-i1-GGUF">IMATRIX</a>
</div>
<div class="sx-sep"><i></i><b><em>☰</em>The merge</b><i></i></div>
<div class="sx-lead">Two <b style="color:#eef3f9;">della_linear</b> passes over <b style="color:#eef3f9;">gemma-4-31B-it</b>, on <a href="https://github.com/zerofata/mergekit/tree/gemma-4-support">zerofata’s gemma-4-support fork</a>, plus the StyleSwap grafting on top.</div>
<div class="sx-card L"><div class="sx-ph"><div class="sx-med">I</div><div><h3>Comprehension</h3><small>Of The Residual Stream</small></div></div><p>The theory: Adjust <b>q_proj + k_proj</b> for the attention mechanism, and alter <b>gate_proj + up_proj</b> for the knowledge routing.</p><div class="sx-grid"><div class="h">Model</div><div class="h">Purpose</div><div class="h">Scope</div><div><b>Melinoe-VL-heretic</b></div><div>persona tracking, world-state stability</div><div>q/k 0.45 → 0.30 by depth · spine q/k 0.32<br>gate/up 0.28 → 0.18</div><div><b>Glistening-Gem v2.1</b></div><div>format adherence, in-character</div><div>q/k 0.14 → 0.10 · spine 0.0<br>gate/up 0.06</div><div><b>MeroMero v2</b></div><div>swipe variety</div><div>gate/up only · 0.30 · layers 6-47 minus spine</div></div>
<details class="sx-det"><summary>▸ Phase 1: della_linear</summary>
merge_method: della_linear
base_model: ./google/gemma-4-31B-it
dtype: bfloat16
tokenizer_source: base
parameters:
normalize: false
epsilon: 0.05
lambda: 1.0
int8_mask: true
models:
- model: ./bgg1996/Melinoe-Gemma4-31B-VL-heretic
parameters:
density:
- { filter: norm, value: 1.0 }
- { filter: layer_scalar, value: 1.0 }
- { value: 0.85 }
weight:
- { filter: embed_tokens, value: 0.0 }
- { filter: lm_head, value: 0.0 }
- { filter: embed_vision, value: 0.0 }
- { filter: vision_tower, value: 0.0 }
- { filter: multi_modal_projector, value: 0.0 }
- { filter: model.language_model.layers.5.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.5.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.11.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.11.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.17.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.17.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.23.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.23.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.29.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.29.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.35.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.35.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.41.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.41.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.47.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.47.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.53.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.53.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.59.self_attn.q_proj, value: 0.32 }
- { filter: model.language_model.layers.59.self_attn.k_proj, value: 0.32 }
- { filter: model.language_model.layers.0.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.0.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.1.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.1.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.2.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.2.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.3.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.3.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.4.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.4.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.5.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.5.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.6.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.6.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.7.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.7.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.8.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.8.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.9.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.9.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.10.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.10.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.11.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.11.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.12.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.12.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.13.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.13.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.14.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.14.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.15.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.15.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.16.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.16.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.17.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.17.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.18.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.18.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.19.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.19.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.20.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.20.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.21.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.21.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.22.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.22.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.23.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.23.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.24.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.24.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.25.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.25.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.26.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.26.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.27.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.27.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.28.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.28.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.29.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.29.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.30.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.30.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.31.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.31.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.32.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.32.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.33.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.33.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.34.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.34.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.35.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.35.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.36.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.36.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.37.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.37.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.38.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.38.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.39.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.39.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.40.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.40.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.41.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.41.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.42.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.42.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.43.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.43.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.44.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.44.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.45.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.45.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.46.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.46.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.47.self_attn.q_proj, value: 0.45 }
- { filter: model.language_model.layers.47.self_attn.k_proj, value: 0.45 }
- { filter: model.language_model.layers.48.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.48.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.49.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.49.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.50.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.50.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.51.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.51.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.52.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.52.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.53.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.53.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.54.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.54.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.55.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.55.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.56.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.56.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.57.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.57.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.58.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.58.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.59.self_attn.q_proj, value: 0.3 }
- { filter: model.language_model.layers.59.self_attn.k_proj, value: 0.3 }
- { filter: model.language_model.layers.0.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.0.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.1.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.1.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.2.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.2.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.3.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.3.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.4.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.4.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.5.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.5.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.6.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.6.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.7.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.7.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.8.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.8.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.9.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.9.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.10.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.10.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.11.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.11.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.12.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.12.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.13.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.13.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.14.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.14.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.15.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.15.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.16.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.16.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.17.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.17.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.18.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.18.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.19.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.19.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.20.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.20.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.21.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.21.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.22.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.22.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.23.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.23.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.24.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.24.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.25.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.25.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.26.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.26.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.27.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.27.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.28.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.28.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.29.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.29.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.30.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.30.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.31.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.31.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.32.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.32.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.33.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.33.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.34.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.34.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.35.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.35.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.36.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.36.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.37.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.37.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.38.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.38.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.39.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.39.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.40.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.40.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.41.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.41.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.42.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.42.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.43.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.43.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.44.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.44.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.45.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.45.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.46.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.46.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.47.mlp.gate_proj, value: 0.28 }
- { filter: model.language_model.layers.47.mlp.up_proj, value: 0.28 }
- { filter: model.language_model.layers.48.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.48.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.49.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.49.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.50.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.50.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.51.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.51.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.52.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.52.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.53.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.53.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.54.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.54.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.55.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.55.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.56.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.56.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.57.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.57.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.58.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.58.mlp.up_proj, value: 0.18 }
- { filter: model.language_model.layers.59.mlp.gate_proj, value: 0.18 }
- { filter: model.language_model.layers.59.mlp.up_proj, value: 0.18 }
- { value: 0.0 }
- model: ./sophosympatheia/Glistening-Gem-31B-v2.1
parameters:
density:
- { filter: norm, value: 1.0 }
- { filter: layer_scalar, value: 1.0 }
- { value: 0.55 }
weight:
- { filter: embed_tokens, value: 0.0 }
- { filter: lm_head, value: 0.0 }
- { filter: embed_vision, value: 0.0 }
- { filter: vision_tower, value: 0.0 }
- { filter: multi_modal_projector, value: 0.0 }
- { filter: model.language_model.layers.5.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.5.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.11.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.11.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.17.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.17.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.23.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.23.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.29.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.29.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.35.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.35.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.41.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.41.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.47.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.47.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.53.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.53.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.59.self_attn.q_proj, value: 0.0 }
- { filter: model.language_model.layers.59.self_attn.k_proj, value: 0.0 }
- { filter: model.language_model.layers.0.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.0.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.1.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.1.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.2.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.2.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.3.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.3.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.4.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.4.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.5.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.5.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.6.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.6.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.7.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.7.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.8.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.8.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.9.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.9.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.10.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.10.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.11.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.11.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.12.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.12.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.13.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.13.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.14.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.14.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.15.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.15.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.16.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.16.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.17.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.17.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.18.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.18.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.19.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.19.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.20.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.20.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.21.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.21.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.22.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.22.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.23.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.23.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.24.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.24.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.25.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.25.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.26.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.26.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.27.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.27.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.28.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.28.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.29.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.29.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.30.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.30.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.31.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.31.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.32.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.32.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.33.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.33.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.34.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.34.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.35.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.35.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.36.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.36.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.37.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.37.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.38.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.38.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.39.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.39.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.40.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.40.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.41.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.41.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.42.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.42.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.43.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.43.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.44.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.44.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.45.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.45.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.46.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.46.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.47.self_attn.q_proj, value: 0.14 }
- { filter: model.language_model.layers.47.self_attn.k_proj, value: 0.14 }
- { filter: model.language_model.layers.48.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.48.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.49.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.49.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.50.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.50.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.51.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.51.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.52.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.52.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.53.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.53.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.54.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.54.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.55.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.55.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.56.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.56.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.57.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.57.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.58.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.58.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.59.self_attn.q_proj, value: 0.1 }
- { filter: model.language_model.layers.59.self_attn.k_proj, value: 0.1 }
- { filter: model.language_model.layers.0.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.0.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.1.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.1.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.2.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.2.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.3.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.3.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.4.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.4.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.5.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.5.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.6.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.6.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.7.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.7.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.8.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.8.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.9.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.9.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.10.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.10.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.11.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.11.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.12.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.12.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.13.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.13.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.14.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.14.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.15.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.15.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.16.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.16.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.17.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.17.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.18.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.18.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.19.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.19.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.20.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.20.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.21.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.21.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.22.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.22.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.23.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.23.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.24.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.24.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.25.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.25.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.26.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.26.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.27.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.27.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.28.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.28.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.29.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.29.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.30.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.30.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.31.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.31.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.32.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.32.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.33.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.33.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.34.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.34.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.35.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.35.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.36.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.36.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.37.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.37.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.38.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.38.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.39.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.39.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.40.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.40.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.41.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.41.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.42.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.42.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.43.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.43.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.44.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.44.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.45.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.45.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.46.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.46.mlp.up_proj, value: 0.06 }
- { filter: model.language_model.layers.47.mlp.gate_proj, value: 0.06 }
- { filter: model.language_model.layers.47.mlp.up_proj, value: 0.06 }
- { value: 0.0 }
- model: ./zerofata/G4-MeroMero-v2-31B
parameters:
density:
- { filter: norm, value: 1.0 }
- { filter: layer_scalar, value: 1.0 }
- { value: 0.60 }
weight:
- { filter: embed_tokens, value: 0.0 }
- { filter: lm_head, value: 0.0 }
- { filter: embed_vision, value: 0.0 }
- { filter: vision_tower, value: 0.0 }
- { filter: multi_modal_projector, value: 0.0 }
- { filter: model.language_model.layers.5., value: 0.0 }
- { filter: model.language_model.layers.11., value: 0.0 }
- { filter: model.language_model.layers.17., value: 0.0 }
- { filter: model.language_model.layers.23., value: 0.0 }
- { filter: model.language_model.layers.29., value: 0.0 }
- { filter: model.language_model.layers.35., value: 0.0 }
- { filter: model.language_model.layers.41., value: 0.0 }
- { filter: model.language_model.layers.47., value: 0.0 }
- { filter: model.language_model.layers.53., value: 0.0 }
- { filter: model.language_model.layers.59., value: 0.0 }
- { filter: model.language_model.layers.6.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.6.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.7.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.7.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.8.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.8.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.9.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.9.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.10.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.10.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.12.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.12.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.13.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.13.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.14.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.14.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.15.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.15.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.16.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.16.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.18.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.18.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.19.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.19.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.20.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.20.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.21.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.21.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.22.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.22.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.24.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.24.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.25.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.25.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.26.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.26.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.27.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.27.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.28.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.28.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.30.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.30.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.31.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.31.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.32.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.32.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.33.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.33.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.34.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.34.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.36.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.36.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.37.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.37.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.38.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.38.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.39.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.39.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.40.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.40.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.42.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.42.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.43.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.43.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.44.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.44.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.45.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.45.mlp.up_proj, value: 0.3 }
- { filter: model.language_model.layers.46.mlp.gate_proj, value: 0.3 }
- { filter: model.language_model.layers.46.mlp.up_proj, value: 0.3 }
- { value: 0.0 }
merge_method: della_linear
base_model: ./Split-31B-read
dtype: bfloat16
tokenizer_source: base
parameters:
normalize: false
epsilon: 0.05
lambda: 1.0
int8_mask: true
models:
- model: ./ReadyArt/gemma-4-31B-it-scotoma-2
parameters:
density:
- { filter: norm, value: 1.0 }
- { filter: layer_scalar, value: 1.0 }
- { value: 1.0 }
weight:
- { filter: embed_tokens, value: 0.0 }
- { filter: lm_head, value: 0.0 }
- { filter: embed_vision, value: 0.0 }
- { filter: vision_tower, value: 0.0 }
- { filter: multi_modal_projector, value: 0.0 }
- { filter: model.language_model.layers.7.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.7.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.8.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.8.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.9.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.9.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.10.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.10.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.11.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.11.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.12.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.12.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.13.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.13.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.14.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.14.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.15.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.15.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.16.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.16.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.17.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.17.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.18.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.18.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.19.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.19.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.20.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.20.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.21.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.21.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.22.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.22.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.23.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.23.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.24.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.24.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.25.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.25.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.26.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.26.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.27.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.27.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.28.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.28.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.29.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.29.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.30.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.30.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.31.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.31.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.32.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.32.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.33.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.33.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.34.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.34.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.35.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.35.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.36.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.36.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.37.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.37.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.38.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.38.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.39.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.39.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.40.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.40.mlp.down_proj, value: 1.0 }
- { filter: model.language_model.layers.41.self_attn.o_proj, value: 1.0 }
- { filter: model.language_model.layers.41.mlp.down_proj, value: 1.0 }
- { value: 0.0 }
- model: ./Ateron/Gemma-4-Dark-Thoughts-V2-31B
parameters:
density:
- { filter: norm, value: 1.0 }
- { filter: layer_scalar, value: 1.0 }
- { value: 0.75 }
weight:
- { filter: embed_tokens, value: 0.0 }
- { filter: lm_head, value: 0.0 }
- { filter: embed_vision, value: 0.0 }
- { filter: vision_tower, value: 0.0 }
- { filter: multi_modal_projector, value: 0.0 }
- { filter: model.language_model.layers.24.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.24.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.24.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.25.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.25.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.25.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.26.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.26.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.26.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.27.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.27.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.27.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.28.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.28.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.28.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.29.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.29.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.30.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.30.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.30.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.31.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.31.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.31.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.32.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.32.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.32.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.33.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.33.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.33.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.34.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.34.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.34.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.35.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.35.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.36.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.36.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.36.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.37.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.37.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.37.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.38.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.38.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.38.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.39.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.39.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.39.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.40.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.40.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.40.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.41.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.41.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.42.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.42.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.42.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.43.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.43.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.43.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.44.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.44.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.44.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.45.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.45.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.45.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.46.self_attn.v_proj, value: 0.5 }
- { filter: model.language_model.layers.46.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.46.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.47.self_attn.o_proj, value: 0.5 }
- { filter: model.language_model.layers.47.mlp.down_proj, value: 0.5 }
- { filter: model.language_model.layers.48.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.48.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.48.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.49.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.49.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.49.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.50.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.50.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.50.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.51.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.51.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.51.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.52.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.52.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.52.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.53.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.53.mlp.down_proj, value: 0.3 }
- { value: 0.0 }
- model: ./sophosympatheia/Glistening-Gem-31B-v2.0
parameters:
density:
- { filter: norm, value: 1.0 }
- { filter: layer_scalar, value: 1.0 }
- { value: 0.60 }
weight:
- { filter: embed_tokens, value: 0.0 }
- { filter: lm_head, value: 0.0 }
- { filter: embed_vision, value: 0.0 }
- { filter: vision_tower, value: 0.0 }
- { filter: multi_modal_projector, value: 0.0 }
- { filter: model.language_model.layers.12.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.12.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.12.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.13.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.13.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.13.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.14.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.14.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.14.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.15.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.15.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.15.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.16.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.16.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.16.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.17.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.17.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.18.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.18.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.18.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.19.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.19.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.19.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.20.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.20.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.20.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.21.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.21.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.21.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.22.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.22.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.22.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.23.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.23.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.24.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.24.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.24.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.25.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.25.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.25.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.26.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.26.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.26.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.27.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.27.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.27.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.28.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.28.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.28.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.29.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.29.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.30.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.30.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.30.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.31.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.31.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.31.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.32.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.32.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.32.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.33.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.33.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.33.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.34.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.34.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.34.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.35.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.35.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.36.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.36.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.36.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.37.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.37.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.37.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.38.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.38.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.38.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.39.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.39.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.39.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.40.self_attn.v_proj, value: 0.22 }
- { filter: model.language_model.layers.40.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.40.mlp.down_proj, value: 0.22 }
- { filter: model.language_model.layers.41.self_attn.o_proj, value: 0.22 }
- { filter: model.language_model.layers.41.mlp.down_proj, value: 0.22 }
- { value: 0.0 }
- model: ./bgg1996/Melinoe-Gemma4-31B-VL-heretic
parameters:
density:
- { filter: norm, value: 1.0 }
- { filter: layer_scalar, value: 1.0 }
- { value: 0.80 }
weight:
- { filter: embed_tokens, value: 0.0 }
- { filter: lm_head, value: 0.0 }
- { filter: embed_vision, value: 0.0 }
- { filter: vision_tower, value: 0.0 }
- { filter: multi_modal_projector, value: 0.0 }
- { filter: model.language_model.layers.42.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.42.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.42.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.43.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.43.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.43.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.44.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.44.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.44.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.45.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.45.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.45.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.46.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.46.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.46.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.47.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.47.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.48.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.48.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.48.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.49.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.49.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.49.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.50.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.50.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.50.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.51.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.51.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.51.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.52.self_attn.v_proj, value: 0.3 }
- { filter: model.language_model.layers.52.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.52.mlp.down_proj, value: 0.3 }
- { filter: model.language_model.layers.53.self_attn.o_proj, value: 0.3 }
- { filter: model.language_model.layers.53.mlp.down_proj, value: 0.3 }
- { value: 0.0 }
python graft_head.py \
--body ./Split-31B \
--donor ./Gryphe/Pantheon-Reasoning-31B-1.1 \
--t 0.85 \
--out ./Split-Untied-31B \
--clip-q 0.999 \
--protect-chars em_dash
- Type
- gemma4
- License
- apache-2.0
- Created
- Sep 6, 2026
- Updated
- Sep 6, 2026