Qwen3.8-20B-Minitron

Qwen3.8-20B-Minitron

An experimental structurally-pruned derivative of Qwen/Qwen3.8-27B.

This is not a quantization of the 27B model.

The language model was physically reduced from 64 to 44 decoder layers using iterative marginal-KL group pruning, followed by knowledge-distillation recovery and targeted repair training.

Architecture

Approximately 7.61 billion parameters were physically removed.

Original language-layer pattern:

[linear_attention, linear_attention, linear_attention, full_attention] × 16

Surviving groups:

[0, 2, 4, 6, 7, 8, 11, 12, 13, 14, 15]

Removed groups:

[1, 3, 5, 9, 10]

The surviving layers retain their original Qwen weights and are reindexed after pruning.

Recovery

The raw 19.746B child remained coherent but showed noticeable reasoning and factual instability.

Recovery was performed in multiple stages using:

• original Qwen3.8-27B teacher logits
• top-k knowledge distillation
• next-token training
• mixed educational/instruction/math/code data
• targeted repair examples
• custom LoRA recovery
• final LoRA merge into the standalone weights

The uploaded checkpoint contains the fully merged weights. No adapter or pruning script is required to load it.

Internal release checks

A custom held-out evaluation used during release selection scored:

• overall accuracy: 81.2%
• arithmetic: 100%
• logic: 100%
• money/Dutch: 83%
• probability: 80%
• Python: 75%
• strict instruction following: 83%
• sequence/pattern reasoning: 20%

These are custom internal checks, not standardized benchmark scores, and should not be compared directly with official Qwen benchmark results.

The final legacy regression suite scored 11/12 (91.7%).

Known limitations

This is an experimental compressed model.

In particular:

• pattern/sequence reasoning remains a weakness
• some unusual wording can still produce incorrect reasoning
• pruning may have removed capabilities not represented by our tests
• recovery and evaluation focused primarily on text
• the vision tower is retained from the parent model but was not the focus of the recovery process
• this model should not be assumed to match Qwen3.8-27B quality

Usage

import torch
from transformers import AutoModelForMultimodalLM, AutoTokenizer

model_id = "exnivo/Qwen3.8-20B-Minitron"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "user", "content": "What is the capital of Australia?"}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(
    text,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=128,
    )

print(
    tokenizer.decode(
        output[0][inputs.input_ids.shape[1]:],
        skip_special_tokens=True,
    )
)

Method

The pruning process did not rank all layers once and then remove them simultaneously.

Instead, after every group removal, candidate importance was measured again on the current pruned model using marginal KL divergence.

This mattered because layer importance changed substantially after earlier groups were removed.

The process was:

1. start with 16 four-layer hybrid groups
2. protect boundary groups
3. temporarily bypass each candidate group
4. measure marginal KL divergence
5. permanently remove the least disruptive group
6. recompute all candidate scores
7. repeat until 11 groups remained
8. recover the resulting model using distillation
9. run targeted repair training
10. merge recovery weights into this standalone checkpoint

Attribution

Based on Qwen/Qwen3.8-27B.

This is an independent experimental derivative and is not an official Qwen release.