qwen3-5-4b-mlp7808-distilled

Qwen3.5-4B — Wanda-Pruned + Distilled (MLP 9216→7808), Sharded

<!-- Provide a quick summary of what the model is/does. -->

A structurally MLP-pruned and knowledge-distilled derivative of Qwen/Qwen3.5-4B, saved as 16 safetensors shards (~ 7.9 GB, bf16). The SwiGLU MLP intermediate dimension is reduced 9216 → 7808 (~15.3% MLP pruning, a 128-multiple for kernel alignment) via Wanda importance scoring, then recovered with logit-level knowledge distillation from the original bf16 model (and a LoRA recovery pass).

⚠️ Research artifact. Pruning measurably reduced instruction-following ability (IFEval ~0.87 → ~0.71; see Limitations). Best used as a smaller, faster base for further fine-tuning — not as a faithful drop-in replacement for the parent on instruction- or format-strict tasks.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

This checkpoint takes the hybrid (linear-attention + full-attention) Qwen3.5-4B, narrows every SwiGLU MLP block's intermediate width from 9216 to 7808 by keeping the top-scoring channels under the Wanda criterion (|W| · ‖X‖₂), and re-trains the slimmed model to track the original via knowledge distillation. Attention layers, the embedding, and the tied LM head are left unpruned.

• Developed by: [Suraj Sharan]
• Funded by [optional]: [N/A]
• Shared by [optional]: [Suraj Sharan]
• Model type: Decoder-only causal LM, hybrid attention (linear + full), structurally pruned + distilled
• Language(s) (NLP): English (pruning/distillation data was English; other languages inherited from the parent and untested here)
• License: Inherits the Qwen3.5-4B license as a derivative (set to the base model's terms; do not relicense to MIT without confirming compatibility)
• Finetuned from model [optional]: Qwen/Qwen3.5-4B

Model Sources [optional]

<!-- Provide the basic links for the model. -->

• Repository: [More Information Needed]
• Paper [optional]: N/A
• Demo [optional]: N/A

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

General-purpose text generation and research on structured pruning + distillation of small hybrid LLMs.

Downstream Use [optional]

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

A lighter base for task-specific fine-tuning. A short instruction-following SFT/DPO pass is recommended before assistant-style use to recover the pruning regression.

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

Not a faithful substitute for Qwen3.5-4B on strict instruction-following, exact-format output, or safety-critical applications, given the measured IFEval regression.

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

• Instruction-following regression (key limitation): on IFEval (inst_level_strict_acc), the original Qwen3.5-4B scores ≈0.87 while this pruned+distilled checkpoint scores ≈0.71. MLP pruning removed capacity that instruction-following relies on, and generic-corpus distillation did not recover it.
• Inherited biases: carries the biases, knowledge cutoff, and safety profile of the parent.
• Generic fidelity ≠ task parity: ~98% next-token argmax agreement with the teacher does not imply parity on instruction- or reasoning-heavy benchmarks.

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Run an instruction-following SFT (or DPO) pass before assistant-style deployment, and evaluate on your own target tasks rather than relying on the parent's reported numbers.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration
import torch

model_id = "your-username/your-repo"  # or local path to this folder
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = Qwen3_5ForConditionalGeneration.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, trust_remote_code=True
).eval().cuda()

msgs = [{"role": "user", "content": "Explain Wanda pruning in two sentences."}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").cuda()
out = model.generate(inputs, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

yahma/alpaca-cleaned (open instruction corpus) served as both the Wanda calibration source and the distillation corpus. No proprietary or private data was used.

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Preprocessing [optional]

Calibration activations were collected over the corpus to compute per-channel Wanda scores (|W| · ‖X‖₂) for each MLP intermediate channel; sequences truncated to the training sequence length.

Training Hyperparameters

• Training regime: bf16 mixed precision
• Pruning: Wanda scoring → MLP intermediate 9216 → 7808 (gate/up output rows + down-proj input columns pruned consistently; attention, embeddings, LM head untouched)
• Distillation objective: KL(student ‖ frozen bf16 teacher) over full-vocabulary logits, computed in fp32, chunked over the sequence dimension
• Distillation steps: ~12,000
• LoRA recovery: rank 32 on gate/up/down + q/k/v/o projections, distilled then merged

Speeds, Sizes, Times [optional]

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

• Parameters: ~4B (MLP intermediate slimmed ~15% vs parent)
• Format: bf16, 16 safetensors shards (~7.9 GB total)
• Teacher–student argmax agreement: ~98% on the distillation corpus

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

IFEval prompt set; the distillation corpus for argmax-agreement checks.

Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

Generic next-token fidelity vs instruction-following capability.

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

• Argmax agreement vs the bf16 teacher (generic fidelity)
• IFEval inst_level_strict_acc (instruction-following)

Results

Summary

High generic-text fidelity to the parent, with a clear instruction-following regression introduced by MLP pruning. Treat as a research checkpoint / fine-tuning base.

Model Examination [optional]

<!-- Relevant interpretability work for the model goes here -->

Pruning was concentrated in the SwiGLU MLP intermediate dimension; attention (linear + full), embeddings, and the tied LM head were preserved.

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: NVIDIA H100 (80GB)
• Hours used: [More Information Needed]
• Cloud Provider: [More Information Needed]
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

Hybrid Qwen3.5-4B backbone (Qwen3_5ForConditionalGeneration): 32 decoder layers (mix of linear-attention and full-attention), hidden size 2560, MLP intermediate 7808 (pruned from 9216), grouped-query attention (16 query / 4 KV heads, head_dim 256), vocabulary 248,320, tied input/output embeddings. Objective: knowledge distillation toward the bf16 parent.

Compute Infrastructure

Hardware

NVIDIA H100 (80GB), multi-GPU data-parallel.

Software

PyTorch, Hugging Face Transformers, PEFT, Accelerate.

Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

@misc{qwen3_5,
  title  = {Qwen3.5},
  author = {Qwen Team},
  year   = {2025}
}

APA:

Qwen Team. (2025). Qwen3.5. (Base model for this pruned + distilled derivative.)

Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

• Wanda: pruning by weight magnitude × input activation norm (|W| · ‖X‖₂).
• MLP intermediate dimension: the inner width of the SwiGLU feed-forward block, reduced here from 9216 to 7808.
• Argmax agreement: fraction of positions where this model's top token matches the teacher's.

More Information [optional]

Pruning targeted only the MLP intermediate width; for stronger instruction-following, fine-tune downstream.

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]