Feihua-n1-64M-prune

中文版请点击 / Chinese version: README_CN.md

<div align="center">

Feihua-n1-64M-prune: A Quarter of Its Brain Removed, Still Talking Nonsense

A quarter of the brain is gone; not a gram of the nonsense was lost.

</div>

Abstract

Feihua-n1-64M-prune is a structurally pruned variant of the zero-information Chinese model Feihua-n1-64M: guided by ShortGPT-style Block Importance, we removed the two most redundant of its 8 layers (layers 3 and 5), cutting parameters from 63.91M to 49.16M (−23%). Naive pruning followed by IQ2_M quantization collapsed perplexity on the Feihua corpus to 1795 and reduced the output to word salad; a single recovery fine-tuning epoch with the original SFT recipe (79 seconds end to end) brought PPL back to 660.2 and restored full Feihua capability. In a controlled comparison at roughly 21MB, this model (6 layers, ~2.7bit effective precision) beats the unpruned 8-layer 1.56bit extreme-quantization route (938.6) with a PPL of 660.2. The main content of this model remains the content in which its main content mainly consists.

Keywords: Feihua (废话) literature; model pruning; recovery fine-tuning; model quantization; small models; depth vs. precision

1. Introduction and Related Work

1.1 Research question: what happens when you prune a model that is already tiny

The default subject of the model-compression literature is a large model of ten billion parameters or more that carries real capabilities, with the cost of pruning measured in benchmark points lost. A fringe question that has not been systematically answered: when a model is as small as 64M and its sole capability is producing zero-information text, what is left for structural pruning to cut, and what does it actually cut? We answer it on a Feihua model, for three reasons. First, a zero-risk baseline: this model holds no factual knowledge, so the worst outcome of a bad cut is nonsense regressing to garbled output — no capability loss whatsoever. Second, reproducible metrics: perplexity on the Feihua corpus (PPL, lower is better) is logged throughout our local experiment records, the f16 parent baseline is 120.5, and every operation has a before-and-after control. Third, failure itself has comedic value, and comedic value is likewise a recordable experimental phenomenon. Spoiling the results up front: after removing a quarter of the layers, naive pruning did indeed crash, and a 79-second recovery fine-tune did indeed bring it back. For a model that only ever spoke nonsense, what is worth measuring is not how much capability it has left, but how much nonsense it has left.

1.2 Related work

Layer pruning. ShortGPT (arxiv.org/abs/2403.03853) proposed Block Importance (BI) as a layer-importance metric: compare, layer by layer, the magnitude of change in the hidden state before and after removing that layer — the lower the BI, the closer the layer's transform is to an identity mapping, and the cheaper its deletion. The method locates redundant layers without gradients and is the direct source of the scalpel used here; the original paper validated layer deletion plus lightweight recovery on billion-scale models, and we re-pose the question at the 64M scale.

Training framework and base. minimind (github.com/jingyaogong/minimind, Apache-2.0) provides a minimalist Chinese LLM training pipeline that is fully reproducible from tokenizer to RLHF. The parent of this model was trained from its official full_sft_768 base (63.91M parameters, 768 dims × 8 layers, 6400-token vocabulary, Qwen3-aligned architecture); the recovery fine-tune reuses the same framework's SFT script and the original recipe, with no hyperparameter touched.

Parent model. Feihua-n1-64M (https://huggingface.co/ZZRI/Feihua-n1-64M) is the parent of this model — a 64M Chinese Feihua model with zero information content as its explicit optimization target. The full methodology of its data distillation, SFT, GRPO, and ablations is in the main report and is not repeated here; this card only records the part where we took the knife to it.

2. Method

2. Method

2.1 Block Importance measurement

We adopt Block Importance (BI), proposed by ShortGPT (arxiv.org/abs/2403.03853), as the layer-redundancy criterion. For the i-th transformer block, with input hidden state h_in and output hidden state h_out, BI is defined as the expectation of one minus the cosine similarity of the two over a calibration corpus, i.e. E[1 − cos(h_in, h_out)]. Intuition: if a layer's output points in nearly the same direction as its input, the layer approximates an identity mapping and contributes nothing substantial to the information flow — the lower the BI, the more redundant the layer, and the better it suits removal.

The calibration corpus consists of 50 mass-produced incoherent long essays from BullshitGenerator, each truncated to its first 2048 tokens, aligned with the distribution of the model's native genre. All forward computation ran in fp32 on a Tesla P100: this machine has a prior record of logits overflow in fp16 forward passes (see the Pascal hardware pitfalls log in the training report), and the measurement phase takes no such risk.

2.2 Removal and renumbering

Sorted by BI ascending, we removed the two most redundant layers — layers 3 and 5 (BI 0.229 and 0.217) — and renumbered the remaining 6 layers as layers 0–5 in their original order; the embedding, final norm, and lm_head stayed in place. Parameter count dropped from 63.91M to 49.16M (−23%); all other architectural dimensions (dim 768, vocab 6400) are unchanged.

2.3 Recovery fine-tuning

A naively pruned model needs one recovery fine-tune to regain its language ability. The fine-tune reuses the SFT v2 clean dataset of 2,481 distilled Feihua dialogues (19 parroting samples filtered out of 2,500 by quality control), at lr 5e-5 for 1 epoch, taking 79 seconds on a Tesla P100. The recovery cost is low enough to be counted as fixed overhead of the pruning pipeline rather than an independent training task.

3. Experiments

3. Experiments

3.1 Measured layer importance

The measured BI values of the 8 layers are [0.812, 0.281, 0.235, 0.229, 0.259, 0.217, 0.264, 0.370] — a distinctly high-at-both-ends, flat-in-the-middle profile. Layer 0 reaches 0.812, the single most un-cuttable layer in the network; layer 7 is next at 0.370; the middle layers 1–6 all fall in the narrow 0.217–0.281 band, with redundancy close to one another.

Our interpretation of this profile is as follows. Layer 0 performs the first transform from discrete tokens to contextual hidden states: the embedding information of the 6400-token vocabulary must complete its initial mixing in this layer, so its output naturally differs greatly from its input. Layer 7 connects directly to lm_head and is responsible for shaping hidden states into the directions the logits require — likewise far from an identity mapping. The task facing the middle layers, however, is Feihua literature: a distribution with extremely strong stylistic signal and extremely low semantic load. The genre's patterns can be captured well enough in the shallow layers, and after convergence the middle layers gradually degenerate into slight perturbations of the hidden state — hence the collective BI collapse. In other words, the middle layers of this model are not incapable of learning; the Feihua genre simply has very little to learn. Four of the six layers are treading in place, and cutting two merely saves it two steps.

3.2 The crash and the recovery

After naive pruning plus IQ2_M quantization, PPL jumped from the f16 parent's 120.5 to 1795. The output at this point is typical word salad: the vocabulary is still the high-frequency words of the Feihua corpus, but syntactic connection is completely broken — Feihua skeleton words such as "although" (虽然), "but" (但是), and "that is to say" (也就是说) pile up at random, adjacent clauses no longer form any collocational relation, and the whole thing reads like a basket of nonsense poured into a paper shredder.

After the 79-second recovery fine-tune described in §2.3, PPL under IQ2_M fell back to 660.2. Generated samples recovered complete conditional-complex-sentence syntax: the "although … but …" construction appears in pairs again, transitions between sentences are smooth, and the content remains utterly empty — Feihua capability is restored in the functional sense. The gap to the parent's 120.5 is mostly contributed by IQ2_M bit-width loss (the parent alone at IQ2_M already reaches 191.8); the marginal damage caused by pruning itself is limited.

3.3 Quantization matrix

The size and PPL (Feihua corpus, lower is better) of the recovered 6-layer pruned model at seven quantization levels are listed below; all GGUF files are collected in the Feihua-n1-64M-prune-GGUF repository.

The output at IQ1_S has left the category of prose entirely and presents a fractured verse-like form. A real sample: "This machine, is that one called called what. You ask ask, it has a name ask? … So, it is called Kou." From the standpoint of quantization engineering this is complete failure; from the standpoint of Feihua literature it is a genre transgression: compressed to the point where it can no longer finish a sentence of nonsense, the model instead uttered a few lines that almost sound sensible.

3.4 Deathmatch at equal size

Three compression routes constrained to the same ~21MB budget, compared head-to-head:

There are two conclusions. First, depth is worth more than precision: cutting a quarter of the layers while keeping 2.7bit precision beats, across the board, keeping all 8 layers squeezed to 1.56bit — at the 64M scale, the representational depth carried by layer count is less expendable than weight bit-width. Second, recovery fine-tuning is worth more than extreme bit-width: at the same 6 layers @2.7bit, a 79-second fine-tune pulls PPL from 1795.4 to 660.2, a payoff far beyond any single quantization choice — and it is only a 79-second fine-tune, the step with the highest return per unit time in this entire compression pipeline; after all, all it does is teach the model to say once more the things it already said.

3.5 Serious benchmark evaluation

We also evaluated this model on the eight benchmarks of the official Spark-X2.5 model card (the full protocol and all seven reference systems are in §3.5 of the main repository's research report; only this model's scores are listed here):

The total across the eight is 2.78 points, of which 2.5 come from IFBench, where parroting the English source text happened to satisfy the constraint count — that is not instruction following, it is vomit that matched the format. One observable difference between the pruned variant and the main model: the main model's zero-score answers are mostly grammatically fluent Chinese nonsense, whereas about half of this model's zero-score answers degrade into mixed Chinese-English gibberish (e.g. "Janjouria 的 Janjourney"). Pruning removed a quarter of the Feihua organ, and the grammatical integrity of the nonsense suffered accordingly — but the zero score was unaffected. This also counts as a rigorous verification of the "law of conservation of capability": what was never there will not be one point less.

4. Qualitative Analysis and Statements

4.1 Qualitative analysis

On real hardware (Tesla P100 + GTX 1080), using the recovered IQ2_M tier (21MB, PPL 660.2), we asked the model "What is machine learning?" and received the following complete output:

"Machine learning is the use of machine-learning methods to solve those datafication problems that require machine learning and machine learning. If machine learning and machine learning are not machine learning, then it is not machine learning."

This output deserves sentence-by-sentence dissection. The skeleton of the circular argument is complete: the definition uses the definiendum itself as evidence, the conditional "if … then it is not …" forms a textbook Feihua closing, and the claim–evidence–loop three-part structure is all present; the full text carries no world knowledge whatsoever, and its information content is precisely zero. This shows that after removing 2 of the 8 layers and reducing parameters from 63.91M to 49.16M (−23%), the core voice of Feihua literature is unharmed. Flaws are also visible: "machine learning and machine learning" shows slight repetitive stacking, which can be regarded as a kind of stutter. The phenomenon concentrates at the 21MB extreme tier and is an occupational injury of size compression, consistent with the degradation direction of PPL rising from the main model's 191.8 at the same tier to 660.2; at the IQ4_XS tier (27MB, PPL 486.1) it is no longer perceptible.

4.2 Limitations and ethics

This model fully inherits the zero-information property of the main model: no output contains any fact, and it must not be used for Q&A, retrieval, writing assistance, or any scenario that demands "correctness". After pruning and recovery there is a slight stutter, expressed as an increased probability of exact clause repetition — worst at extreme quantization tiers — which is a known and reproducible degradation.

The ethics statement has three sentences. This model does not spread rumors, because it never holds any fact that could be distorted; it does not leak secrets, because there are no secrets in its training data; it carries no harmful content, because the greatest harm it can do to a reader is wasting a few seconds. Zero information content is this model's only safety claim, and also its most honest sentence.

Acknowledgments and License

The pruning methodology comes from ShortGPT (arxiv.org/abs/2403.03853), the training framework from minimind (github.com/jingyaogong/minimind, Apache-2.0), and the mass-produced corpus from BullshitGenerator (github.com/menzi11/BullshitGenerator). All pretraining, SFT, GRPO, pruning, and quantization work was done on two Pascal GPUs from 2016 (a Tesla P100 and a GTX 1080). This model and the main model are both released under Apache-2.0, free to use and redistribute.

Citation

@misc{feihua-n1-64m-prune,
  title        = {Feihua-n1-64M-prune: A 6-Layer Pruned Nonsense-Literature Language Model},
  author       = {ZZRI},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/ZZRI/Feihua-n1-64M-prune}},
  note         = {大脑少了四分之一，废话一克都没少}
}

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ZZRI/Feihua-n1-64M-prune"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

inputs = tokenizer("什么是机器学习？", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128, do_sample=True, temperature=0.8)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

One-command llama.cpp run (recommended sweet-spot tier IQ4_XS):

llama-server -hf ZZRI/Feihua-n1-64M-prune-GGUF:feihua-v6-prune6-heal-IQ4_XS

The full 8-layer parent (63.91M, same Feihua purity as this model) is in the main repository: https://huggingface.co/ZZRI/Feihua-n1-64M .