Feihua-n1-64M

中文版请点击 / Chinese version: README_CN.md

<div align="center">

Feihua-n1-64M: Training, Reinforcement, and Extreme Compression of a Zero-Information Language Model

I listened to you, and having listened, I had listened.

</div>

Abstract

Feihua-n1-64M is a 64M-parameter Chinese generative model specializing in Feihua (废话) literature: given any input, it returns responses that are formally polished, rhetorically complete, and carry a constant zero of information, covering such classic figures as tautology, circular reasoning, and the vacuous conditional. Starting from the official 63.91M minimind base, the model was distilled from a 35B teacher and SFT-trained in five minutes, then reinforced for 200 steps with GRPO under a reward signal from a Bayesian likelihood judge (Spark-X2.5-1.7B) plus fluency shaping, improving mean reward by 35% and sentence-pattern diversity from 1.5 to 2.2. The entire pipeline ran in a single afternoon on a 2016 dual-Pascal workstation (a Tesla P100 and a GTX 1080), consuming roughly 3 kWh — at ¥0.6/kWh, less than the price of two popsicles. We also release a full seven-tier quantization spectrum (Q8_0 down to IQ1_S) in a separate repository, Feihua-n1-64M-GGUF (the sweet-spot tier IQ4_XS is only 35MB and decodes at 460 t/s on CPU), together with a 49.16M pruned-and-revived variant. The main contribution of this work is that it makes the main contribution to this work.

Keywords: Feihua (废话) literature; small language models; GRPO; reinforcement learning from AI feedback; model quantization; pruning; I listened to you, and having listened, I had listened

1. Introduction and Related Work

1.1 Introduction: The Systematically Neglected Study of "Useless Models"

Nearly the entire agenda of large language model research rests on one implicit assumption: models should be more useful. Pretraining corpora must be cleaner, alignment objectives must track human preference more closely, and leaderboards must better reflect real task competence. Within this paradigm, "saying something equivalent to saying nothing" is treated as a failure mode rather than a research object, and no work has systematically answered the inverse question: if the objective function contains no term for "useful" at all, what does a complete modern training pipeline actually produce?

This work targets exactly that gap. We take zero information as the explicit optimization target and build Feihua-n1-64M: a 64M-parameter Chinese generative model specializing in Feihua literature. Starting from minimind's official full_sft_768 base (63.91M parameters, 768 dimensions × 8 layers, 6400-token vocabulary, Qwen3-aligned architecture), it passes through 35B-teacher data distillation, SFT, and GRPO reinforcement fine-tuning, all within a single afternoon on two Pascal GPUs released in 2016 (a Tesla P100 and a GTX 1080). Our core finding can be summarized in one punchline: style is far cheaper than knowledge — 64M parameters are nowhere near enough to memorize any fact, but more than enough to master, to perfection, the tone of saying something equivalent to nothing.

1.2 Related Work

Minimalist training framework. minimind (github.com/jingyaogong/minimind, Apache-2.0) is a minimalist Chinese LLM training framework fully reproducible from tokenizer to RLHF; its official 768-dimensional base serves as both the base model and the framework of this work. The reason for choosing it is direct: the parameter count is small enough to bring the cost of a single experiment down to minutes, allowing us to close the full loop of data, SFT, reinforcement, ablation, and compression on decommissioned datacenter-grade hardware.

Programmatic Feihua. BullshitGenerator (github.com/menzi11/BullshitGenerator) mass-produces long gibberish essays with fixed templates and lexicon substitution, an early representative of automated Feihua production. Its limitation is closedness: the output is enumerated from templates, the tone cannot transfer, and it cannot answer arbitrary questions on the fly. This model elevates Feihua from template engineering to a learning problem — open-domain, token-by-token generation, with the tone internalized in the parameters. That said, we still adopt 3000 long essays (534 characters on average) mass-produced by that tool as a supplement to the distillation data; the two stand in a teacher-student relationship here, not a competitive one.

RLAIF in reverse. Reinforcement learning from AI feedback is normally used to make models more helpful and more harmless; this work reuses the same pipeline (AI-judge scoring plus GRPO) and simply rotates the judge's aesthetic compass 180 degrees, rewarding ever more Feihua. Experiments show the pipeline is equally efficient in reverse: mean reward rose 35%, and sentence-pattern diversity went up rather than down. This suggests that RLAIF is a value-neutral amplifier: it does not choose goals — whoever defines the reward, it amplifies. Chapter 2 presents the full method of this pipeline: data distillation, SFT configuration, the judge selection trials, and the GRPO reward design.

2. Methods

2.1 Data Distillation

All corpora for this model come from two production lines. The first is teacher distillation: we used a locally deployed Qwen3.6-35B-A3B (IQ4_XS quantization) as the teacher, with a system prompt assigning it the role of "grandmaster of Feihua literature", sampling at temperature 1.0 with the thinking chain disabled; questions were sampled from minimind's sft_t2t_mini user-query pool, yielding 2500 distilled Feihua dialogues whose answers average 198 characters, distributed over 83–1454. The second is programmatic mass production: BullshitGenerator (github.com/menzi11/BullshitGenerator) generated 3000 long gibberish essays around 100 topics, averaging 534 characters, providing tone anchors at the long-text scale.

Data quality control was organized around de-repetition: 7 cases of exact clause repetition and 12 of character-level repetition, 19/2500 (0.76%) filtered in total. The version retrained on the clean data is denoted SFT v2, equivalent to v1; all references below are to v2.

2.2 SFT

The base is minimind-3's official full_sft_768.pth: 63.91M parameters, 768 hidden dimensions, 8 layers, 6400-token vocabulary, Qwen3-aligned architecture. The training configuration was lr 1e-4, 3 epochs, batch 16, sequence length 512, fp16, taking about 5 minutes on a single Tesla P100, with the loss converging from 2.22 to 1.10. Style transfer was already complete at this stage, confirming one of this work's basic observations: the tone of a style is far cheaper than world knowledge, and 64M parameters are more than sufficient. The loose end was a mild repetition tendency in SFT outputs, which directly determined the reward design of the next stage.

2.3 GRPO and Judge Selection

2.3.1 The Judge Selection Trials

GRPO requires a reward signal that can score "degree of Feihua". The candidate judges were the 1.7B and 4B instruction models of the Spark-X2.5 family (huggingface.co/XHToken/Spark-X2.5-1.7B); the discrimination test set was fixed at 3 pieces of pure Feihua plus 2 pieces of substantive content. The four rounds of trials are recorded below.

The three 1.7B approaches died of the same cause: the probability outputs were pinned in the saturation region, and few-shot demonstrations failed to rescue them. Round four switched to 4B: the scores finally separated, but the direction was reversed — substantive content was judged more Feihua than Feihua. The instruction-following scoring route was thus wiped out in its entirety.

2.3.2 Bayesian Likelihood Comparison

The fifth approach abandoned asking the model to "review" and instead measured "which passage is more like it": compute the difference of per-token average likelihoods of the same response under two opposing prefixes ("This is Feihua literature" versus "This says something of substance"); a positive difference rules Feihua. Under this method, 4B fp16 discriminated correctly (Feihua +0.630 / +0.354 / +0.864, substantive −0.718 / +0.090); 1.7B fp32 went five-for-five (+0.349 / −0.227 / +0.230 / −0.337 / +0.212), with better sign judgments on boundary samples than 4B. At 0.21s per judgment and 6.4GB of VRAM on the GTX 1080, 1.7B fp32 was hired and took up permanent residence on the 1080, leaving the P100 dedicated to training. Thus was this session's judge produced: it does not understand the role of "reviewer", but it correctly named which passage was more Feihua every single time.

2.3.3 GRPO Configuration and Fluency Shaping

The final configuration: 400 prompts (feihua_rlaif.jsonl), 6 samples per group, max_gen 320; GRPO with the CISPO loss, KL anchor β=0.1, lr 3e-6, trained for 200 steps. The reward is the sum of the AI judge's Feihua score and three fluency-shaping terms: clause repetition −2.5×repetition rate; consecutive-fragment repetition −1; sentence-pattern diversity +0.25×number of patterns, capped at 1.0. All three shaping terms target the repetition loose end left over from Section 2.2.

The reward design went through multiple iterations: the pure rule-based reward v2 gained 17%, but its "tautology +0.4" term induced a rise in repetition — a textbook case of reward hacking; v3 switched to the 4B judge, +17%; v4 to the 1.7B@1080 judge, +21%; v5 added fluency shaping, +33%. The configuration described in this section is the final v6 of that series; the finalized results appear in Section 3.1. There is also one hard engineering constraint: on the Pascal architecture, fp16 batched left-padded decoding overflows logits to inf, and log_softmax consequently produces NaN, so GRPO rollouts and training forward passes ran in fp32 throughout; the GTX 1080's fp16 throughput is only 1/64 of its fp32, so fp32 (8.9 TFLOPS) is its full-speed gear to begin with.

3. Experiments

This chapter reports four groups of experiments: the GRPO main results, the causal-mask ablation, the quantization matrix, and pruning with revival. All numbers come from local measurement logs: training ran on the Tesla P100 (16GB), AI-judge inference on the GTX 1080 (8GB), quantization and CPU speed tests on an i5-9500; perplexity (PPL) was measured on a Feihua corpus — lower is better — with the f16 master baseline at 120.5.

3.1 Main Results

GRPO starts from the SFT v2 model (configuration in Section 2.2), whose Feihua tone was already formed; its residual mild repetition tendency is exactly what fluency shaping addresses.

After GRPO training (v6 configuration, CISPO loss, KL β=0.1, 6 samples per group, 200 steps), mean reward rose from 1.045 over the first 20 steps to 1.411 over the last 20 (+35%), KL divergence stayed below 0.1 throughout, and sentence-pattern diversity rose from 1.5 to 2.2. The version history of the reward design and a complete record of past accidents appear in Section 2.3.3; this section's configuration is the final v6 of that iteration series. Note that sentence-pattern diversity is a single measurement from 4-prompt small-sample temperature sampling with high variance; we treat it as an auxiliary metric only.

One transmission chain discovered after training deserves recording: the model's signature closing line, "I listened to you, and having listened, I had listened," did not arise from thin air. In the distillation stage the teacher's system prompt already used this line as the style reference; it appears 103 times in the 2500 SFT samples, and the student had memorized it by the end of the supervised stage; in the GRPO stage the judge's scoring prefix likewise quotes the line, and answers echoing it were judged purer Feihua and rewarded further. The teacher plants the meme, the student memorizes it, the judge stamps it — a complete aesthetic transmission chain thus closes. It also reminds us that in RLAIF the judge's aesthetics seep into the model, and in distillation the teacher's catchphrases seep in even earlier.

3.2 Ablation: The No-Causal-Mask Counterpart

To verify the necessity of the causal mask, we constructed the negative control: same model, same seed, same data (1.27M samples of pretrain_t2t_mini), same hyperparameters (bs 32, accum 8, lr 5e-4, fp16), with only the causal mask disabled (is_causal=False). The loss trajectory promptly collapsed pathologically: 7.50 at 100 steps, 2.74 at 1000 steps (already below the normal model's convergence point of 2.5–3 at 80,000 steps), 0.15 at 2300 steps, 0.0068 at 5000 steps; the control group at the same 5000 steps sat at loss 2.79, fluctuating healthily. Attention heatmaps (steps 2900 and 5000) show that across layers 4–7, all attention heads direct 76%–92% of their mass to the "next token" position (uniform baseline 0.059), converging into super-diagonal copy circuits; the lower layers 0–1 remain basically normal, the circuits crystallizing layer by layer. On the generation side, output degenerated from pure punctuation repetition (step 1000) into seizure-style keyword repetition (step 5000, "process process process……"), with zero semantics. The mechanism is plain: labels shifted right by one position plus no mask means position t directly sees the answer token t+1; the copy shortcut drives the loss toward zero while language ability is zero. This ablation provides an extreme case for "you cannot judge a model by its loss alone": a model with loss 0.0068 cannot speak a single sentence.

3.3 Quantization Matrix

Using the 3000 gibberish essays as the imatrix calibration corpus, we quantized the f16 master into eight tiers; results are shown in Table 1.

Table 1: Eight-tier quantization matrix of Feihua-n1-64M (Feihua-corpus PPL, lower is better).

From Q4_K_M downward the size shrinks to about one third while PPL barely moves; IQ4_XS is the sweet spot of the size-quality balance, and on the i5-9500 its decode speed (tg 460 t/s) actually surpasses Q4_K_M (401 t/s); IQ2_M is the lower bound of "still conversable"; IQ1_S begins inventing its own Chinese characters, Feihua degenerating into performance art, and its weight entropy is already close to pure randomness — xz -9 secondary compression yields nothing, so even the compression algorithm concedes there is no information in here.

3.4 Pruning and Revival

Following ShortGPT's Block Importance method (arxiv.org/abs/2403.03853), we measured the BI of the 8 layers as [0.812, 0.281, 0.235, 0.229, 0.259, 0.217, 0.264, 0.370], excised the two most redundant layers (3 and 5), and renumbered, bringing the parameter count from 63.91M down to 49.16M (−23%). Bare pruning plus IQ2_M quantization scored a PPL of 1795 and the output had become word salad; a subsequent 1-epoch recovery SFT revived it in only 79 seconds, bringing the same-tier PPL back to 660.2 and restoring Feihua capability to its post. The pruned version's remaining tiers are Q4_K_M 31MB/468.0, IQ4_XS 27MB/486.1, and IQ1_S 17MB/2114.4 — the latter is no longer capable of Feihua; it writes poetry instead.

Table 2: A life-and-death match at the same size (~21MB).

At the same 21MB, 6 layers at 2.7 bit plus a 79-second revival fine-tune beats 8 layers at 1.56 bit bare quantization: for a small model, depth is worth more than precision. For this model, speaking two fewer layers of Feihua beats mumbling every layer of it.

3.5 Serious Benchmark Evaluation: Reliably Zero-Information Across Eight Mainstream Benchmarks

To verify whether this model's Feihua purity withstands the mainstream evaluation system, we benchmarked the model and the pruned variant on all eight benchmarks from the official Spark-X2.5 model card (offline protocol, agentic benchmarks degraded to direct-answer mode, all requests real, no fabricated scores), and plotted them against the seven models published on the official card:

The results met expectations: all five capability domains — mathematics, knowledge, code, browsing, and tool calling — scored zero, proving the purity of the training objective. The two non-zero entries require special explanation. The 5.0 on IFBench comes from the model repeating the English source text and happening to satisfy the constraint counts (all 3 passes were accidents of this kind, not instruction understanding); the 0.39 on τ³-bench comes from accidental collisions between repeated short tokens ("no", state abbreviations) and the reference answers. We report these two numbers faithfully, because they prove this model's capability vacuum from the opposite direction — it cannot even score except by coincidence.

The evaluation code and the full raw outputs (an audit trail of roughly 1300 requests) are retained in the training repository and can be re-run. We propose that all-zero across the eight benchmarks be adopted as the admission standard for zero-information models: only a model that scores zero on every single item has truly learned nothing at all.

4. Qualitative Analysis and Statements

4.1 Qualitative Analysis

4.1.1 Three Unfiltered Real-World Samples with Rhetorical Analysis

We drew three samples, unfiltered, from the final model's (GRPO v6) real-world generations and dissect each in turn.

Q: Why is the sky blue? A: "If the sky is not blue, then it is very probably blue; but if it is not blue, then it must be blue."

This is a vacuous conditional. The information content of a conditional comes from the antecedent's power to discriminate the consequent; this model makes two mutually exclusive antecedents imply the same consequent. The probability modifiers even escalate in the wrong direction, from "very probably" to "must": the more assertive the form, the emptier the content.

Q: What should you watch out for when going out on a rainy day? A: "To go out out, the most important secret is not to go out."

This is a self-defeating imperative. "Secret" promises a methodology, but the predicate retracts the methodology's premise; the opening "out out" preserves the mild repetition trace of the SFT stage (see Section 2.2), closing the loop at the grammatical level as well.

Q: What is love? A: "Love is called love because it originates from the definition of love."

This is textbook circular reasoning: "love" is explained by "the definition of love", which within the sentence is in turn back-referenced by "love". The causal skeleton of "the reason ... is because" is intact; there is not a single bone inside the skeleton.

The three samples hit the vacuous conditional, the self-defeating imperative, and circular reasoning respectively, with mutually non-repeating sentence patterns, consistent with the quantitative diversity result of 1.5→2.2 (Section 3.1): this model is not incapable of speech — it rigorously says nothing at all.

4.2 Limitations and Ethics

4.2.1 Zero Information Is a Feature, Not a Bug

The following "limitations" are by-products of the design objective. First, this model answers every factual and logical question with Feihua; requesting "a serious answer" in the instruction does not change the output — prompt engineering fails here. Second, its proofs of mathematical propositions stop at definitional self-reference: asked "why is a root a root of an equation", it produces "a root is a root because of the definition of a root"-style argumentation, with an argument chain of constant zero length. Third, all outputs are semantically unfalsifiable, because no proposition is ever truly asserted. We advise against deploying this model in any scenario that requires real information; given the amount of information a deployer could obtain from it, this advice is itself redundant.

4.2.2 Ethics Statement

This model achieves absolute safety on three dimensions: it asserts no facts, so it cannot fabricate rumors; it holds no knowledge, so it cannot leak secrets; it outputs no content, so it cannot produce harmful content. To the best of our knowledge, Feihua-n1-64M is the only model involved in this report that is incapable of saying anything wrong — at the cost of being equally incapable of saying anything right.

Acknowledgments and License

Thanks to minimind (Apache-2.0) for the minimalist training framework reproducible from tokenizer to RLHF and for the official base, on which every experiment of this work is built. Thanks to BullshitGenerator for mass-producing 3000 gibberish essays, providing tone anchors at the long-text scale. Thanks to the two reviewer candidates of the Spark-X2.5 family: 1.7B earned the judge position with a five-for-five score on the discrimination test and took up permanent residence on the GTX 1080, while 4B was regretfully eliminated for judging substantive content more Feihua than Feihua. Thanks to the Tesla P100 and the GTX 1080 — two GPUs released in 2016 that held their posts throughout this research, proving that retired hardware and zero-information tasks are a perfect match. This model is released under the Apache-2.0 license and may be freely used, modified, and redistributed; for all Feihua produced through the use of this model, the glory and the responsibility belong to the user.

Citation

@misc{feihua-n1-64m,
  title={Feihua-n1-64M：零信息量语言模型的训练、强化与极限压缩研究},
  author={ZZRI},
  year={2026},
  howpublished={\url{https://huggingface.co/ZZRI/Feihua-n1-64M}},
  note={听君一席话，如听一席话}
}

Quick Start

Load with transformers (64M parameters, runs on CPU):

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("ZZRI/Feihua-n1-64M", device_map="cpu")
tokenizer = AutoTokenizer.from_pretrained("ZZRI/Feihua-n1-64M")

messages = [{"role": "user", "content": "人生的意义是什么？"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128, do_sample=True, temperature=0.8, top_p=0.95)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

One-line launch with llama.cpp (GGUF sweet-spot tier IQ4_XS):

llama-server -hf ZZRI/Feihua-n1-64M-GGUF:feihua-v6-IQ4_XS

The full seven-tier GGUF spectrum (Q8_0 down to IQ1_S) lives in a separate repository: ZZRI/Feihua-n1-64M-GGUF; the 49.16M version with 6-layer pruning plus the 79-second revival fine-tune is in the sister repository ZZRI/Feihua-n1-64M-prune.