Qwen35-0.8B-FlyFFN-v2

Qwen3.5-0.8B FlyFFN-v2

Experimental FFN-only conversion of Qwen/Qwen3.5-0.8B using the FlyFFN-v2 architecture from TinyCeNN-LM.

This checkpoint explores whether Qwen3.5's feed-forward networks can be progressively replaced by graph-routed sparse FlyFFN-v2 modules while preserving the pretrained model's token-mixing backbone. The original Gated DeltaNet / full-attention token mixers are left unchanged. Only FFN blocks are modified, and every fourth FFN is retained as a dense anchor.

Research status: experimental checkpoint. It is intended for architecture research, ablation studies, and reproducibility—not as a drop-in production replacement for the base Qwen3.5 model.

Links

• Source repository: https://github.com/vtavakkoli/TinyCeNN-LM
• Canonical Colab notebook: https://github.com/vtavakkoli/TinyCeNN-LM/blob/main/notebooks/Qwen35_08B_FlyFFN_v2_Colab.ipynb
• Base model: https://huggingface.co/Qwen/Qwen3.5-0.8B
• Model repository: https://huggingface.co/vtava/Qwen35-0.8B-FlyFFN-v2

Architecture

The model starts from Qwen/Qwen3.5-0.8B and replaces selected FFN blocks with ProgressiveFlySwiGLU / FlyFFN-v2 modules.

Key properties of the recorded run:

The intended progressive routing schedule is increasingly sparse, but each layer is quality-gated. A stage that exceeds the allowed CE gap is rolled back rather than blindly committed.

In the recorded strong run, the accepted final routing states were:

layers 0,1,2,4,5,6,8,9,10   -> k=4, mix=0.50
layers 12,13,14,16,17,18    -> k=4, mix=0.25
layers 20,21,22             -> k=6, mix=0.10
dense anchors               -> 3,7,11,15,19,23

This means the experiment should be interpreted as progressive, quality-gated FFN conversion, not as a claim that all layers reached the sparsest target state.

Training setup

The published notebook records a strong run with:

• sequence length: 128
• batch size: 1
• 3,000 global training updates
• gradient accumulation: 2
• 12 calibration steps per stage
• FineWeb-Edu blocks for calibration, probing, training, and evaluation
• NVIDIA L4 GPU
• bfloat16
• rewired-topology control disabled in this run
• seed: 5321

The teacher/base model remains frozen during the conversion experiment.

Recorded quality results

The notebook reports the following held-out language-model diagnostics:

The model therefore stayed close to the teacher on the reported CE/perplexity probe, but the current reference implementation was slower than the base model. This checkpoint should not be presented as a demonstrated inference-speed improvement.

Important performance caveat

The notebook run used reference PyTorch fallbacks for some Qwen kernels because causal_conv1d and flash-linear-attention were not installed. In addition, the current FlyFFN-v2 quality prototype computes all shards during dense/sparse blending instead of using a fully fused selected-shard dispatch. The recorded speed ratio is therefore an implementation result, not a hardware-level limit of the architecture.

FastEval sanity check

The notebook evaluates 50 sampled items per available benchmark using identical prompts for the base model and FlyFFN-v2. In the recorded run, GPQA-Diamond was unavailable because the dataset was gated, leaving three evaluated benchmarks (150 items total).

Macro delta: -7.3 percentage points.

These are small sampled sanity checks, not official benchmark scores. They indicate that the converted model remains functional, but they also expose task-level degradation—especially on MMLU-Pro and MMMLU-DE. The checkpoint should therefore not be described as matching or surpassing the base model on downstream capability.

Qualitative behavior

The notebook includes:

• deterministic chat samples after training;
• a side-by-side interactive multi-turn chat comparing original Qwen3.5 with FlyFFN-v2;
• a fresh standalone reload test.

The model generates coherent text and successfully reloads as a standalone checkpoint. However, qualitative samples also show factual and reasoning imperfections. Users should independently evaluate the model on any target task before drawing conclusions about capability retention.

Standalone checkpoint

The exported package was verified in the notebook:

The authoritative weights are stored in:

standalone_state.pt

Because the architecture contains custom FlyFFN-v2 modules, this checkpoint is not intended to be loaded with a plain AutoModelForCausalLM.from_pretrained() call. Use the supplied standalone loader.

Installation

pip install -U torch transformers huggingface_hub safetensors

Load from Hugging Face

from huggingface_hub import snapshot_download
import sys
import torch

model_dir = snapshot_download("vtava/Qwen35-0.8B-FlyFFN-v2")
sys.path.insert(0, model_dir)

from qwen35_standalone import load_standalone

model, tokenizer = load_standalone(
    model_dir,
    device="cuda" if torch.cuda.is_available() else "cpu",
)

Generate text

import torch

messages = [
    {
        "role": "user",
        "content": "Explain sparse feed-forward networks in simple terms."
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(
    prompt,
    return_tensors="pt",
).to(next(model.parameters()).device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=160,
        do_sample=False,
        use_cache=True,
        pad_token_id=tokenizer.eos_token_id,
    )

answer = tokenizer.decode(
    output[0, inputs.input_ids.shape[1]:],
    skip_special_tokens=True,
)

print(answer)

Reproduce the experiment

The canonical notebook is:

notebooks/Qwen35_08B_FlyFFN_v2_Colab.ipynb

or open it directly:

https://github.com/vtavakkoli/TinyCeNN-LM/blob/main/notebooks/Qwen35_08B_FlyFFN_v2_Colab.ipynb

The corresponding implementation and runner are:

src/tinycenn_lm/qwen35_flyffn_v2.py
src/tinycenn_lm/qwen35_standalone.py
scripts/run_qwen35_flyffn_v2.py

The notebook also exports artifacts such as:

report.json
summary.csv
bio_progressive_calibration.csv
bio_training_history.csv
fast_eval_50_qwen35.csv
chat_samples.json
standalone_manifest.json

Intended uses

This model is suitable for:

• research on sparse or graph-routed FFN alternatives;
• progressive post-training architecture conversion;
• FFN quality-gating and rollback experiments;
• comparison of dense-anchor and sparse-FFN hybrids;
• reproducibility of the FlyFFN-v2 Qwen3.5 experiment;
• studying architecture/quality trade-offs.

It is not presented as a production-optimized Qwen3.5 replacement.

Limitations

• The recorded downstream sanity check is small.
• The FastEval run shows a macro accuracy drop versus the base model.
• No GPQA-Diamond score was obtained in the recorded run.
• The implementation currently does not demonstrate a decode-speed advantage.
• The parameter count is slightly larger than the base model in this prototype.
• The run does not include the rewired-topology control.
• Training/evaluation sequence length is limited relative to long-context use cases.
• Custom Python modules are required to reconstruct the architecture.
• The model may inherit biases, factual errors, safety limitations, and other failure modes from the base Qwen3.5 model and from its post-training conversion.

Interpretation

FlyFFN-v2 should be viewed as a research prototype for controlled FFN replacement, not as evidence that sparse FlyFFN layers universally outperform dense Qwen FFNs.

The strongest result of this experiment is that a large fraction of Qwen3.5 FFNs can be replaced progressively while keeping the token mixers untouched and maintaining a relatively small CE/perplexity gap on the reported probe. At the same time, the sampled downstream evaluation shows that preservation on language-model loss does not automatically imply full task-level capability preservation.

That distinction is important and is one of the main reasons the repository keeps both positive results and regressions visible.

Citation

If you use this checkpoint or the FlyFFN-v2 experiment, please cite the TinyCeNN-LM repository and the exact notebook/commit used for your experiment.

@software{tinycenn_lm_2026,
  title  = {TinyCeNN-LM: CeNN/Recurrent-Memory and Sparse-FFN Experiments for Pretrained Language Models},
  author = {Tavakkoli, Vahid and Mohsenzadegan, Kabeh and Kyamakya, Kyandoghere},
  year   = {2026},
  url    = {https://github.com/vtavakkoli/TinyCeNN-LM}
}

Acknowledgements

This experiment builds on Qwen/Qwen3.5-0.8B and the TinyCeNN-LM research codebase. Please also follow the base model's license, usage conditions, and citation guidance.