Qwen3.5-0.8B-CeNN-Integrated-V1

Qwen3.5-0.8B + TinyCeNN Integrated Memory V1

Experimental text-backbone adapter for Qwen/Qwen3.5-0.8B. The original Qwen3.5 model has 24 text decoder layers: 18 native Gated DeltaNet linear-attention layers and 6 full-attention layers. This experiment leaves all native linear-attention layers untouched and replaces selected full-attention layers with TinyCeNN bounded memory.

Validation-selected candidate: partition_conservative
Replaced full-attention layers: [3, 23]
Remaining original full-attention layers: 4
Base revision: 2fc06364715b967f1860aea9cf38778875588b17
TinyCeNN source commit: 2e4deedc7eddbb276b42ca683e752ad5f684b735

This repository stores an adapter checkpoint plus the exact custom loader/source required to reconstruct the model. It is not a standalone save_pretrained() checkpoint and does not include the original Qwen weights.

Architecture

Qwen3.5-0.8B uses a 3:1 hybrid text stack (Gated DeltaNet linear attention plus periodic full attention). TinyCeNN is applied only to the original full-attention positions. Qwen3.5's Q/K normalization, partial MRoPE, and post-attention output gate are preserved. The TinyCeNN readout remains explicit because it cannot be folded through Qwen3.5's elementwise output gate without changing the computation.

The cenn_partition memory keeps sink/current/previous-block information exact and compresses older history into a bounded recurrent state. Therefore this is not "attention-free" in the strict sense: local exact attention remains inside the replacement, while unbounded global full attention is replaced.

Held-out benchmark

Selection used validation NLL only; held-out test documents were not used for model selection.

The current implementation is research PyTorch, not a fused production kernel. Cache reduction and quality should be interpreted separately from wall-clock speed.

Load

from load_model import load_model
model, tokenizer = load_model('.')

For generation, use batch size 1 with the custom greedy_generate helper in tinycenn_lm.qwen35_integrated_memory until a standard Transformers cache/generation integration is packaged.

Reproducibility files

manifest.json, selection.json, integrated_report.json, validation/test CSVs, and generation examples are included. The manifest pins the exact base-model revision, dataset revision, source commit, package versions, split hashes, and experiment settings.

Limitations

• Experimental research adapter; not a production model.
• Text backbone only. The original Qwen3.5 vision tower is not modified or packaged here.
• Benchmark confidence intervals are over held-out documents, not multiple independent training seeds.
• Custom TinyCeNN cache currently targets batch-one greedy decoding; beam/batch cache reordering is unsupported.
• Replacing all six full-attention layers removes original quadratic global attention from the text backbone, but native Gated DeltaNet layers and TinyCeNN local exact attention remain.

Licenses

The base Qwen3.5 checkpoint is Apache-2.0. TinyCeNN-LM source is distributed under its repository license; a copy is included when available.