SmolLM2-135M-CeNN-Partition-V3

SmolLM2-135M CeNN Partition V3

Validation-selected partition_conservative TinyCeNN Integrated Memory V3 checkpoint over HuggingFaceTB/SmolLM2-135M.

Architecture

• Variant: cenn_partition
• Replaced attention layers: [0, 29]
• Feature dimension: 64
• Block size: 32
• Sink tokens: 4
• Exact base revision: 93efa2f097d58c2a74874c7e644dbc9b0cee75a2
• TinyCeNN source commit: 0efdb79575204e9f02624ef64e0e6286076ab227

The remaining Transformer layers retain standard attention. This is a partial hybrid research checkpoint, not a fully attention-free model.

Held-out evaluation

Ratios below 1.0 are better for perplexity/cache. Speedups above 1.0 are faster. Current PyTorch CeNN kernels are experimental and are not yet optimized like GPU SDPA.

Load

import sys
from huggingface_hub import snapshot_download
folder = snapshot_download('vtava/SmolLM2-135M-CeNN-Partition-V3')
sys.path.insert(0, folder)
from load_model import load_model
model, tokenizer = load_model('vtava/SmolLM2-135M-CeNN-Partition-V3')

Reproducibility

• Validation NLL: 2.8096204151709876
• Before-joint validation NLL: 2.815813680489858
• Joint updates: 300
• Trainable TinyCeNN parameters: 124050
• Training precision: bfloat16
• Full benchmark evidence is under benchmark/.

Limitations

This checkpoint uses a limited held-out set and one training seed. It does not establish universal superiority over Transformer attention. Qualitative generations are not benchmark evidence.

Source and licenses

TinyCeNN-LM: https://github.com/vtavakkoli/TinyCeNN-LM Base model: https://huggingface.co/HuggingFaceTB/SmolLM2-135M SmolLM2 is Apache-2.0. TinyCeNN-LM source is MIT; its license copy is included as LICENSE-TinyCeNN-LM.