abl_2_1_frontloaded

CRUMB abl_2_1_frontloaded

Model Overview

abl_2_1_frontloaded is a hybrid decoder-only language model from the CRUMB (Compact Recurrent-hybrid Underlying Mamba Blocks) project. It interleaves Mamba-3 selective state-space layers with GQA (Grouped-Query Attention) layers at a 2:1 Mamba-majority ratio with frontloaded placement, and was pre-trained exclusively on Python source code.

It is one of eleven ablation variants trained to study how the Mamba-to-attention ratio and the placement of attention layers affect small-scale (~150M-parameter) language models on Python program synthesis.

Architecture

Mamba : Attention ratio — 2 : 1

8 Mamba layers + 4 GQA attention layers (2 Mamba blocks per attention block).

Placement — Frontloaded

Attention layers are stacked at the beginning of the network. Layer order: A A A A M M M M M M M M

This placement forces attention-based token-level matching in the early layers, with the final layers dominated by linear-time Mamba blocks before the output projection.

Training

Evaluation Method

Perplexity (primary metric)

Per-token cross-entropy loss with BF16 autocast, computed over the full held-out evaluation set.

Generation-based metrics

• Python syntax validity — 200 free-form completions generated per model from 49 diverse Python prompts at temperature=0.8, top_k=50, max_new_tokens=128; each completion checked with ast.parse(). Implementation: src/evaluation/syntax_validity.py.
• Qualitative side-by-side completions — 10 fixed prompts at temperature=0.6, top_k=50, max_new_tokens=200, identical random seed per prompt. Implementation: src/evaluation/qualitative_comparison.py.

Evaluation Results

Rank Summary

Out of 11 ablation configurations evaluated at the same token budget:

abl_2_1_frontloaded ranks 10th overall. Frontloaded placement is uniformly inferior across all three ratios — placing attention early forces token-level matching before the network has built up enough representational depth. Notably, abl_2_1_frontloaded has the highest syntax validity rate of any model (33.5 %), but this is within the 95 % confidence interval (±6.3 % at n=200) and is not statistically discriminative.

Intended Use & Limitations

• Domain: Python source-code language modelling.
• Base model only: no instruction tuning, no chat alignment, no safety filtering. Outputs are unconstrained code completions.
• Repetitive degeneration: all base CRUMB models tend to repeat function signatures / docstrings during free-form generation; this is expected behaviour for unaligned base models.

Citation / Context

This model is part of the CRUMB Phase-1 ablation study:

Efficient Architectural Hybrids for Small-Scale Language Models in Python Program Synthesis — Department of Computer Science and Engineering, Daffodil International University. Findings documented in documents/phase1_ablation_findings.md.

How to Load

from tokenizers import Tokenizer
import torch
from src.model.config import CRUMBConfig
from src.model.model import CRUMBModel

config = CRUMBConfig.from_yaml("configs/model/abl_2_1_frontloaded.yaml")
model = CRUMBModel(config)
state = torch.load("saved/model/abl_2_1_frontloaded/model.pt", map_location="cpu")
model.load_state_dict(state)
model.eval()

tok = Tokenizer.from_file("saved/tokenizer/crumb_tok_hf/tokenizer.json")
ids = tok.encode("def fibonacci(n):\n").ids
x = torch.tensor([ids])
with torch.no_grad():
    y = model(x)