Speck2-140M

Speck2-140M

Speck2-140M is a 140.7M parameter English base language model that interleaves global grouped-query attention with gated causal convolution. It was pretrained from scratch on 20B tokens using a three-phase curriculum that shifts from broad web text toward math, synthetic, and stylistically diverse high-quality text.

This is a base model, not instruction-tuned or specialized in any way. It has no chat template and no safety alignment.

Summary

Architecture

18 residual blocks: 8 global attention + 10 gated causal convolution, each followed by a SwiGLU feed-forward.

Input/output embeddings (640-wide) are tied and connect to the 768-wide residual stream via learned projections.

Usage

Speck2-140M works with the Transformers Auto classes through its bundled custom model and tokenizer code. Set trust_remote_code=True when loading it.

pip install "transformers==5.1.0" torch sentencepiece safetensors

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "specklabs/Speck2-140M"
device = "cuda" if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype="auto",
).to(device)

prompt = "The meaning of life is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

output = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=False,
)
generated = output[0, inputs.input_ids.shape[1] :]
print(tokenizer.decode(generated, skip_special_tokens=True))

The bundled generation path is validated for single-prompt greedy decoding. Direct forward passes support right-padded batches when use_cache=False.

Training

Muon optimized 2D matrix parameters; AdamW (beta 0.9/0.95, epsilon 1e-8) handled embeddings, norms, and conv kernels. The run used a single RTX 5090 rented through Vast.ai.

Training documents were globally deduplicated after NFKC, lowercase, and whitespace normalization. The curriculum used three mixture phases:

Evaluation

The quality columns combine the Open SLM Leaderboard at revision 2eafcfc647b667e67f3b0288e9b67da497a78052 and BananaMind Base Bench 1.1 at revision d4aade51312889e8580963e1ce960c6eaef1a450. No chat template or generation was used for the four Speck evaluations.

Benchmarks and speed

Open SLM Int Index means the chance-normalized Intelligence Index reported by the Open SLM Leaderboard. BananaMind Base Bench 1.1 Elo means the overall Elo reported by BananaMind Base Bench 1.1. Speed and memory values are local batch-1 measurements described below. Reference models were pretrained on 1.5 to 100 times as many tokens. This is a parameter-adjacent comparison, not a compute-matched one.

Inference speed

Speed was measured locally at batch 1 with eager PyTorch, model-native caches, last-token logits, and tokenization excluded. Prefill uses 512 tokens. Decode measures 64 greedy cached steps after a 448-token prefix and includes argmax. CPU runs use FP32 with 16 threads; RTX 3090 runs use BF16. Reported throughput is calculated from the median duration.

Memory is unique live BF16 model tensor storage plus cache/state tensor storage after a 2,048-token prefill at batch 1. It excludes framework RSS, CUDA allocator reservations, and temporary operator workspace. FP32 CPU tensor memory is approximately twice the reported BF16 memory. For another context length N, approximate memory as model tensor memory + State@2K x N / 2,048; Speck's small convolution history is fixed rather than context-scaled.

Speck2 has the same architecture and release implementation as Speck1, so its speed and memory entries use the validated Speck1 measurements. The Speck architecture ranks 2nd among the compared models on both CPU and GPU batch-1 prefill and decode throughput, behind the smaller Supra2-100M-Base. The deduplicated BF16 Safetensors release is 281.3 MB.

Limitations

• Not instruction-tuned: can't reliably follow requests.
• No safety alignment: can produce biased, harmful, or incorrect text.
• Weak at arithmetic and quantitative reasoning.
• Mostly English: multilingual ability untested.
• Validated only up to 2,048 tokens despite a 4,096-token config.
• Trained on web-derived data that may contain bias, errors, or copyrighted text.
• No red-team or misuse evaluation performed.

Reproducibility

Full training and eval code: github.com/alkinun/speck

Citation

@misc{alkinun2026speck2,
  author       = {alkinun},
  title        = {Speck2-140M: A Compact Hybrid Attention-Convolution Language Model},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/specklabs/Speck2-140M}},
  url          = {https://github.com/alkinun/speck}
}