yatnmn-softplus-sb-ca-d12-chinchilla-261M-pytorch

YatNMN-Softplus + scalar_bias + constant α=1 d=12 Chinchilla (261M) — PyTorch

A 261M-parameter nanochat-architecture GPT with YatNMN-Softplus MLP where both α is fixed at 1 and bias is a single shared scalar (1,). This is the minimal YatNMN ablation — the pure (x·W+b)²/(||x−W||²+ε) formula without learnable per-layer scaling or per-neuron bias.

Result: the bare YatNMN formula alone barely beats GELU (−0.03 nats on C4). The 0.14-nat advantage of full YatNMN-Softplus comes from the synergy between per-neuron bias and learnable α, not from the formula itself.

Ablation table (d=12, 261M, Chinchilla 20×, 3-seed mean)

Quick start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "mlnomad/yatnmn-softplus-sb-ca-d12-chinchilla-261M-pytorch",
    trust_remote_code=True, dtype=torch.float32,
).eval()
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

ids = tokenizer("The meaning of life is", return_tensors="pt").input_ids
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=50, do_sample=False,
                         use_cache=True, pad_token_id=tokenizer.eos_token_id or 0)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Config

Scalar (1,) bias + softplus_bias + learnable_epsilon + constant_alpha=True (α=1 fixed).

Related

• mlnomad/yatnmn-softplus-sb-d12-chinchilla-261M-pytorch — with learnable α (loss 3.06, PPL 39.5)
• mlnomad/yatnmn-softplus-ca-d12-chinchilla-261M-pytorch — per-neuron bias + constant α
• mlnomad/gelu-d12-chinchilla-261M-pytorch — GELU baseline

License

Apache 2.0.