LMA_bonus_nopos_hindi

LMA bonus ablation — Hindi without positional embeddings

An 11-layer decoder-only Transformer trained on Hindi with the learned absolute positional table removed entirely (positional: "none"), so the model has no explicit representation of token order.

Everything else matches the standard Hindi model it is compared against — the same 16K tokenizer, the same corpus and split, the same one-epoch horizon, and the same optimiser recipe (peak LR 3e-3, 750 warmup steps, 32 x 2 x 512 tokens per step). Training-set size, schedule and seed are held constant, so the only variable is the positional table.

What removing the table does and does not do

It does not make the model order-blind. Causal attention is itself a positional signal: with the mask in place, position t attends to 0..t, so representations at different positions are built from different prefixes. A one-layer no-positional model is genuinely permutation-invariant over its prefix, but from two layers onward that invariance is gone. This model therefore has to infer order from the causal mask rather than read it from an embedding — a weaker and less direct signal, which is what the evaluation measures.

Loading

from huggingface_hub import snapshot_download
from safetensors.torch import load_file
import json

d = snapshot_download("Prateek-Tiwari10/LMA_bonus_nopos_hindi")
cfg = json.load(open(f"{d}/config.json"))
weights = load_file(f"{d}/model.safetensors")

The architecture is lm/model.py in the project repository, not a transformers class. build_model re-ties the output head to the token embedding, which is why no lm_head tensor is stored.