goat-vvv-d12-fineweb-5x-pytorch

GOAT_VVV d=12 · FineWeb-Edu 5× Chinchilla (PyTorch)

GOAT_VVV attention: no Q/K projections and no V projection. Queries, keys, AND values are all the same RoPE'd x_heads (hence "VVV"). Only attention params per layer: c_proj + per-head bias / ε scalars.

x_heads = RoPE(x.reshape(B, T, H, D))
dots    = x_heads @ x_heads^T
dist²   = ||x_i||² + ||x_j||² − 2·dots
scores  = (dots + softplus(b))² / (dist² + softplus(ε))
scores  = L1_normalize(scores)    # strict causal j<i
y       = scores @ x_heads         # ← no V projection
y       = c_proj(y)

Architecture

Complexity vs GPT-2

Per-layer: GPT-2 = 12 n², GOAT_VVV = 9 n² (25% fewer params — dropped all of Q/K/V projections). Activation memory during forward is ~⅓ of GPT-2's for the attention block (no separate Q/K/V tensors). No KV cache needed — values are derived from the residual.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "mlnomad/goat-vvv-d12-fineweb-5x-pytorch",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

ids = tokenizer("The capital of France is", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=30, do_sample=True, temperature=0.8, top_p=0.9)
print(tokenizer.decode(out[0], skip_special_tokens=True))

License

Apache 2.0.