gslm-scaling-823m-16p5b

gslm-scaling-823m-16p5b

Model-only FP32 weights for a final-budget scaling-curve endpoint from Scaling Properties of Speech Language Models (EMNLP 2024). This checkpoint has 823,179,264 parameters and saw exactly 16,499,220,000 tokens at 934,800 tokens per update.

import torch
from slm import TransformerLM

model = TransformerLM.from_pretrained("tiagoCuervo/gslm-scaling-823m-16p5b", device="cuda")
prompt = torch.tensor([[12, 91, 204]], device="cuda")
units = model.generate(prompt, max_new_tokens=200, temperature=0.8, top_k=50)

Evaluation scores are 61.08% sBLIMP phenomenon macro accuracy, 58.08% sBLIMP voice-pair accuracy, 55.21% sStoryCloze, and 75.52% tStoryCloze.

The model predicts consecutive-run-collapsed 25 Hz layer-11 mHuBERT K-means units (K=500; EOS=500; PAD=501). A vocoder is not bundled.

Speech-tokenizer and vocoder artifacts are credited in the SLM third-party notices. See provenance.json for exact hashes and evaluation metadata.