sparklm-31m-tinystories

SparkLM-31M-TinyStories

This is a SparkLM checkpoint exported into a HuggingFace-compatible directory. It is intended to load through the Transformers remote-code path.

Architecture

• Model type: SparkLMForCausalLM
• Parameters: 30,814,720
• Hidden size: 512
• Layers: 8
• Attention heads: 8
• KV heads: 2
• MLA: True
• MoE: False
• MTP: False
• Q/K Norm: False
• Tokenizer source: results/mps_31m_tinystories_final

Checkpoint Metadata

• Step: 199
• Loss: 1.921442563533783

Training loss is not a benchmark score. Publish evaluation results separately before comparing this checkpoint with other public models.

Intended Use

This release is suitable for reproducibility checks, loading smoke tests, and lightweight text-generation experiments. It is not a safety-aligned assistant release unless a downstream alignment and evaluation report is attached.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "gatilin/sparklm-31m-tinystories"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
inputs = tokenizer("Hello SparkLM", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

The exported remote code wraps SparkLM's native PyTorch implementation in a transformers.PreTrainedModel interface.

Release Contract

A public release is considered loadable when the repository root includes:

• config.json
• generation_config.json
• tokenizer_config.json
• tokenizer vocabulary artifacts
• model.safetensors or pytorch_model.bin
• configuration_sparklm.py
• modeling_sparklm.py
• tokenization_sparklm.py
• release_manifest.json

The config and tokenizer config should expose auto_map entries so AutoConfig, AutoTokenizer, and AutoModelForCausalLM can load with trust_remote_code=True.

Expert Parallel Training

This model supports end-to-end expert-parallel training via a differentiable all-to-all dispatch/combine primitive (_AllToAllSingle autograd.Function). The backward pass through the EP all-to-all is a reverse all-to-all with swapped split sizes, enabling gradient flow across EP ranks.

Files

The exporter writes model.safetensors when the safetensors package is available, falling back to pytorch_model.bin in minimal environments.

Release Validation

Run this before uploading or after cloning the repository:

python -m sparklm.cli.release.validate_hf_release gatilin/sparklm-31m-tinystories --load --generate

Launch a local inference UI against the same release:

python -m sparklm.cli.inference_webui --model gatilin/sparklm-31m-tinystories