min-spark

min-spark

min-spark is a 5.76M-parameter language model with native effort levels. It introduces controllable depth to the sub-10M model class. The same checkpoint can produce a quick completion or spend more computation on the same prompt, selected with one inference argument.

The model was trained on 10.01B tokens. Its looped decoder reuses a compact transformer core across multiple passes, giving a small model the computation of a deeper network while keeping the parameter count fixed.

Introducing native effort levels

Effort levels have usually been associated with large reasoning models, where they adjust the thinking token budget or modify the system prompt to change how hard the model thinks. min-spark brings the idea directly into the language model. Each effort level changes the model's internal computation by selecting a different number of passes through its shared core.

Effort is a generation-time choice.

Usage

min-spark is compatible with Transformers and requires remote code loading.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "MinimaLabs/min-spark",
    trust_remote_code=True,
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained(
    "MinimaLabs/min-spark",
    trust_remote_code=True,
)

prompt = "The meaning of life is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(
    **inputs,
    effort="high",
    max_new_tokens=64,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

The same model works with the Transformers pipeline API:

from transformers import pipeline

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
result = pipe(prompt, effort="high", max_new_tokens=64)
print(result[0]["generated_text"])

A lightweight Transformers-free generation script is included in the repository:

python generate.py -p "The meaning of life is" -e high

Generation currently runs one sequence at a time, and right-padded batches are supported for evaluation. This model currently does not have KV cache support so long generations recompute the prompt at each step which may result in slightly longer generations. The context window is 512 tokens.

Evaluation

Scores below come from zero-shot evaluation with lm-eval 0.4.12. BLiMP uses accuracy. ARC-Easy, ARC-Challenge, HellaSwag, and PIQA use length-normalized accuracy. WikiText-2 is reported as byte-level perplexity, where lower is better.

min-spark reaches 69.19% on BLiMP at medium effort. ARC-Easy reaches 37.08%. ARC-Challenge reaches 23.21%. HellaSwag reaches 27.92%. PIQA reaches 54.35%. Its best WikiText-2 byte perplexity is 2.7747.

Effort levels

The table shows how the model responds to additional internal computation. Grammar improves most clearly from low to medium effort while the common-sense tasks remain close across the three settings.

Benchmarks

The comparison places min-spark alongside published results for GPT-S2-5M, SLM-10M, and michel-nano-v2. It covers the benchmarks reported across this group.

min-spark reaches 37.08% on ARC-Easy, the highest score in this comparison. Its 23.21% ARC-Challenge result is close to SLM-10M at 23.46%. HellaSwag reaches 27.92%, and PIQA reaches 54.35%.

More in Less Parameters

KeyLM was the previous model made by us, and is a 75.25M-parameter language model trained on 18B tokens. min-spark uses less than one thirteenth of its parameter count while reaching a similar range on several small-model evaluations.

KeyLM reports 29.9% on its ARC average, 29.7% on HellaSwag, and 60.0% on PIQA. min-spark reaches 37.08% on ARC-Easy, 23.21% on ARC-Challenge, 27.92% on HellaSwag, and 54.35% on PIQA. The comparison highlights the value of repeated computation in a compact model. min-spark closes much of the size gap on these tasks while retaining a 5.76M parameter footprint.

Architecture

Training

Reproducing the evaluation

The evaluation runner is included in the repository. It supports each effort level and the benchmark suite used for this card.

python run_lmeval.py \
  --effort medium \
  --tasks blimp,arc_easy,arc_challenge,hellaswag,piqa,wikitext

Use --limit N to run a smaller evaluation during development.

Limitations

min-spark is a small base language model. It is not instruction-tuned and does not provide conversational alignment or safety filtering. Factual recall, multi-step reasoning, and long-form coherence are limited by its scale and training objective.

The model supports a 512-token context window. Generation accepts one sequence at a time and currently has no KV cache. Results were collected with a 512-token context and should not be assumed to transfer to longer inputs.

License

Apache-2.0. See LICENSE.