InterSpec-demo-GGUF

InterSpec demo models (GGUF)

GGUF (f16) versions of a Pythia-1.4b verifier and four Pythia-31m drafts, for running speculative decoding with llama.cpp. The first three come from EugenTK10/InterSpec; the two EX4 drafts were trained after the thesis and are published only here. The code, the thesis and all evaluation outputs are at github.com/EugenTK10/InterSpec.

All drafts are meant to be used with the teacher as verifier. All five models share the GPT-NeoX tokenizer; each draft's GGUF vocabulary was checked token by token against the teacher's.

EX2 vs EX4. All four drafts start from the raw EleutherAI/pythia-31m. The EX2 drafts are the thesis models: draft LR 5e-6, one epoch of XSum (199,793 samples). The EX4 drafts use draft LR 3e-4 and three epochs (599,379 samples: 3 passes over the same 199,793). The frozen target the EX4-FS3 draft trained against is a 3-epoch teacher→target distillation that is not published.

Acceptance against the teacher

Measured with real speculative decoding in PyTorch (the thesis evaluation code) on 1000 XSum test prompts, draft length γ = 5. α is the acceptance rate, τ is the number of tokens produced per verifier forward pass.

These are single-seed numbers for these checkpoints. The thesis's across-seed comparisons cover the EX2 drafts only and are in the thesis; the EX4 drafts are not part of it. The EX2 and EX4 rows differ in both learning rate and training budget, so the gap between them is not attributable to either one alone. They were measured on the source (safetensors) checkpoints; llama.cpp's sampling and numerics differ, so the acceptance you see with these files will be close but not identical.

Prompt format

The models were trained on one prompt format and expect it exactly:

Article:
{document}

Summary:

The model continues with {summary} and ends with <|endoftext|>. These are not chat models and have no chat template.

Usage (llama.cpp)

llama-server \
  -m  pythia-1.4b-xsum-teacher-f16.gguf \
  --spec-type draft-simple \
  --spec-draft-model pythia-31m-xsum-draft-EX2-TT-S1-f16.gguf \
  --spec-draft-n-max 5 --spec-draft-n-min 5 --spec-draft-p-min 0.0

--spec-draft-n-min 5 --spec-draft-p-min 0.0 makes every round draft the full 5 tokens, as in the thesis. With the defaults, the server shortens drafts adaptively. Flag names are those of the llama.cpp build below; older builds use -md / --draft-max.

Plain decoding with the teacher alone:

llama-cli -m pythia-1.4b-xsum-teacher-f16.gguf -p $'Article:\n...\n\nSummary:' -n 64

Provenance

• EX2 drafts and teacher: EugenTK10/InterSpec at revision 17530560695709a3c7704657f618649e2a1ba759. The model.safetensors of all three checkpoints were checked against that revision by SHA-256.
• EX4 drafts: the best-validation checkpoint of each run, not on the Hub. SHA-256 of the source model.safetensors: EX4-FS3-S1 e1c9b3a6bb469c4258584121af8be1308e0b0ca176dea320bfba3f48af595f55, EX4-PB3-S1 c6b0a0fc49957ad018465adaca754682ec6deb5896f1ee914af4b90d1a80dc24.
• Converted with llama.cpp convert_hf_to_gguf.py (--outtype f16) at commit 4ceb1719101f32637b841206c172f3f058ffc182. One local patch was applied to the GPT-NeoX converter: it reads rope_parameters.partial_rotary_factor when the flat rotary_pct key is absent, as in configs saved by transformers ≥ 5. This changes only how the converter reads the config, not the weights.
• All source checkpoints are fp32. f16 is the only precision published here.
• SHA256SUMS lists the hashes of the five GGUF files.

License

Apache-2.0, inherited from Pythia. The models were fine-tuned on XSum.

Citation

@mastersthesis{falca2026interspec,
  author = {Falca, Eugen-Florian},
  title  = {Distilling Draft Models Through an Intermediate Target for Faster Speculative Decoding},
  school = {Jheronimus Academy of Data Science (JADS)},
  year   = {2026},
  url    = {https://github.com/EugenTK10/InterSpec}
}