flame-27m-instruct

flame-27m-instruct

A 27.1M-parameter English instruction-tuned model: flame-27m-base (pretrained on FineWeb-Edu-dedup + Cosmopedia-v2 + ClimbMix + FineMath) fine-tuned on a ~1.18M conversation instruction mixture. ~5× smaller than SmolLM-135M-Instruct.

• Architecture: Llama-style decoder — hidden 512, 8 layers, 8 heads / 2 KV heads (GQA), intermediate 1280, RoPE (θ=1e6), context 2048, vocab 12000 (English BPE).
• Training: base → SFT (5 epochs, EMA weights) on SmolTalk + Tulu-3-Persona-IF + No-Robots + WildChat (English, non-toxic).

Usage (chat)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("ifx-pse-sys-ml/flame-27m-instruct", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("ifx-pse-sys-ml/flame-27m-instruct")

messages = [{"role": "user", "content": "Tell me about the moon in one sentence."}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(ids, max_new_tokens=64, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

The decoder also accepts inputs_embeds (exactly one of input_ids / inputs_embeds), so a vision projector can splice visual tokens in — usable as a small VLM text backbone. A raw PyTorch checkpoint (pytorch_model.pth) is included alongside the safetensors weights.

Generation parameters

The model ships its own generate() (loaded via trust_remote_code). Two things to know:

• It does not read generation_config.json. A bare model.generate(ids) uses the built-in defaults — temperature 0.85, top_p 0.85, top_k 50, repetition_penalty 1.0 — so pass the settings you want explicitly.
• Greedy is do_sample=False, not temperature=0 (which divides by zero).

Measured presets, from a 28-prompt VLM-style sweep over 13 decoding configs:

# VQA-style: answer from the context, deterministically
out = model.generate(ids, max_new_tokens=64, do_sample=False,
                     repetition_penalty=1.3, no_repeat_ngram_size=3, penalize_prompt=False)

# Captions / open-ended
out = model.generate(ids, max_new_tokens=120, do_sample=True, temperature=0.7, top_p=0.9,
                     repetition_penalty=1.3, no_repeat_ngram_size=3, penalize_prompt=False)

Benchmarks

Accuracy (%) via lm-evaluation-harness 0.4, same harness and shots for every model, so columns are directly comparable.

These academic benchmarks measure base knowledge, which SFT cannot add. The richer instruction mixture was chosen to improve instruction-following and response quality, which it does (held-out assistant-token loss 1.34 → 1.23 vs a SmolTalk-only SFT, and cleaner format adherence) — at a small cost on the knowledge probes above. Benchmarks are the wrong lens for an instruct model's quality; they are shown only for comparability with the base and SmolLM.

Honest limitations

At 27M parameters this is near random chance on knowledge/reasoning benchmarks; the gap to SmolLM-135M is capacity, not data or tuning. Instruction-tuning adds response format, not facts. It follows simple instructions but cannot reliably satisfy hard multi-constraint prompts (IFEval ≈ 0) — that capability is bound by the 27M base, not the SFT data. A research/prototyping instruct model and a lightweight decoder, not a knowledge model. English only. Trained with the Nexus codebase.