lfm2-quantum-128m-sft-v2

lfm2-quantum-128m-sft-v2 (SFT checkpoint, step 1291)

SFT (instruction-tuned) checkpoint of MarkChenX/lfm2-quantum-128m, an LFM2-style hybrid quantum GPT, trained with Quantum-GPT runs/run_quantum_sft_128m.sh (Stage 3: SFT).

This is a continuation of MarkChenX/lfm2-quantum-128m-sft (step 849), resumed and further fine-tuned to step 1291 with an expanded data mixture (added UltraChat, MetaMath, CodeAlpaca).

Unlike the base checkpoint, this model follows a chat format (single/multi turn conversation with <|user_start|>/<|assistant_start|> special tokens), not raw text continuation.

Architecture

Same architecture as the base model:

• 16 layers, hybrid mixer: 10 gated short-conv blocks + 6 GQA attention blocks (conv-first, pattern CCCCCCCCCCAAAAAA)
• n_embd=1024, 16 query heads / 8 KV heads (GQA), head_dim=64, seq_len=1024
• vocab_size=32768
• Quantum feed-forward network in every block (4 qubits, circuit depth 2)
• RoPE base theta=1e6, full (non-windowed) attention, value embeddings disabled

Full config: config.json (metadata only) and meta_001291.json (exact training config this checkpoint was produced with). This is a custom architecture, not a transformers model -- there is no AutoModel support.

SFT details

• Resumed from MarkChenX/lfm2-quantum-128m-sft (step 849), warm-started from its optimizer state (momentum buffers)
• Data mixture: SmolTalk (general chat) + UltraChat (general chat) + MMLU auxiliary_train x3 epochs (multiple choice) + GSM8K train x4 epochs (math) + MetaMath (math) + CodeAlpaca x2 epochs (code) + ARC x2 epochs (science QA)
• Validation bpb at step 1291: 0.4539

Inference

This repo bundles the minimal nanochat/ source needed to load and run the model, so it's self-contained (no need to clone the full training repo).

pip install torch tiktoken rustbpe filelock kernels
hf download MarkChenX/lfm2-quantum-128m-sft-v2 --local-dir ./lfm2-quantum-128m-sft-v2
cd lfm2-quantum-128m-sft-v2
python inference.py --prompt "What is the capital of France?"

See inference.py for the full loading + generation code: it builds the model from model_001291.pt + meta_001291.json, loads the tokenizer from tokenizer/, wraps the prompt in the chat special tokens (<|user_start|>...<|user_end|><|assistant_start|>), and streams the response via the model's own .generate() until <|assistant_end|>.

Resuming SFT / continuing to RL

optim_001291_rank0.pt is the matching Muon/AdamW optimizer state for this step. To continue training with the original Quantum-GPT repo, place model_001291.pt, optim_001291_rank0.pt and meta_001291.json under $NANOCHAT_BASE_DIR/chatsft_checkpoints/lfm2-quantum-128m/, then run:

RESUME_FROM_STEP=1291 bash runs/run_quantum_sft_128m.sh   # continue SFT
# or
MODEL_TAG=lfm2-quantum-128m SKIP_SFT=1 bash runs/run_lfm2_sft_rl.sh  # move to RL