LFM2-8B-A1B-smoltalk-LoRA

LFM2-8B-A1B SmolTalk LoRA

LoRA adapter for LiquidAI/LFM2-8B-A1B (8B total / 1B active MoE) fine-tuned on SmolTalk.

Published in the per-expert layout (...experts.<N>.w1/w3/w2.lora_*, Mixtral-style) — the interchange format that both HF PEFT and inference engines load:

• HF PEFT (peft > 0.19.1, transformers ≥ 5.8): loaded via transformers' registered lfm2_moe weight conversion, which folds the per-expert keys into the batched expert layout at load time. Verified token-identical to the originally trained adapter.
• SGLang (with sgl-project/sglang#27813): loaded directly via the per-expert MoE LoRA path.

Full module coverage: attention (q/k/v/out_proj), ShortConv (in_proj/out_proj), dense MLP (layers 0–1), MoE router (gate), and all 32 experts on every MoE layer.

The rank_pattern/alpha_pattern entries in adapter_config.json are required: the fused experts.gate_up_proj slot reconstructed from independent per-expert gate/up halves has rank 2r=16 (alpha scaled to 32 to preserve the effective scale of 2.0). The per-expert tensors themselves are rank 8.

Training

Trained with PEFT target_parameters on the batched expert tensors; this artifact is a lossless repacking (bundled script: convert_lfm2_moe_lora_to_per_expert.py). The original target_parameters-format weights remain available at revision e3d7523.

Usage with HF PEFT

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

MODEL_ID = "LiquidAI/LFM2-8B-A1B"
base = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "LiquidAI/LFM2-8B-A1B-smoltalk-LoRA")

Requires peft > 0.19.1 (e.g. pip install git+https://github.com/huggingface/peft.git until the next release — 0.19.1 has a WeightConverter incompatibility with transformers ≥ 5.6 that affects loading any LoRA on this architecture).

Usage with SGLang

sglang serve \
    --model-path LiquidAI/LFM2-8B-A1B \
    --port 30000 \
    --enable-lora --max-lora-rank 8 \
    --lora-paths "smoltalk=LiquidAI/LFM2-8B-A1B-smoltalk-LoRA" \
    --lora-target-modules q_proj k_proj v_proj out_proj gate_proj up_proj down_proj in_proj gate \
    --lora-strict-loading

When serving with --dtype float16, also set SGLANG_MAMBA_CONV_DTYPE=float16 (the ShortConv state cache defaults to bf16). The default bf16 serving needs no extra flags.

curl -sS http://localhost:30000/generate \
    -H "Content-Type: application/json" \
    -d '{
      "text": "Write a short story set in an unusual location.",
      "sampling_params": {"temperature": 0.0, "max_new_tokens": 80},
      "lora_path": "smoltalk"
    }'

Correctness

Both load paths were verified by teacher-forced top-k logprob comparison (greedy, fp16, same GPU), with this exact artifact on both sides: HF+PEFT and SGLang produce token-identical generations; prefill logprob divergence mean 0.011 / max 0.026.