questions-lfm2-4bit

questions-lfm2-4bit

LFM2.5-230M fine-tuned with MLX LoRA to extract the most recent question from a multi-turn dialogue. Input is a transcript tagged with [S] and [M] line prefixes. The model prioritizes questions appearing under the [S] tag; if no question is present there, it falls back to the latest [M] block.

End-to-end latency on M-series Mac: 100–150 ms per call (greedy decoding, max 80 tokens, 4-bit quantized).

Pipeline

1. LoRA fine-tune in MLX (rank 32, 200 iters, 6.19M trainable params / 2.7%)
2. Fuse adapter into base model → fp16
3. Quantize to 4-bit (group size 64)

Input format

Transcript with [S] and [M] line prefixes:

[S] what would you like to discuss today
[M] i was thinking about the architecture of the new service
[S] ok
[M] could you walk me through the current approach

Output

A single extracted question (no question mark, no quotes). Priority order:

1. Latest question found under [S]
2. Otherwise, latest question found under [M]
3. Otherwise, the most recent [M] content

Usage (Python)

from mlx_lm import load, generate

model, tokenizer = load("GameGC/questions-lfm2-4bit")

messages = [
    {"role": "system", "content": "Extract the most recent question from the dialogue."},
    {"role": "user", "content": "[S] what would you like to discuss\n[M] i was thinking about the architecture\n[S] ok\n[M] could you walk me through the current approach"},
]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
out = generate(model, tokenizer, prompt=prompt, max_tokens=80)
print(out)

Usage (Swift)

import MLXLMCommon

let config = ModelConfiguration(id: "GameGC/questions-lfm2-4bit")

Performance

Limitations

• Declarative statements under [M] containing question-sounding words ("how", rhetorical "right") may trigger false-positive extraction. Will be addressed in v2.
• 4-bit quantization introduces a minor quality regression vs the fp16 fused version.
• Maximum context length: 1024 tokens. Longer transcripts are truncated.

Training