Downloads · 30 days
20
8% of all-time downloads
sakamakismile/LFM2.5-1.2B-JP-202606-NVFP4
LFM2.5-1.2B-JP-202606-NVFP4 is a text generation model from sakamakismile. Use it when you need the model to write or continue text. It is set up for vllm. The card lists the license as other.
NVFP4 (W4A4) quantization of LiquidAI/LFM2.5-1.2B-JP-202606 — Liquid AI's latest general-purpose Japanese chat model (1.2 B-param dense lfm2 hybrid: short-convolution + attention, 128 K context). The 4-bit build is 1.…
Downloads · 30 days
20
8% of all-time downloads
All-time downloads
265
Public
Parameters
818M
1.2 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors1.2 GB · 100%
How the weights are stored.
U8403M · 49%
From the Hugging Face model README
NVFP4 (W4A4) quantization of LiquidAI/LFM2.5-1.2B-JP-202606 — Liquid AI's latest general-purpose Japanese chat model (1.2 B-param dense lfm2 hybrid: short-convolution + attention, 128 K context). The 4-bit build is 1.1 GiB on the GPU and leaves room for a ~1.87 M-token KV cache on a single 16 GB Blackwell card.
Quantized by Lna-Lab with NVIDIA TensorRT Model-Optimizer (modelopt). Loads directly in vLLM as modelopt_fp4 — no checkpoint surgery, no --quantization flag required (auto-detected).
NVFP4 (W4A4) 量子化版 — リキッドAIの最新汎用日本語チャットモデル(1.2B dense lfm2、短畳み込み+注意のハイブリッド、128Kコンテキスト)を4bit化。GPU上の重みは1.1 GiB、16GBのBlackwell 1枚で約187万トークンのKVキャッシュを残せます。
Why it's nice: the model is tiny (~1.2 B dense) and edge-tuned, so 4-bit weights turn the whole 16 GB card into KV + concurrency headroom. On a single power-capped RTX PRO 2000 it sustains 24 k tok/s aggregate; two cards run as independent replicas for 46 k tok/s — at ~330 tokens per joule.
vLLM 0.22.0, --quantization modelopt, --kv-cache-dtype fp8, --max-model-len 16384, 256-token pinned decode (ignore_eos+min_tokens), FlashInfer CUTLASS NVFP4 GEMM.
| config | in-flight | aggregate tok/s | per-stream | tok/joule |
|---|---|---|---|---|
| 1 GPU · single stream | 1 | 212 | 212 | — |
| 1 GPU · C256 (knee) | 256 | ≈ 24,200 | ~95 | ~346 |
| 2 GPU · 2× TP1 replicas (DP) | 512 | 🥇 46,020 | ~90 | ~329 |
-c 20480 / --max-model-len 20480, fp8 KV, out=256)| TP | GPUs | board W | max concurrency @20K ctx | KV pool | single stream (C1) | aggregate peak | tok/joule (agg) |
|---|---|---|---|---|---|---|---|
| 1 | 1 | 70 | 91× | 1.87 M tok | 213 | 24,287 (C256) | ~347 |
| 2 | 2 | 140 | 185× | 3.78 M tok | 352 | ~24,200 (C384) | ~173 |
| 4 | 4 | 280 | 409× | 8.39 M tok | 518 | 25,387 (C256) | ~91 |
Aggregate curves (tok/s): TP1 → C64 11,435 / C128 17,886 / C256 24,287 · TP2 → C64 12,798 / C128 18,510 / C384 24,241 · TP4 → C64 13,531 / C128 22,056 / C256 25,387.
Read it as: tensor-parallel scales single-stream latency (213 → 352 → 518 tok/s) and KV concurrency (91× → 185× → 409×, the model shards so each card adds its 16 GB to the KV pool) — but the aggregate plateaus at ~24–25 k tok/s regardless of TP, because one power-capped (70 W) card already saturates there and TP just spreads the same work and adds a PCIe all-reduce tax (no NVLink). So TP buys latency + KV headroom, not throughput. For raw throughput run data-parallel instead (N independent TP1 replicas): 2 replicas → 46,020 tok/s @ 140 W (329 tok/J), ~2× what TP=2 delivers at the same power. Rule of thumb: TP for latency/long-context concurrency, DP for max aggregate throughput & efficiency. TP>1 on a no-NVLink box needs NCCL_P2P_DISABLE=1 NCCL_CUMEM_ENABLE=0 … --disable-custom-all-reduce.
--gpu-memory-utilization 0.9)| weights on GPU | 1.13 GiB |
| KV cache (fp8, 16 K ctx) | 10.69 GiB → 1,865,284 tokens |
| max concurrency @ 16 K ctx | ≈ 114× |
Greedy spot-check (temp 0): "The capital of France is" → "Paris."; 「機械学習における量子化とは?」 → 「…精度を低減し数値の範囲を圧縮して計算効率を向上させる手法です。浮動小数点の値を低ビット幅の整数に変換することで、モデルの軽量化と推論速度の高速化を実現します。」 — fluent, accurate Japanese; code & instruction-following preserved.
A conservative, dense-lfm2-aware recipe — only the SwiGLU MLP is taken to 4-bit:
| kept in BF16 | taken to NVFP4 (W4A4) |
|---|---|
embed_tokens (tied lm_head), all RMSNorms | feed_forward.w1 / w2 / w3 — every one of the 16 layers |
self_attn.{q,k,v,out}_proj (6 attention layers) | |
conv.{in_proj, out_proj} + conv.conv (10 short-conv layers) |
w1+w3 → w13 internally (MergedColumnParallelLinear) and merges the per-shard NVFP4 scales — the stock checkpoint key names are used as-is, no remap needed (unlike the MoE sibling).conv.conv (Conv1d, K=3) is explicitly excluded — block-scaling a 3-tap kernel is meaningless and the weights are negligible.Note — use this W4A4 build, not weight-only. A W4A16 (weight-only) NVFP4 export of this architecture is not servable by vLLM: modelopt still writes a static-FP4 input spec into the config but omits the
input_scaletensors, so vLLM reads a garbage activation scale and emits empty output. This W4A4 build carries the input scales and runs correctly.
# Single GPU — serve
vllm serve LFM2.5-1.2B-JP-202606-NVFP4 \
--quantization modelopt \
--kv-cache-dtype fp8 \
--max-model-len 16384 \
--max-num-seqs 512 \
--gpu-memory-utilization 0.90 \
--port 8000
from openai import OpenAI
c = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="x")
r = c.chat.completions.create(
model="LFM2.5-1.2B-JP-202606-NVFP4",
messages=[{"role": "user", "content": "富士山について3文で教えて。"}],
temperature=0.3, max_tokens=200,
)
print(r.choices[0].message.content)
Max aggregate throughput — run one replica per GPU and load-balance across them
(data-parallel), e.g. CUDA_VISIBLE_DEVICES=0 … --port 8000 and CUDA_VISIBLE_DEVICES=1 … --port 8001.
Needs an SM ≥ 75 (Blackwell here) for the NVFP4 GEMM kernels.
LICENSE). All credit for the model itself goes to Liquid AI; this repo only changes the numeric precision.modelopt 0.44.0. arXiv: 2511.23404.