MiniCPM5-2B-LiteRT-LongContext

<!-- MOBILE_INT8_VALIDATION_20260913 -->

Corrected mobile CPU/GPU models and SM8750 NPU artifact

The Mobile INT8 files fix the tested Adreno GPU initialization failure and use INT8 attention, MLP, embedding, and output-head weights, with FP16 activations and KV cache. Stored precision is audited. These are not 4-bit or lossless files.

For this REDMAGIC 10 Pro, use Mobile INT8 16k for GPU. All three GPU capacities initialized and answered the arithmetic control correctly on CPU and GPU using Gallery 1.0.19's actual native runtime. The corrected 16k model also passed the Gallery chat UI check (answer 4, displayed 1.4 seconds, excluding initialization). The 64k GPU control took about 331 seconds including initialization and places much more pressure on this phone's 12 GB RAM. Allocated context capacity is not a full-length retrieval result.

NPU: device-verified within the recorded checks

The SM8750 file is A16W8, 1,024-token context, one-token prefill. It passed five short checks on the physical REDMAGIC 10 Pro / NX789J / Android 16 using Gallery's exact native runtime and Qualcomm HTP libraries: addition, multiplication, Chinese and English subtraction, and recall of a code after a 307-token prompt. The last three were also checked against fresh BF16 outputs. Actual HTP dispatch is recorded; tokenization and some auxiliary work use CPU. The public LiteRT-LM 0.17 SDK also passed the Chinese regression control. The imported model also passed the actual Gallery NPU chat screen check.

Two repairs were necessary. Grouped causal-mask repetition compiled incorrectly; explicit-head reshapes and a broadcast ADD correct it without changing any original weight bytes. Unused last-layer prefill cache inputs then caused the older runtime to reuse CPU memory for NPU decode. Removing those unused inputs makes it allocate caches from decode's NPU requirements, without changing QNN code. Disabling the newer runtime's cache-allocation fix reproduced the older runtime's exact wrong answer, confirming that second cause.

The one-token prefill is a conservative compatibility choice and is slower on long prompts. A faster 64-token-prefill variant failed the Chinese control and is excluded. This release does not claim NPU support for other SoCs, long context beyond 1,024 tokens, broad BF16 quality parity, or lossless conversion. NPU results and library hashes.

Quality evidence and canceled phone benchmark

The full BF16 GPQA Diamond run completed 198 distinct questions, with 83/198 correct (41.92%), 64 outputs reaching the 2,048-token cap, and 48 unparsed answers. Protocol: EvalScope 1.11.1, seed42, batch1, greedy decoding, thinking disabled. This differs from the upstream thinking-enabled model-card protocol.

The phone run was canceled at the user's request after 67 saved distinct predictions. No full phone GPQA score or paired accuracy delta is claimed. The benchmark is stopped. AutoBencher now enforces all 198 questions for full results, preserves validated resume records, rejects incomplete comparisons, and exposes token-count and missing-answer differences. Coverage status.

A separate host experiment compared the actual Mobile INT8 16k LiteRT CPU artifact with BF16 on 16 held-out prompts and 255 teacher-forced positions:

This demonstrates real generation differences. It is a host CPU comparison, not an Android GPU/NPU quality certificate. The independent precision recipe uses separate calibration and evaluation data and retains higher precision where proxy error is too large. It follows principles discussed in Unsloth Dynamic v3; it is an independent LiteRT recipe, not an official Unsloth port. The strict selection retained INT8 in all transformer groups, embedding, and output head.

Using the files

In Gallery's local import form, set compatible accelerators to CPU/GPU for the Mobile INT8 files, or NPU for the SM8750 file. For the NPU model, set the maximum context to 1,024 and start with thinking disabled. Restart Gallery after replacing an entry with the same filename so its capability metadata reloads.

The original GPU failure was a 2 GiB prefill attention allocation exceeding Adreno's 1 GiB per-buffer limit. The corrected GPU exports use 64-token prefill. An intermediate export accidentally retained an INT4 output head; the corrected exporter and independent vocabulary audit now prevent that. Historical files below retain their old limitations and do not certify the new artifacts.

Detailed evidence and reproduction · CPU/GPU device matrix · Host fidelity results · Checksums · AutoBencher changes

<!-- END_MOBILE_INT8_VALIDATION_20260913 -->

MiniCPM5-2B LiteRT — 16K / 32K / 64K context

Long-context LiteRT-LM (.litertlm) conversions of openbmb/MiniCPM5-2B. The upstream checkpoint supports up to 131,072 positions; this repository provides three separate fixed KV-cache builds at exactly 16,384, 32,768, and 65,536 tokens.

<!-- DYNV31A_AUDITED_VALIDATION_20260912 -->

Experimental DynV31A: completed sampled BF16 comparison

Validation update (2026-09-12): not lossless on this diagnostic; statistical near-losslessness is not established. All three DynV31A artifacts lose the same two BF16-correct IFEval cases, with zero gains. Maths and GPQA correctness match question by question on the small samples below. These results apply only to experimental/dynv31a-2026-09-11/, not the older root INT4 or DynV3Mixed files.

Nine of nine planned artifact/benchmark comparisons are complete and audited, not the full model-card suite. There are 21 distinct fully paired prompts, not 63 independent questions; the larger BF16-only maths run brings the primary record to 99 scored responses across 36 distinct prompts. These are deterministic development slices, not a random sample or a claim of newly held-out evaluation. Percentages must not be generalized to full-benchmark or population-level degradation.

The added failures are required prompt repetition (IFEval key 1012, below the output cap on both sides) and required keywords (key 1069, both sides at the cap). Instruction-level strict accuracy on the same 20 constraints is 90% for BF16 versus 80% for each DynV31A build. The corrected BF16 IFEval denominator is 12: a missing NLTK resource initially caused one response to be omitted, showing 10/11 (90.91%); unchanged cached outputs re-scored with the resource installed give 10/12 (83.33%).

Runs use seed 42, temperature 0, top-p 1, thinking disabled and batch size 1. Prompt hashes, target hashes, input-token counts, generation budgets, output model identities and scored counts are checked. BF16 uses vLLM/CUDA; LiteRT uses eight CPU threads, so this is end-to-end conversion/runtime/quantization validation, not proof that weight quantization alone caused every difference. It is not a speed comparison. MATH uses the documented first-item-per-difficulty scope amendment made after BF16 generation but before candidate maths generation; only those same five BF16 responses form the paired baseline.

In the separately recorded 24,114-token NoLiMa replay, both fresh BF16-vLLM and DynV31A 32K returned the same wrong answer. That one case is not evidence of a quant-only regression or of general long-context parity. Short questions in a 64K-capacity file do not validate 64K retrieval. Cap-hit outputs and the separate 512-token-budget experiment are reported separately in the evidence.

Experimental files and checksums · Detailed report · Paired scores CSV · Complete evidence ZIP · Validation index and reproduction notes

All nine existing .litertlm files in this repository are unchanged by this documentation/evidence update. The DynV31A family remains experimental and is not an official Unsloth quantization or GGUF.

<!-- END_DYNV31A_AUDITED_VALIDATION_20260912 -->

Historical plain INT4 quality summary

Plain INT4 summary (2026-09-10; not DynV31A): direct / thinking-off quality is close to the original BF16 checkpoint (90/100 vs 91/100 on the first 100 GSM8K test questions), and four-needle retrieval passed at ~14K, ~28K, and ~56K accumulated context. Thinking-on is not at BF16 parity in INT4: a matched 2,048-token reasoning test failed to finish and produced the wrong extracted answer, while BF16 finished correctly. For these INT4 files, thinking-off is currently the recommended mode when correctness and predictable latency matter.

Earlier DynV3Mixed mixed-precision variants (not DynV31A)

BF16-relative fidelity update (2026-09-11): the mixed-precision quants retain all six clean BF16-first frozen short-context source-success checks, but they are not lossless: a matched AutoBencher GPQA-Diamond item falls from BF16 100% to 0% on all three quants, and the selected NoLiMa long-context case also degrades relative to BF16. See the BF16-relative sections below.

A second set of three .litertlm files is provided using a Dynamic-v3-inspired LiteRT PTQ adaptation. These are not official Unsloth Dynamic v3 GGUFs. Unsloth publicly describes Dynamic 3.0 as using a higher-quality diverse imatrix calibration set, improved layer selection, additional quantization techniques, pure post-training quantization, and held-out divergence checks. The exact complete model-agnostic layer-selection recipe is not published, so this LiteRT port adapts those principles to the quantization controls LiteRT exposes.

Dynamic-v3-inspired files

Each mixed file is 1,878,673,760 bytes, about 20.3% larger than the plain INT4 files. The tradeoff is intentional: 411 sensitive weight tensors are kept at INT8 while 1,618 remain INT4; embedding and LM head are also protected at INT8. The underlying graph topology is unchanged.

Calibration and layer selection

• 48 calibration-only prompts spanning code, mathematics, instruction following, science/general knowledge, tool/agent workflows, multilingual text, and long/document inputs.
• Calibration prompts are separate from the benchmark questions below.
• Input second moments were captured for attention (q/k/v/o) and MLP (gate/up/down) projections.
• An activation-weighted INT4→INT8 error-reduction proxy selected sensitive layers for INT8 rescue.
• Attention INT8 layers: 0, 33, 34, 35, 37, 38, 39, 40, 41.
• MLP INT8 layers: 0, 7, 35, 36, 37, 38, 39, 40, 41.
• Remaining linears: blockwise-32 INT4 with OCTAV clipping.

The full sensitivity map is in dynamic_v3_sensitivity.json; reproduction scripts are under dynamic_v3_reproduction/.

Creator-benchmark category retention check

The upstream MiniCPM5-2B card reports benchmark families covering Code Reasoning, Math Reasoning, Instruction Following, General Knowledge, Long Context, Tool Use, Coding Agent, Search Agent, and General Agent. One reported benchmark was selected from each category: LiveCodeBench v6, MATH-500, IFEval, MMLU-Pro, NoLiMa, BFCL v4, SWE-bench Verified, GAIA Text-103, and Claw-Gym.

For the eight non-long-context categories, this repository uses a BF16-first frozen retention protocol rather than pretending a one-item local slice is a full leaderboard run. The BF16 source model was evaluated first; a source-passing item was frozen before any quant result was inspected, then the identical prompt/decoding settings were run on all three LiteRT files. enable_thinking=false and greedy decoding were used to make the comparison deterministic. SWE-bench is a bounded semantic-patch proxy where BF16 itself fails; GAIA uses identical frozen search evidence rather than live browsing; Claw-Gym is a bounded content proxy rather than the official tool-loop reward.

On this frozen eight-category slice, all three mixed LiteRT builds exactly retained every BF16-success case: the same LiveCodeBench program passed all tests, MATH-500 returned 14/3, IFEval passed strict evaluation, MMLU-Pro returned the same F, BFCL emitted the accepted tool call, and GAIA returned 34689. SWE-bench remained a source-model failure in the bounded proxy, and Claw-Gym remained 8/9 content checks on BF16 and all three quants. The observed bounded mean is therefore unchanged at 0.8611 for BF16, 16K, 32K and 64K.

This means no observed degradation on the frozen source-success cases; it does not mean these one-item checks reproduce the upstream full-benchmark scores. See dynamic_v3_benchmark_results.json for exact prompts, item IDs, timings, and scoring notes.

AutoBencher BF16-relative sampled fidelity check (2026-09-11)

The three Dynamic-v3-inspired artifacts were exercised through the Rust AutoBencher pipeline and compared against the original BF16 checkpoint on the same selected items and deterministic decoding settings. This is a targeted fidelity diagnostic, not a formal leaderboard reproduction. Thinking was disabled; decoding was greedy (temperature=0, top_p=1), seed 42, evaluator batch size 1.

The GPQA comparison was rerun specifically to avoid a misleading baseline. With a 512-token ceiling the BF16 response was truncated and initially scored 0%; at a 2,048-token ceiling BF16 completed naturally after 1,447 output tokens and selected the correct 10^-4 eV answer. All three Dynamic-v3 artifacts were then rerun with the same 2,048-token ceiling. Each stopped naturally after 401 output tokens, scored 0%, and produced the same wrong output byte-for-byte. This is therefore an observed quantization-fidelity regression rather than a timeout or output-budget artifact.

The two fresh BF16-passing AutoBencher diagnostics retain 1/2 items (50%) for each quant. That percentage must not be generalized to the full benchmarks; it is only a two-item diagnostic. In parallel, the earlier BF16-first frozen retention suite retained all six clean source-success short-context checks (LiveCodeBench v6, MATH-500, IFEval, MMLU-Pro, BFCL v4, GAIA Text-103) on every quant (6/6). Together these results show that several selected short-context behaviors are preserved while at least one reproducible reasoning regression remains.

A full 34-row AutoBencher dry-run was also completed for every artifact: all 34 model-card rows were traversed, 20 had bundled runnable plans, 14 remained explicitly blocked where exact external reproduction prerequisites are unavailable, and there were no dry-run infrastructure failures/timeouts.

LiteRT-LM 0.17.0 reads OpenAI max_completion_tokens while EvalScope supplies legacy max_tokens; the AutoBencher integration uses a small compatibility proxy that mirrors the field. A validation request capped at 16 tokens returned exactly 16 completion tokens. Full BF16-relative provenance, artifact SHA-256 values, run IDs, response hashes, and caveats are in benchmarks/autobencher/2026-09-11/REPORT.md and results.json. Runtime numbers are not used for BF16-vs-LiteRT performance claims because BF16 ran on RTX 4090/vLLM while LiteRT claims used XNNPACK CPU.

NoLiMa long-context result

A hard NoLiMa case (0408Inv_T04_C02_twohop) was selected from a BF16-only scan before quant outputs were inspected. It asks which character has been to Madrid; the exact gold answer is Yuki.

The 16K/32K failures are real observed long-context quality regressions and are intentionally preserved rather than tuned away. For 64K, the source model also passed a BF16-only 36K selection point, but the stock LiteRT-LM 0.17 XNNPACK runtime could not finish either the 36,114- or 40,114-token 64K prefill within its fixed 10-minute session limit. Those runs are runtime timeouts, not scored as wrong model answers. The supplemental 64K/12K run does complete and returns Rebecca, so the same NoLiMa case is not retained there either.

The BF16-only length scan was non-monotonic for this needle/haystack construction: 30,114, 32,114 and 34,114 tokens returned Van, while 36,114 returned Yuki. This is why only source-passing lengths are used for quantization-retention claims. The scan is included in dynamic_v3_benchmark_results.json and the reproduction helper is under dynamic_v3_reproduction/scan_nolima_bf16_lengths.py.

The stock 64K LiteRT WebGPU path was also tested and could not initialize this cache geometry because the selected backend reports a 128 MiB maximum single buffer, smaller than the required 64K KV allocation. CPU initialization/inference still passes (OK) on all three mixed files.

Files

All variants use the same prefill-signature ladder: 1024, 256, 64, 16, 4, 1.

Benchmark results

Benchmarks below compare the converted artifacts against the original openbmb/MiniCPM5-2B BF16 checkpoint. They are intended as conversion-parity checks, not claims of full benchmark coverage.

GSM8K — thinking off

Protocol: first 100 examples from the official GSM8K test split; zero-shot step-by-step suffix; the upstream MiniCPM5 chat template; enable_thinking=false; greedy decoding; maximum 2,048 new tokens; identical numeric answer extraction on both sides.

Paired analysis across the same 100 examples:

• 91/100 predictions were exactly identical between BF16 and LiteRT INT4.
• BF16-correct → INT4-wrong occurred on only 2 examples: IDs 7 and 37 (160→120, 2→0).
• INT4 corrected one BF16 miss on ID 41 (400→200, gold 200).
• Eight examples were wrong for both: IDs 4, 12, 21, 62, 75, 87, 89, 93.
• No LiteRT response in this 100-example thinking-off run was unfinished.

The sensitive cases were replayed on the larger-cache builds. The 32K build reproduced the 16K decisions on all eight replayed controls; the 64K build reproduced the same three decisive outcomes on IDs 7, 37, and 41. This is evidence that the observed short-context differences come from the shared INT4 quantization, not from the 16K / 32K / 64K KV-cache compilation size.

Long-context retrieval

A synthetic four-needle retrieval test placed four independent access codes at approximately 8%, 35%, 65%, and 90% of the archive. The model was asked to return all four codes exactly. BF16 recovered 4/4 at every tested length.

The expected answer was recovered exactly at every scored LiteRT length:

A=K4M7Q2; B=V9R2L6; C=H3T8N5; D=P6W1X9

The 64K artifact was additionally tested with a single 56,000-token message, but LiteRT-LM 0.17's stock engine has a hard 10-minute per-message/session completion timeout and XNNPACK did not finish the prefill before that deadline, even with 16 CPU threads. To test the KV cache rather than the timeout, the same archive was supplied in six sequential chunks in one persistent conversation. The final ~56K accumulated-context retrieval was 4/4, demonstrating functional retention beyond 32K and near the intended 64K operating range.

Intermediate acknowledgement text in that staged 64K run became mildly noisy after later turns (CONTINUEgJC, etc.), although the final four-code retrieval remained exact. This should be treated as a long-history generation-quality caution rather than a retrieval failure.

Thinking / reasoning mode

INT4 is not currently at BF16 parity for long reasoning trajectories.

Matched test: GSM8K item 0, enable_thinking=true, greedy decoding, 2,048-token output budget.

BF16 completed the reasoning chain and emitted the correct final answer. The INT4 artifact used the entire budget inside a substantially longer reasoning trajectory, never emitted final answer text, and the best number extract from the unfinished thought was wrong.

A second thinking-on attempt with a 3,584-token output budget also exposed a practical runtime problem: the first LiteRT INT4 item did not finish before LiteRT-LM 0.17's hard 10-minute session deadline. The equivalent BF16 five-item run produced recoverable/correct answers on 5/5 (four with normal final-answer completion; one answer recoverable from a reasoning chain that itself reached the generation cap).

Recommendation: use enable_thinking=false for these INT4 artifacts when you need the most reliable behavior. A higher-precision long-context conversion (for example INT8 / mixed precision) is a better candidate for reasoning-heavy use.

Benchmark environment and interpretation

• BF16 reference: upstream openbmb/MiniCPM5-2B, PyTorch BF16 on NVIDIA RTX 4090 / CUDA.
• LiteRT artifact tests: LiteRT-LM 0.17.0 on WSL2 / XNNPACK CPU; long-context 28K/56K tests explicitly used 16 CPU threads.
• The WSL LiteRT gpu backend was not used for benchmark claims because the current WebGPU path selected Mesa llvmpipe (software) rather than the RTX 4090.
• Runtime timings across BF16 CUDA and LiteRT XNNPACK are therefore not comparable performance measurements. Accuracy, completion, and retrieval are the metrics reported here.

Conversion

• Format: LiteRT-LM .litertlm
• Quantization: BOCTAV4 — blockwise-32 INT4 + OCTAV clipping for linear weights, with INT8 embedding
• Embedder: externalized
• Prompting: upstream chat_template.jinja is preserved through the MiniCPM5 LiteRT Jinja path, and the bundle declares a thought channel. The format supports the upstream thinking switch, but the benchmark results above show that INT4 thinking quality is not at BF16 parity.
• Start/stop behavior: inherited from the upstream MiniCPM5 checkpoint (<s> start; EOS IDs 1 and 130073)
• Context cache lengths: exact powers of two (16384 / 32768 / 65536)

MiniCPM5 contains all-zero rows in layer 0 MLP weights. The INT4 block quantizer emits zero scales for those blocks, which XNNPACK refuses. Each bundle therefore applies the established in-place zero-scale repair: 1,664 zero fp16 scales are replaced by the smallest positive scale already present in the affected scale tensor. Exactly 3,328 bytes change; quantized block values are zero, so this does not change their dequantized weight values.

Toolchain

Built on 2026-09-10 in WSL2 using current toolchain revisions at conversion time:

• Python 3.13.13
• litert-torch 0.10.0 (6d4c622c9a3aade4d411a1e859c6bbe571d38ee0)
• LiteRT-LM 0.17.0 (0d43b55250dfd2a69d64b761e238f2fc4b12c959)
• litert-lm-builder 0.17.0
• litert-converter 0.4.0
• ai-edge-quantizer-nightly 0.10.0.dev20260910
• torch 2.14.0
• transformers 5.17.0
• huggingface-hub 1.31.0
• conversion/reproduction reference a72dd53df14f50e56b6115081406d4415e88ab38

See litertlm_manifest.json, validation_report.txt, and convert_minicpm5_longcontext.sh for conversion reproducibility. The benchmark results above were produced after the initial upload and supersede the original smoke-test-only quality statement in validation_report.txt.

Runtime notes / known limitations

• These files change the compiled KV-cache capacity; they do not fine-tune or extend the upstream model's learned context behavior.
• The 64K build remains below the upstream 131,072-position limit.
• Long-context memory and compute costs increase with KV-cache length. The 32K and especially 64K artifacts can be expensive on CPU.
• LiteRT-LM 0.17 currently uses a hard 10-minute engine/session completion timeout. On the tested WSL2 XNNPACK setup, a single 56K-token prefill exceeded that limit; staged persistent-context ingestion succeeded.
• After a separate thinking-on run hit the 10-minute timeout, LiteRT-LM also emitted an Abseil mutex-corruption fatal during cleanup. This occurred after the timed-out session and is treated as a runtime robustness issue, not evidence of corrupted model weights.
• Thinking-off is the recommended mode for this INT4 release. Reasoning-heavy use should prefer a higher-precision conversion when available.

License

Apache-2.0, following the upstream MiniCPM5-2B license. The weights are transformed/quantized from openbmb/MiniCPM5-2B; no fine-tuning was applied.