RaceBench-MiniCPM5-v1.1

RaceBench-MiniCPM5-v1.1

<div align="center"> <img src="https://photu.kashyalabanavli.site/racer-is-op.png" alt="RACER IS OP" width="100%"> </div>

Quality over quantity — and v1.1 is the receipt.

Full-parameter fine-tune of openbmb/MiniCPM5-1B on RaceBench-v1.1 (376,736 rows: 353,456 distilled + 23,280 premium agent traces at 4x upsample).

The honest headline: v1.1 does NOT beat base. The premium-agent experiment reversed v1.0's reasoning gain (BBH -9.4, z=-12.3) while keeping a smaller math regression (GSM8K -5.7, z=-3.0). The 4x-upsampled long agent traces look harmful to a 1B model. This is a negative result, published honestly - and it cleanly redirects v1.2.

Results: RaceBench-MiniCPM5-v1.1 vs base

lm-evaluation-harness v0.4.12, identical settings head-to-head (fp16, seed 42, single NVIDIA H100). Deltas in points; z = delta / pooled SE; |z| > 2 is significant at ~95%.

Reading this correctly

• Significant regressions: BBH -9.4, GSM8K -5.7, MMLU -1.6. Nothing improves significantly.
• Do not compare these absolutes to the v1.0 model card table - that run used different task configs (zero-shot BBH variant, MMLU subset, 2xT4 venue). Within THIS run, both models were evaluated identically, so deltas are apples-to-apples.
• vs v1.0 directionally: v1.0 delivered BBH +2.7 / GSM8K -9.9. v1.1 delivers BBH -9.4 / GSM8K -5.7. Adding 4x-upsampled premium traces traded the reasoning gain away for a smaller math hit. Bad trade at 1B scale.

Why this happened (hypothesis)

The premium agent traces average 46 turns / ~2.7k chars (max 135k). At 1B scale, 6.2% of training mass in very long multi-turn traces appears to have taught verbose drift, not planning. Per the pre-registered decision tree this is Branch B: ablate premium threshold (>=0.85 or >=0.90 instead of >=0.8), cap trace length (<8192 chars, <=12 messages), re-test.

Deployment implications (read before adopting)

• Do not use v1.1 over the base model today. Every significant delta is negative.
• Use saidutta69/RaceBench-MiniCPM5 (v1.0) if you want the reasoning-gain checkpoint - with its documented GSM8K cost.
• IFEval/HumanEval/MBPP were attempted on H100 and are pending harness fixes; not reported here and no claims are made about instruction-following or code.

Training details

• Base model: openbmb/MiniCPM5-1B
• Dataset: saidutta69/RaceBench-v1.1
• Method: full-parameter SFT, seq_len 4096, sample packing, assistant-only loss masking
• Precision: bf16, SDPA attention, adamw_bnb_8bit
• Schedule: lr 2e-05 cosine, warmup 100, wd 0, seed 42, effective batch 8 x 4096 tokens (32k tok/step), 1 epoch = 10,445 steps, final train_loss 1.301
• Data volume: 376,736 rows -> 342.2M tokens -> 83,557 packed sequences
• Hardware: NVIDIA H100 80GB, ~3h wall

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "saidutta69/RaceBench-MiniCPM5-v1.1",
    trust_remote_code=True,
    torch_dtype="bfloat16",
)
tokenizer = AutoTokenizer.from_pretrained("saidutta69/RaceBench-MiniCPM5-v1.1")

License

Apache-2.0 (base model weights); RaceBench data is MIT.

Citation

@misc{racebench-minicpm5-v11,
  author = {Sai Dutta},
  title = {RaceBench-MiniCPM5-v1.1 -- Premium-Agent Ablation Checkpoint},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/saidutta69/RaceBench-MiniCPM5-v1.1}}
}