ceed-b2

CEED B2 — gemma-4-e4b-it, distilled from a sparse MoE teacher

A LoRA fine-tune of google/gemma-4-e4b-it trained with cross-entropy plus top-k logit distillation from the teacher. The teacher is the sparse mixture-of-experts google/gemma-4-26b-a4b-it.

The adapter has been folded into the base weights, so this is a standalone checkpoint: load it exactly like the base model, with no PEFT and no CEED code.

This is Group B2 of the CEED study (Causal Expert–Evidence Distillation), a research artifact published for reproducibility. It is not a product.

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained("gnitoahc/ceed-b2", dtype="float16")
processor = AutoProcessor.from_pretrained("gnitoahc/ceed-b2")

The model was trained and scored with a short-answer instruction in the prompt. Without it an instruction-tuned model answers "The total written in the image is **28**." against gold "28" and scores zero on every metric here.

Training

Evaluation

Scored by CEED's own harness (harness_version: ceed-direct-1) with greedy decoding, on CEED's own 10% validation split.

These numbers are not comparable to published DocVQA / GQA / ChartQA leaderboard results. Different splits, different prompt, different decoding. They are meaningful only against the other CEED Groups, which were scored identically.

Limitations

• This is a LoRA result. Merging folds the adapter into the weights; it does not turn a rank-4 adapter into a full fine-tune. CEED's own ADR-0005 bars LoRA numbers from the study's headline table, because a null result under a small adapter cannot be attributed between "the signal does not transfer" and "the adapter lacked the capacity to hold it". Read any comparison involving this checkpoint with that in mind.
• The distillation gain is not established. The no-teacher control (CEED B1), trained identically but with kd_weight: 0, scored above this checkpoint on every dataset (docvqa 0.8506 vs 0.8798; gqa 0.6191 vs 0.6959; chartqa 0.5783 vs 0.7871). Whatever this checkpoint's objective contributes, it is not visible as an advantage over supervised fine-tuning here.
• Trained on document, natural-image and chart VQA in English only. Behaviour outside that is untested.
• Inherits the base model's limitations and the Gemma licence.

ceed_provenance.json beside the weights carries the source run's identity, parameter-efficiency mode, and metrics.