joecr-chandra-2-eagle3.1

joecr-chandra-2-eagle3.1

An EAGLE‑3.1 speculative‑decoding draft head for datalab-to/chandra-ocr-2 (a 5B Qwen3.5‑based vision‑language OCR model). Drop it into vLLM as the speculator to accelerate single‑stream (latency‑bound) OCR decoding losslessly — every drafted token is verified by the target, so output quality is preserved.

Results

Speedup (single‑stream, greedy, vLLM + CUDA graphs, RTX 3090 Ti):

Measured on olmOCR‑bench pages.

We also measured an average acceptance length of ~2.5 tokens.

Batching note: this speculative decoding head wins at low concurrency. At large batch sizes disable speculative decoding or drop num_speculative_tokens to 1–2 for bulk throughput.

Architecture

EAGLE‑3.1 (single decoder layer) over Chandra‑2's Qwen3.5 text backbone:

• hidden_size 2560, head_dim 256, GQA 16/4
• EAGLE 3.1: fc_norm (per‑aux‑layer RMSNorm) + norm_output (post‑norm recurrence)
• Vocab pruning: draft_vocab_size 32768 (from 248320; 99.99% token coverage) → ~7.6× smaller lm_head. d2t buffer maps pruned ids back to the full vocab.
• Aux hidden states from the target's full‑attention layers [3, 15, 27] (Chandra‑2 is a hybrid 3:1 linear/full‑attention model).

Usage (vLLM)

from vllm import LLM
llm = LLM(
    model="datalab-to/chandra-ocr-2",
    speculative_config={
        "model": "jbarrow/joecr-chandra-2-eagle3.1",
        "method": "eagle3",
        "num_speculative_tokens": 5,
    },
    limit_mm_per_prompt={"image": 1},
)

Use the Chandra ocr_layout prompt in the user turn (image + instruction).

Training

• On‑policy data: Chandra-2 generated Qwen-HTML over ~50k document pages.
• Objective: forward‑KL distillation with TTT (length 7), in vLLM's EAGLE input‑shift convention.
• Length: 2 epochs on 4× RTX 3090 Ti.