molmo2-codec-pruning16-step1000

Molmo2 Codec — Native Pruning-16 Step 1000

This is the first dedicated native-cell Pruning-16 checkpoint for Molmo2 Codec. It is intentionally separate from the learned P-tokenizer and historical fast_gray2 repositories.

What is in this release

• weights/stage2_step1000_consolidated.pt: complete 4B Stage-2 model state.
• config/stage2_step1000.yaml: exact training configuration.
• config/native_p_tokenization.json: native Pruning-16 architecture contract.
• config/gamma_v6_rgbhex_k4.json: RGB-HEX/K4 GOP contract.
• eval/mlvu_n200_summary.json: fixed-ID MLVU Dense/Codec result.
• eval/multibench_n200_summary.json: fixed-ID six-benchmark paired suite.
• SHA256SUMS: byte identity of every published artifact.

There is no Stage-1 P-tokenizer file. P-frames use the frozen native Molmo ViT, score the 81 native connector cells with robust motion/residual saliency, keep 16 cells in raster order, and pass them through an independent native connector copy stored in the Stage-2 checkpoint.

Required input contract

Do not combine this checkpoint with fast_gray2 caches or learned-P tokenizers. Motion mode, gamma, FPS, timeline cap, visual budget and K are cache identity fields.

Evaluation

The release was evaluated on frozen, identical 200-example manifests. Dense and Codec share the Stage-2 checkpoint, prompt, source video and answer scorer; only visual tokenization differs. Exact results are stored in eval/ and in the repository performance record.

The +1.0-point MLVU difference is a paired diagnostic on one frozen 200-item sample, not a claim that compression generally improves accuracy. Consult the six-benchmark summary for the broader Dense/Codec gap.

Every Dense/Codec row passed exact paired-document-ID verification. LVBench is kept in the formal six-task mean and is the largest remaining regression.

Usage

hf download weikaih/molmo2-codec-pruning16-step1000 \
  --local-dir checkpoints/molmo2-codec-pruning16-step1000

export CODEC_P_TOKENIZATION=pruning16
export CODEC_MOTION_MODE=hex
export CODEC_GOP_TARGET_P=4
export CODEC_GOP_MAX_P=4
export CODEC_GOP_PROFILE_CACHE=k4-rgbhex-v2
export CODEC_GOP_TIMELINE_MAX_FRAMES=2048
export CODEC_GAMMA_ARTIFACT=$PWD/codec/configs/gamma_v6_rgbhex_k4.json

Use the molmo2_codec adapter in the linked lmms-eval fork. Native Pruning-16 requires the Stage-2 checkpoint and gamma artifact, but no ptok_checkpoint argument.

accelerate launch -m lmms_eval \
  --model molmo2_codec \
  --model_args \
training_repo=/path/to/molmo2-codec,\
pretrained=$PWD/checkpoints/molmo2-codec-pruning16-step1000/weights/stage2_step1000_consolidated.pt,\
base_checkpoint=/path/to/molmo2-4b-sft-step30000,\
gamma_artifact=$PWD/checkpoints/molmo2-codec-pruning16-step1000/config/gamma_v6_rgbhex_k4.json,\
video_backend=codec,visual_token_budget=8192,timeline_max_frames=2048,seq_len=16384 \
  --tasks mlvu --batch_size 1

The evaluation adapter is maintained on the molmo2-codec-eval branch. Use its checked-in frozen manifests for a paired Dense/Codec comparison; changing only the visual backend while keeping document IDs and scoring fixed is part of the reported protocol.

This is a research checkpoint. It compresses P-frames from 81 to 16 visual tokens; it is not a claim of lossless visual reconstruction.