Vintern-1B-v3_5-BF16-GGUF

Vintern-1B-v3.5 — corrected BF16 GGUF

A community conversion of 5CD-AI/Vintern-1B-v3_5, with corrected end-of-sequence metadata and a runtime-specific vision position-embedding layout. Original model and training credit belong to 5CD-AI. This is not a new fine-tune, and no low-bit weight quantization was applied. Some tensors remain F32.

Runtime compatibility: tested with llama.cpp commit c550d2f60bde72df19fcef1fef627895095b8ba8 (build 11096). The supplied projector is for that implementation's CLS-last InternVL layout. A runtime that prepends CLS instead requires the original, unrotated projector. Do not combine this projector workaround with a runtime fix that switches to CLS-first. Compatibility with other versions has not been established.

Files

SHA-256 hashes are in SHA256SUMS. Both GGUF files and the template are required for the tested setup.

Why this conversion exists

The original checkpoint stores language metadata in llm_config. The pinned converter searched the root and text_config, overlooking its EOS ID. The resulting GGUF omitted EOS metadata; the runtime defaulted to token 11 (comma), causing severely truncated OCR. The included converter patch preserves root/text_config precedence and adds llm_config lookup. The corrected artifact embeds <|im_end|> / 151645.

The original InternVL vision embedding prepends CLS, with position rows [CLS, patches...]. The pinned llama.cpp implementation appends CLS. This projector rotates only v.position_embd.weight from [CLS, patches...] to [patches..., CLS], preserving all values and patch order.

Conversion uses ImageNet normalization, 448px tiles and a maximum of 12 dynamic patches, following the upstream inference example. Upstream base model is InternVL2.5-1B.

Download for llama-router

Download the two GGUF files and chat template together into your router's model directory:

uv tool run --from huggingface-hub hf download duyleekun/Vintern-1B-v3_5-BF16-GGUF \
  Vintern-eos-fixed-BF16.gguf mmproj-Vintern-cls-last-BF16.gguf \
  vintern-exact.jinja SHA256SUMS --local-dir ./models/vintern-1b-v3_5-bf16
(cd ./models/vintern-1b-v3_5-bf16 && sha256sum -c SHA256SUMS)

For a router with the model directory mounted at /models, add this preset:

[vintern-1b-v3_5-bf16]
model = /models/vintern-1b-v3_5-bf16/Vintern-eos-fixed-BF16.gguf
mmproj = /models/vintern-1b-v3_5-bf16/mmproj-Vintern-cls-last-BF16.gguf
chat-template-file = /models/vintern-1b-v3_5-bf16/vintern-exact.jinja
ctx-size = 8192
jinja = true

Adjust paths to your installation. The projector and template are required alongside the language-model GGUF. Use the compatible runtime described above.

Run

Build llama.cpp at the pinned commit with CUDA support, then run from the downloaded model directory:

./llama-server \
  --model Vintern-eos-fixed-BF16.gguf \
  --mmproj mmproj-Vintern-cls-last-BF16.gguf \
  --chat-template-file vintern-exact.jinja --jinja \
  --ctx-size 8192 --parallel 1 --n-gpu-layers 99 \
  --flash-attn on --host 127.0.0.1 --port 8080

Send an image to the OpenAI-compatible endpoint using the included standard-library client:

uv run --no-project python scripts/ocr.py /path/to/document.jpg

This uses temperature 0, repeat penalty 1.1, a 2,048-token output limit and the upstream Vietnamese system prompt. Long pages may require a higher output limit. Review outputs against the source image; OCR can omit, alter or invent text.

Reproduction and provenance

See REPRODUCE.md for conversion steps and compatibility details.

• Original checkpoint: 5CD-AI/Vintern-1B-v3_5, revision b98f263eab246eb5269ade64edbdca8a887dc44d.
• Converter/runtime: llama.cpp at the pinned commit.
• Upstream model license: MIT, as declared by its model card. The source repository at the pinned revision does not provide a separate LICENSE file. See upstream card and llama.cpp license.
• Source model lists Vietnamese, English and Chinese.