keural-vision-encoder-mid

Keural Mid-Level Vision Encoder

A 183.5M-parameter image encoder built from scratch — adaptive tokenization, hierarchical concept tokens, and continuous 2D positional encoding. Not a CLIP/ViT fine-tune.

<p> <img alt="params" src="https://img.shields.io/badge/params-183.6M-blue"/> <img alt="from scratch" src="https://img.shields.io/badge/trained-from%20scratch-success"/> <img alt="precision" src="https://img.shields.io/badge/precision-bfloat16-informational"/> <img alt="status" src="https://img.shields.io/badge/status-training%20complete-success"/> <img alt="pope" src="https://img.shields.io/badge/downstream%20POPE%20vs%20blind-%2B25.95-brightgreen"/> <img alt="mme" src="https://img.shields.io/badge/downstream%20MME%20vs%20blind-%2B446.9-brightgreen"/> </p>

Developed by MKD Co., Ltd. · Code: github.com/mkd-hika/Keural-Vision-Encoder-Mid-level

---

At a Glance

Measured at 384×384 with token_budget=512, matching training. Accuracy trails same-size peers — see Limitations and Evaluation for why, and for the higher-k retrieval results where the gap closes.

---

Downstream Results — Vision-Language Model

The encoder is frozen and used as-is inside a bilingual (English + Korean) vision-language model: mkd-hika/keural-mid-vlm-bilingual. Only a projector (8.9M) and LoRA adapters (5.0M) are trained on top; the decoder is Qwen2.5-7B-Instruct.

A vision-language model can score respectably while barely using the image, because many questions are answerable from language priors alone. Each benchmark below is therefore run twice — once normally, once with the visual tokens removed and everything else held constant. The difference is what this encoder contributes, and it is the only figure the language model cannot produce on its own.

The POPE yes-ratio is 0.492 against a gold ratio of 0.500, so the model genuinely discriminates on object-existence questions rather than defaulting to one answer — accuracy alone is not interpretable without that check. The blind VQAv2 score of 39.83% lands where language-prior performance is expected to, which is corroborating evidence that the measurement is sound.

Notably, the encoder reaches these numbers while scoring 37.2% zero-shot on ImageNet. Zero-shot accuracy measures image–text alignment; the 75.0% CIFAR-100 linear probe is the better indicator of representation quality for downstream use, because a trained projector relearns the alignment from scratch.

---

Model Description

Keural Mid is a mid-scale vision encoder that maps an image to a single 768-dim embedding aligned to a text embedding space, plus a variable-length sequence of semantically-typed tokens. It is trained from random initialization (no CLIP/ViT/DINO backbone) with a combination of sigmoid contrastive learning, hierarchical concept alignment, saliency regularization, and dual-teacher knowledge distillation.

It introduces three mechanisms not found together in existing encoders (CLIP, SigLIP, DINOv2):

---

Intended Uses & Limitations

Intended uses

• Image feature extraction — a global pooled embedding for retrieval, clustering, or as a frozen backbone for downstream heads (e.g. linear probing, detection, captioning front-ends).
• Research into adaptive tokenization, hierarchical token typing, and resolution-elastic position encoding.

Limitations

• Accuracy trails same-size peers. Zero-shot ImageNet Top-1 is 37.2%, versus 68.6% for CLIP ViT-B/16 (~86M params) and 76.7% for SigLIP ViT-B/16 (~93M params) — both smaller than this model. This is attributed to a training-budget gap (18K steps on ~20M pairs, vs. peers trained on 12M-400M+ pairs over far more steps) rather than an architecture ceiling: training loss was still declining at the final step. See Evaluation.
• Saliency mechanism is real but token placement is still settling. The learned saliency map produces stable, object-tracking contours on images with a clear dominant subject. However, the discrete tokens it samples continued shifting meaningfully until very late in training (only ~50% overlap with the final checkpoint's token positions as of step 13-14K, reaching ~73% by step 16K) — exact token placement had not fully converged even at the final checkpoint.
• Domain/bias. Trained on web image–text pairs; it inherits the coverage and biases of that distribution and is English-text aligned.
• Not a safety-filtered model. No content moderation or de-biasing has been applied.

---

How to Use

import torch
from PIL import Image
from torchvision import transforms
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "mkd-hika/keural-vision-encoder-mid",
    trust_remote_code=True,
).eval()

transform = transforms.Compose([
    transforms.Resize(384, interpolation=transforms.InterpolationMode.BICUBIC),
    transforms.CenterCrop(384),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

image = Image.open("example.jpg").convert("RGB")
pixel_values = transform(image).unsqueeze(0)          # (1, 3, 384, 384)

with torch.no_grad():
    out = model(pixel_values=pixel_values, token_budget=512)  # match training exactly

image_embedding = torch.nn.functional.normalize(out.pooled, dim=-1)  # (1, 768)

print(out.tokens.shape)            # (1, 512, 768)  per-token features
print(out.pooled.shape)            # (1, 768)       global image embedding
print(out.level_ids.shape)         # (1, 512)       0=global, 1=region, 2=detail
print(out.spatial_metadata.shape)  # (1, 512, 4)    cx, cy, scale_w, scale_h
print(out.saliency_scores.shape)   # (1, 512)       per-token importance

Checkpoints. The repo root holds the final model (step 18,000). Earlier checkpoints are available as subfolders for studying training dynamics: checkpoint-9000, checkpoint-10000, checkpoint-15000, checkpoint-18000 — load one with AutoModel.from_pretrained(..., subfolder="checkpoint-15000").

Note: pass token_budget=512 explicitly. Training always used a fixed budget of 512 tokens; leaving token_budget unset falls back to resolution-based auto-scaling (e.g. 1,152 tokens at 384px), which the model was not trained on. Evaluation showed this mismatch moves results by <1 point, so it's a minor effect, but 512 is the methodologically correct setting to match training.

Output fields

---

Architecture

The figure above is the complete specification: (a) the tokenizer stack (CNN stem → saliency → ATB → HCT), (b) the spatial transformer block and continuous 2D-RoPE, (c) the output contract, dual-teacher distillation, and the training objective.

The pooled image embedding is the [POOL] token after the final RMSNorm (no separate projection head). For contrastive training it is aligned against a trainable projection of a frozen CLIP text encoder.

Model specifications

---

Training

Loss. L_total = L_primary + λ_hct·L_hct + λ_sal·β_sal·L_saliency + L_distill, where L_primary and L_hct are sigmoid image↔text losses (on the pooled embedding and on the mean of global tokens respectively), L_saliency is an anti-collapse + total-variation regularizer on the saliency map, and L_distill is cosine distance to the two frozen teachers.

---

Distillation Teachers

Both teachers are frozen and used only during training; they are not required for inference.

---

Evaluation

Evaluated on the final checkpoint with token_budget=512 (exactly matching training).

Context. Against its own distillation teachers, Keural Mid retains ~45% of SigLIP-SO400M's zero-shot ImageNet accuracy (83.1%) and ~48-56% of its Flickr30K retrieval R@1, after 18K steps on ~20M pairs vs. teachers trained on billions of pairs. Against same-parameter-class peers (CLIP ViT-B/16, SigLIP ViT-B/16 — both smaller than this model), accuracy trails by roughly 2x. Both gaps are consistent with a training-budget shortfall rather than an architecture ceiling: loss was still declining at the final step. A compute/data-matched fixed-grid baseline is the recommended next experiment to isolate the ATB architecture's own contribution from this gap.

The R@1 gap narrows sharply at higher k

The headline R@1 numbers understate retrieval quality. Broken out by k:

On Text→Image R@5, Keural Mid (70.0%) outperforms CLIP ViT-B/16 trained on WIT-400M (57.2%), and at R@10 it leads 79.4% vs. 68.0% — despite that baseline having seen ~20x more training data. The correct-image is usually retrieved within the top few results even when it isn't ranked first, which suggests the embedding space is broadly well-organized and that the R@1 shortfall is largely a ranking-precision issue rather than a representation-quality one. (Note the CC12M-trained CLIP variant still leads at these k, so this is not a uniform win — it is specific to the WIT-400M baseline on the text→image direction.)

Adaptive tokenizer — saliency & token placement

The learned saliency map produces sharp, stable contours that track each image's dominant subject, visible from mid-training through the final checkpoint. Quantitatively, the continuous saliency map stabilizes fast (cosine similarity to the final checkpoint reaches ~0.83-0.90 within the first few thousand steps), but the discrete sampled token positions are considerably less stable — only ~50% overlap with the final checkpoint's tokens by step 13-14K, jumping to 73% at step 16K. Exact token placement was still moving late into training even where the saliency field itself looked converged.

---

---

Comparison with Existing Encoders

---

Roadmap

---

Citation

@misc{keural-mid-2026,
  title  = {Keural Mid-Level Vision Encoder},
  author = {MKD Co., Ltd.},
  year   = {2026},
  url    = {https://huggingface.co/mkd-hika/keural-vision-encoder-mid}
}

License

Copyright © 2026 MKD Co., Ltd. All Rights Reserved. Proprietary — see LICENSE.