lingbot-vision-vit-giant

LingBot-Vision

LingBot-Vision is a family of self-supervised Vision Transformer backbones for dense spatial perception. The models are pretrained with masked boundary modeling, a boundary-centric objective that encourages spatially structured patch features while retaining strong semantic representations.

This Hugging Face repository stores a backbone-only PyTorch checkpoint as model.pt. It is intended for inference, feature extraction, PCA visualization, and downstream dense prediction research.

Model Details

Model Description

LingBot-Vision learns dense patch representations that preserve boundaries, shapes, and semantic regions. The backbone is trained from random initialization with self-supervised teacher-student pretraining. During training, teacher-discovered boundary tokens are forced into the masked set, and boundary tokens receive both semantic self-distillation and categorical boundary-field supervision.

The released model family includes:

• LingBot-Vision-Giant: ViT-g/16 backbone for highest-quality dense features.
• LingBot-Vision-Large: ViT-L/16 backbone for strong dense features and practical inference.
• LingBot-Vision-Base: ViT-B/16 backbone for balanced inference cost.
• LingBot-Vision-Small: ViT-S/16 backbone for lightweight demos and downstream use.

Each checkpoint contains backbone weights only. Training-time heads, optimizer states, and boundary-target generation components are not included.

• Developed by: Zelin Fu, Bin Tan, Changjiang Sun, Shaohui Liu, Kecheng Zheng, Yinghao Xu, Xing Zhu, Yujun Shen, Nan Xue
• Model type: Vision Transformer backbone for dense visual representation learning
• License: Apache 2.0

Model Sources

• Paper: Vision Pretraining for Dense Spatial Perception
• Repository: https://github.com/robbyant/lingbot-vision
• Project Page: https://technology.robbyant.com/lingbot-vision

Related Models

• LingBot-Vision-Giant: https://huggingface.co/robbyant/lingbot-vision-vit-giant
• LingBot-Vision-Large: https://huggingface.co/robbyant/lingbot-vision-vit-large
• LingBot-Vision-Base: https://huggingface.co/robbyant/lingbot-vision-vit-base
• LingBot-Vision-Small: https://huggingface.co/robbyant/lingbot-vision-vit-small

Uses

Direct Use

• Dense Feature Visualization: Extract frozen patch tokens and visualize their PCA components.
• Image Feature Extraction: Use normalized patch tokens as spatial visual features.
• Backbone Initialization: Initialize downstream dense prediction models with LingBot-Vision weights.

Downstream Use

• Depth Estimation: Frozen patch tokens expose spatial structure to lightweight dense readouts.
• Semantic Segmentation: Boundary-faithful features help align region transitions with object contours.
• Video Object Segmentation: Frozen features support training-free label propagation and token matching.
• Depth Completion: LingBot-Vision can serve as the visual encoder initialization for LingBot-Depth 2.0.

How to Load

Install the LingBot-Vision inference repository and dependencies:

git clone https://github.com/robbyant/lingbot-vision.git
cd lingbot-vision

conda create -n lingbot-vision python=3.10 -y
conda activate lingbot-vision

python -m pip install -r requirements.txt
python -m pip install -e .

Load a pretrained backbone:

import torch

from lbot_vision_infer import load_pretrained_backbone

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

backbone, embed_dim = load_pretrained_backbone(
    variant="large",
    device=device,
    dtype=dtype,
)

print(backbone.patch_size, embed_dim)

The variant argument can be giant, large, base, or small. You can also pass an explicit Hugging Face model repo or a local directory to load_pretrained_backbone.

Technical Specifications

Model Architecture

• Backbone: Vision Transformer with patch size 16
• Released variants: ViT-g/16, ViT-L/16, ViT-B/16, ViT-S/16
• Output: Normalized patch tokens from the frozen backbone
• Checkpoint format: Backbone-only .pt file stored as model.pt
• Training objective: Masked boundary modeling with self-distillation

Software Requirements

• Python >= 3.10
• PyTorch >= 2.0.0
• huggingface_hub
• omegaconf

Citation

@article{lingbot-vision2026,
  title={Vision Pretraining for Dense Spatial Perception},
  author={Fu, Zelin and Tan, Bin and Sun, Changjiang and Liu, Shaohui and Zheng, Kecheng and Xu, Yinghao and Zhu, Xing and Shen, Yujun and Xue, Nan},
  year={2026}
}

Model Card Contact

• Issues: https://github.com/robbyant/lingbot-vision/issues
• Email: [email protected], [email protected]