vrkeyb-cvpr2026

<div align="center">

Depth Anything V2 ViT-S — D405 Close-Range Fine-tuned

Real-Time Multimodal Fingertip Contact Detection via Depth and Motion Fusion for Vision-Based Human-Computer Interaction

CVPR 2026

Mukhiddin Toshpulatov^1,2,4,5 · Wookey Lee² · Suan Lee³ · Geehyuk Lee¹

¹SpaceTop, SoC, KAIST    ²VoiceAI, BMSE, Inha University    ³SoCS, Semyung University    ⁴Dep. of CE, Gachon University, South Korea    ⁵Jizzakh branch of the National University of Uzbekistan

</div>

Model Description

This is a fine-tuned Depth Anything V2 ViT-S model, adapted for metric depth estimation at close range (7–50 cm) using data from an Intel RealSense D405 stereo depth camera. The model reduces depth estimation error by 68% (12.3 mm → 3.84 mm MAE) compared to the pre-trained baseline in the operating range relevant to hand-surface interaction.

The fine-tuned model serves as the depth backbone for a real-time vision-based virtual keyboard system that achieves 94.2% contact detection accuracy at 30 FPS on consumer hardware.

Key Improvements over Base Model

Usage

Installation

pip install torch torchvision timm opencv-python numpy
git clone https://github.com/DepthAnything/Depth-Anything-V2.git

Inference

Critical: max_depth must be set to 0.5 at inference. The DPT head computes depth = sigmoid(features) × max_depth. Using the default max_depth=20.0 will produce predictions 40× too large.

import torch
import cv2
import numpy as np
from depth_anything_v2.dpt import DepthAnythingV2

# Initialize model with correct max_depth
model = DepthAnythingV2(
    encoder='vits',
    features=64,
    out_channels=[48, 96, 192, 384],
    max_depth=0.5  # MUST match training config
)

# Load fine-tuned weights
state_dict = torch.load(
    'depth_anything_v2_vits_d405_finetuned.pth',
    map_location='cpu'
)['model']

# Strip DDP prefix if present
cleaned = {k.replace('module.', ''): v for k, v in state_dict.items()}
model.load_state_dict(cleaned)
model.eval().cuda()

# Predict depth from RGB image
image = cv2.imread('your_image.png')
with torch.no_grad():
    depth_m = model.infer_image(image)  # Returns metric depth in meters

# Convert to millimeters
depth_mm = depth_m * 1000.0
print(f"Depth range: {depth_mm.min():.1f} - {depth_mm.max():.1f} mm")

Batch Inference

from torchvision.transforms import Compose, Normalize, Resize, ToTensor

transform = Compose([
    Resize((518, 518)),
    ToTensor(),
    Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

# Process a batch
images = torch.stack([transform(img) for img in image_list]).cuda()
with torch.no_grad():
    predictions = model(images)  # [B, 1, H, W] metric depth in meters

Training Details

Architecture

Training Configuration

Why Relative Pretrained Weights (Not Metric)?

The base depth_anything_v2_vits.pth (relative depth) was chosen over depth_anything_v2_metric_vits.pth (Hypersim-pretrained metric) because:

• The metric checkpoint is trained on indoor scenes at 0.5–20 m range, which introduces strong bias away from our 7–50 cm operating range
• The relative checkpoint provides a better feature foundation for domain adaptation to close-range hand-surface geometry
• Empirically, fine-tuning from relative weights converged faster and achieved lower validation loss

Training Dataset

D405 Hand-Surface Depth Dataset — 53,300 RGB-depth pairs from 15 participants at four camera angles (30°, 45°, 60°, 90°), captured with Intel RealSense D405.

Splits are participant-stratified (no identity leakage).

Evaluation

Depth Metrics on Test Set

Evaluated on 9,021 held-out frames from 2 participants (P05, P08) across all 4 camera angles.

Downstream Task: Contact Detection

When used as the depth backbone for fingertip contact detection with a velocity-gated hysteresis state machine:

Model Files

Checkpoint Contents

checkpoint = torch.load('depth_anything_v2_vits_d405_finetuned.pth')
checkpoint.keys()  # ['model', 'optimizer', 'epoch', 'previous_best']

# To extract model weights only:
model_weights = checkpoint['model']

Limitations

• Depth range: Optimized for 7–50 cm. Performance degrades significantly outside this range.
• Scene type: Trained on hand-surface typing scenes. May not generalize well to arbitrary indoor/outdoor scenes.
• max_depth dependency: Inference MUST use max_depth=0.5. Mismatched values produce incorrect depth scales.
• Camera specificity: Depth supervision comes from Intel RealSense D405. Performance may vary when applied to depth data from other sensors.
• Lighting: Trained primarily under controlled LED lighting (~5000K, 800+ lux). Performance in extreme lighting conditions is not validated.

Intended Use

Primary Use Cases

• Real-time fingertip contact detection for vision-based virtual keyboards
• Close-range metric depth estimation for hand-object interaction
• Monocular depth estimation in tabletop AR/VR scenarios
• Research on depth model adaptation for specialized domains

Out-of-Scope Use

• General-purpose depth estimation at arbitrary ranges
• Outdoor or long-range depth estimation
• Safety-critical applications without additional validation

Citation

@inproceedings{toshpulatov2026realtime,
  title={Real-Time Multimodal Fingertip Contact Detection via Depth and Motion
         Fusion for Vision-Based Human-Computer Interaction},
  author={Toshpulatov, Mukhiddin and Lee, Wookey and Lee, Suan and Lee, Geehyuk},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision
             and Pattern Recognition (CVPR)},
  year={2026}
}

Acknowledgements

• Depth Anything V2 — base model architecture and pre-trained weights
• DINOv2 — vision transformer backbone
• Intel RealSense — D405 depth camera for ground truth supervision

License

This model is released under the Apache 2.0 License.