LFM2.5-VL-450M-Hand-Tracking

LFM2.5-VL-450M Hand Tracking

Fine-tuned LFM2.5-VL-450M-Extract for hand bounding box detection in webcam images. Produces [{"label": "hand", "bbox": [x1, y1, x2, y2]}] in 0-1 normalized coordinates.

Disclaimer: This is a personal learning project — my first VLM fine-tune. It went through ~10 iterations of trial and error, documented honestly below (including what didn't work). It is not a research paper, not a production system, and not affiliated with Liquid AI.

The evaluation data is not scientifically rigorous. Val set results (100% accuracy) reflect controlled conditions on curated images. Empirical webcam testing shows more discrepant results, documented transparently below. This model was built for a specific use case (real-time hand tracking via webcam on consumer hardware) and optimized for that scenario, not for general-purpose hand detection.

Use it as a starting point for your own experiments, not as a drop-in solution.

Model Details

• Base model: LiquidAI/LFM2.5-VL-450M-Extract (450M params, LFM2 architecture)
• Fine-tuning method: LoRA (r=16, alpha=16)
• Trainable parameters: 5,013,504 of 453,732,352 (1.10%)
• Architecture: Lfm2VlForConditionalGeneration (lfm2_vl)
• Vision encoder: SigLIP2 NaFlex 86M
• LM backbone: LFM2.5-350M-Base
• Context length: 4096 tokens
• Inference speed: ~200ms per frame on RTX 3050 (290 tok/s decode)

Training Details

Hardware

• GPU: NVIDIA GeForce RTX 3050 6GB Laptop
• CUDA: 12.6, PyTorch: 2.7.0+cu126
• Training framework: Unsloth 2026.6.9 + TRL 0.24.0

Hyperparameters

Training Results

LoRA Configuration

• finetune_vision_layers: False
• finetune_language_layers: True
• finetune_attention_modules: True
• finetune_mlp_modules: True
• use_gradient_checkpointing: "unsloth"

Dataset

Composition

Data Sources

HaGRID (HAnd Gesture Recognition Image Dataset)

• 554,800 FullHD RGB images, 18 gesture classes
• I used 2 gestures: palm (open hand) + fist (closed hand) for maximum pose diversity
• Bboxes are normalized [x, y, w, h] in 0-1 range, converted to [x1, y1, x2, y2]
• Citation: Kapitanov et al., "HaGRID -- HAnd Gesture Recognition Image Dataset", WACV 2024

COCO val2017 (negative samples)

• 5,000 validation images from COCO 2017
• 2,307 images without any "person" annotation (guaranteed no hands)
• 594 images with person annotation but wrists not visible in keypoints (person on screen, hands occluded)
• The person-no-wrists subset addresses a critical failure mode: the model associated "person in frame" with "hand must exist", producing false positives whenever a person was on camera without hands raised
• Citation: Lin et al., "Microsoft COCO: Common Objects in Context", ECCV 2014

Dataset iteration The negative sampling went through four iterations, each addressing a failure mode observed in testing:

1. No negatives — model always detected a hand, even on empty scenes
2. HaGRID patches only — reduced empty-scene false positives, but still detected hands whenever a person was visible
3. Added COCO no-person — improved diversity, but person-on-screen still triggered false positives
4. Added COCO person-no-wrists — resolved the "person = hand" bias

Content order augmentation (positional bias)

During testing, I discovered that the Q8_0 quantization was sensitive to the order of items in the user content. With image before text, the model correctly returned [] when no hands were visible. With text before image, the same model produced false positive hand detections.

This is a form of positional bias. To mitigate this, I applied Counterfactual Data Augmentation (CDA): each training example has a 50% chance of having the image/text order swapped. This teaches the model that content order should not affect the response.

References:

• AAAI 2024: "Enhance Modality Robustness in Text-Centric Multimodal Alignment"
• ACL 2024: "Unveiling Selection Biases: Order and Token Sensitivity in LLMs"
• arXiv 2404.01430: "Position-Aware PEFT for Reducing Positional Bias"

Seed: 3407 (reproducible augmentation)

HaGRID patches (negative samples)

• 2,000 crops from HaGRID training images, regions outside hand bboxes
• Various sizes (256-640px), same domain as positives

Data Format

{
  "conversations": [
    {
      "role": "user",
      "content": [
        {"type": "image", "image": "path/to/image.jpg"},
        {"type": "text", "text": "Detect all hands in the image."}
      ]
    },
    {
      "role": "assistant",
      "content": [
        {"type": "text", "text": "[{\"label\":\"hand\",\"bbox\":[0.23,0.45,0.67,0.89]}]"}
      ]
    }
  ]
}

Negative examples use "[]" as the assistant response.

Published Files

Model repo: luksamuk/LFM2.5-VL-450M-Hand-Tracking

GGUF repo: luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF

The BF16 file is the merged model (LoRA + base) in full precision. It allows others to create custom quantizations or use the model without quantization. Q8_0 is recommended for PC/laptop. Q4_K_M is recommended for edge devices (Raspberry Pi).

Usage

llama.cpp (GGUF)

llama-server \
  --model LFM2.5-VL-450M-Hand-Tracking-Q8_0.gguf \
  --mmproj mmproj-LFM2.5-VL-450M-Hand-Tracking-F16.gguf \
  --port 8001 \
  --temp 0.1 \
  --jinja

Python (transformers)

from unsloth import FastVisionModel
import torch

model, tokenizer = FastVisionModel.from_pretrained(
    model_name="luksamuk/LFM2.5-VL-450M-Hand-Tracking",
    max_seq_length=4096,
    load_in_4bit=False,
)

Example Application

A demo harness (VLM Webcam Vision) is available at github.com/luksamuk/vlm-webcam-vision. It provides real-time webcam hand tracking with LERP smoothing, label sanitization, and camera controls. Supports any OpenAI-compatible endpoint (llama-swap, Ollama, llama-server) via environment variables.

The llama.cpp server also supports multi-model orchestration natively via model and mmproj flags, enabling hand tracking alongside other models without a proxy.

Output Format

[{"label": "hand", "bbox": [0.23, 0.45, 0.67, 0.89]}]

Coordinates are normalized 0-1. Empty array [] when no hands are visible.

Training Plots

Evaluation Results

250 samples (50 per category). Zero false positives, zero false negatives.

Important: Val set accuracy reflects controlled conditions (curated dataset, ideal images). Empirical webcam testing shows more discrepant results — see below.

Empirical Webcam Testing

Structured protocol: ~10s setup + 5s phases (no hands / 1 hand / 2 hands / alternating) + 3s final.

Key findings:

• 2-hand detection is the main weakness across all models in real webcam conditions
• 450M Q8_0 has the best specificity (69% empty frames) and lowest latency (130ms)
• 1.6B models show zero 2-hand detection despite 100% val set accuracy — domain gap
• Both 1.6B variants were trained with QLoRA (4-bit base) due to VRAM constraints

1.6B experiment (not published)

A 1.6B Extract variant was trained with the same dataset using QLoRA (4-bit base, r=16). Val set showed 100% accuracy, identical to 450M. However, empirical webcam testing revealed zero 2-hand detection and 3x higher latency (400-460ms vs 130-154ms).

The 1.6B is excluded from publication because:

1. It does not improve 2-hand detection in real conditions (the main weakness)
2. Latency is too high for real-time webcam use (target: <200ms)
3. QLoRA may have degraded quality — full LoRA on more capable hardware is left as future work

The domain gap (HaGRID studio images vs webcam real-world) appears to be the primary bottleneck for 2-hand detection, not model size. This affects both 450M and 1.6B equally.

Limitations

• Trained on 2 gestures only (palm + fist); may not generalize to all hand poses
• 2-hand detection in webcam scenarios may flicker between 1 and 2 detections
• Optimized for webcam-resolution images (640x480); may need adjustment for higher resolutions
• Single-person training data; multi-person scenes not tested

What was tried and discarded

• Horizontal flip augmentation: flipped images + bboxes. Introduced label noise (bbox-only flip without image flip in early versions). Corrected version showed no improvement. Discarded.
• Distilling with Qwen3-VL (4B): used Qwen3-VL as teacher to generate hand detection labels on COCO images. IoU 0.805 vs HaGRID ground truth, but introduced patterns that degraded person-no-wrists specificity from 100% to 86%. Discarded.
• Q6_K and Q5_K_M quants: inconsistent behavior across quantization levels. Q6_K produced broken JSON. Q5_K_M over-detected. Only Q8_0 and Q4_K_M published.

Attribution

• Base model: LiquidAI/LFM2.5-VL-450M-Extract -- LFM Open License v1.0
• Training dataset: HaGRID by Kapitanov et al. -- CC BY-SA 4.0
• Negative samples: COCO by Lin et al. -- CC BY 4.0
• Training framework: Unsloth -- Apache 2.0

References

• Kapitanov, A., Kvanchiani, K., Nagaev, A., Kraynov, R., Makhliarchuk, A. "HaGRID -- HAnd Gesture Recognition Image Dataset." WACV 2024. arXiv:2206.08219
• Nuzhdin, A., et al. "HaGRIDv2: 1M Images for Static and Dynamic Hand Gesture Recognition." 2024. arXiv:2412.01508
• Lin, T.-Y., et al. "Microsoft COCO: Common Objects in Context." ECCV 2014. arXiv:1405.0312
• Unsloth LFM2.5-VL fine-tuning guide: https://unsloth.ai/docs/models/tutorials/lfm2.5
• LFM2 Technical Report: arXiv:2511.23404