Wukong

Wukong (悟空): An Open Spatial-Reasoning Vision-Language-Action Model

Wukong is an open-source Vision-Language-Action (VLA) model focused on spatial reasoning for embodied agents. It is designed for robots that must understand, remember, and reason about 3D space from egocentric visual streams — and then act in it.

Named after the Monkey King (Sun Wukong) of Chinese mythology — famed for his piercing "fiery eyes" (火眼金睛) that see through the world — Wukong aims to give embodied agents a comparably sharp sense of space.

What's in This Release (v0.1, T2 stage)

This repository currently hosts Wukong-T2-Adapter: a QLoRA adapter (rank 16, 51.5M trainable params, 206MB) on top of unsloth/Qwen2.5-VL-7B-Instruct, fine-tuned for visual-spatial question answering from egocentric images.

• adapter_config.json + adapter_model.safetensors — the trained adapter (final checkpoint, step 49730)
• Tokenizer / preprocessor / chat template files — everything needed to load and chat
• eval_results.jsonl — per-item evaluation records (question, reference, prediction, score) for full reproducibility

Training setup: 198,919 samples (195,919 VSI-style spatial QA derived from Hypersim + 3,000 programmatic egocentric↔allocentric coordinate-transform QA), 2 epochs, total batch size 8, ~18.5 hours on a single RTX 5090 with Unsloth. Final training loss ≈ 0.19.

Roadmap stages beyond T2 (semantic waypoint prediction, embodiment-specific action adapters, world-action predictive pathway) are under active development and will be released as separate artifacts.

Evaluation

Held-out spatial QA split from our own data pipeline (never seen in training, n=500), scored in VSI-Bench style: exact match for multiple-choice items; MRA (max(0, 1 − |pred−ref|/ref)) for numeric items.

Breakdown by answer format: multiple-choice (n=345) accuracy 0.832; numeric (n=155) MRA 0.671.

Note: the reported 0.782 is measured on our own held-out split, not the official VSI-Bench; official-benchmark evaluation is on the roadmap.

Usage

from unsloth import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained(
    "unsloth/Qwen2.5-VL-7B-Instruct",
    load_in_4bit=True,
    max_seq_length=4096,
)
model.load_adapter("FINWHALE/Wukong")   # adapter files at repo root
FastVisionModel.for_inference(model)

A FastAPI serving wrapper and an evaluation script are available in the companion training pipeline (to be linked from this card).

Intended Use

• Spatial question answering and measurement estimation from egocentric images
• Research on spatial intelligence, world models, and embodied reasoning
• A foundation for downstream fine-tuning on specific robot embodiments (navigation adapters on the roadmap)

Limitations

• Metric-precision spatial estimation (exact distances, absolute cardinal directions) remains the weakest capability (0.55–0.61) — an open challenge shared by all current VLM/VLA systems.
• This release is a spatial-reasoning VLM stage: it does not output motor commands. Closed-loop navigation requires the upcoming waypoint/action adapters.
• Long-horizon imagination may drift; closed-loop correction with real observations is required in deployment.
• Not intended for safety-critical deployment without additional validation.

Citation

@misc{wukong2026,
  title  = {Wukong: An Open Spatial-Reasoning Vision-Language-Action Model},
  author = {FINWHALE},
  year   = {2026},
  url    = {https://huggingface.co/FINWHALE/Wukong}
}

License

Apache 2.0. Please also comply with the licenses of the underlying base models and datasets.