hy-embodied-mlx

hy-embodied-mlx

A from-scratch Apple MLX implementation of the hunyuan_vl_mot architecture (tencent/HY-Embodied-0.5, a 3.79B mixture-of-transformers embodied VLM), plus weight conversion, a quantization ladder, and a pointing-accuracy eval harness.

The measured 4/5/6/8-bit and bf16 MLX weights are published on Hugging Face (links in the results table below, all variants in this collection); the 3-bit probe is measured but unpublished. Verification record: docs/phase0-manifest.md (architecture), docs/parity.md (MLX-vs-reference parity), docs/quant.md (sizes, throughput), docs/results.md (quality ladder).

Why

No quantization of HY-Embodied-0.5 exists in any format, and no public runtime executes its architecture outside the pinned reference transformers commit. This repo provides the runtime plus bf16 and 4/5/6/8-bit MLX weights with measured spatial-grounding quality per tier, and GGUF conversions once a runnable inference path exists for them. Numbers only get published if a script in this repo reproduces them.

Results

Where2Place pointing accuracy (official soft-mask protocol, greedy, fixed prompt template across tiers; Wilson 95% intervals, n=100 — statistical addendum with discordant pairs and McNemar tests in docs/results.md):

Text-only probe, 50 scripted prompts (Wilson 95% intervals, n=50):

Through 5-bit, nothing separates any variant from bf16 — all intervals overlap and every adjacent-tier McNemar test is non-significant. The first clear break is at 3-bit and it appears in the text path: no-think probe accuracy collapses from 0.70 to 0.40 with non-overlapping intervals, while the matching pointing drop stays within intervals and reaches only uncorrected nominal significance. On this evidence the low-bit failure is text-first; grounding degradation is directionally consistent but not separately established. Tencent's published 68.0 (thinking) comes from an unpublished harness and is cited for range only, not compared against.

Reproduce any cell:

python -m hy_embodied_mlx.convert --out mlx-bf16
python -m hy_embodied_mlx.quantize --model mlx-bf16 --out mlx-4bit --bits 4
python evals/where2place.py --model mlx-4bit --mode nothink --out out.csv

Oracle

The reference implementation requires flash-attn and cannot run on macOS as shipped. oracle/ contains a pure-torch flash-attn shim that makes it run on Apple Silicon (MPS, bf16), a state-dict manifest dumper, and the golden fixture generator used for parity testing. Setup:

python3.12 -m venv venv && source venv/bin/activate
pip install torch timm safetensors huggingface_hub pillow accelerate \
  "git+https://github.com/huggingface/transformers@9293856c419762ebf98fbe2bd9440f9ce7069f1a"
PYTHONPATH=oracle python oracle/generate_fixtures.py tests/fixtures mps

Licensing

Code in this repository is Apache-2.0. The model weights are Tencent's, under the Tencent HY Community License Agreement — not an open-source license; among other restrictions it excludes the territory of the EU, UK, and South Korea, and its Section 5 use restrictions pass through to derivatives. Any quantized weights produced by this tooling will carry the license copy, the required NOTICE text, and a modified-files statement. This project is not affiliated with, sponsored, or endorsed by Tencent.