vla-1.7b-qwen3-v6

VLA 1.7B — Qwen3 v6 (seq_length=8192)

A 1.7B parameter Vision-Language-Action model, third Qwen3-backbone release from this project. v6 rebuilds the window=8 architecture that produced vla-1.7b-qwen3-v2 (still the project's strongest baseline) and adds newly-integrated data sources on top of it — Harmony4D (multi-person close-interaction motion capture), MV-Omni (MixtureVitae-Omni, audio+image+text), and the <listen>/<speak> audio-token convention (replacing v2's bare <snac> wrapper). This checkpoint uses seq_length=8192, the best-performing of three seq_length variants (4096/8192/16384) trained in parallel as an ablation.

Key facts

What this model does

Given a text prompt (activity description, image seed2 block, or partial modality sequence), the model generates an interleaved multimodal token sequence spanning 6 categories it was trained on:

<seed2_N> ...                          # 1 FPS semantic image/video keyframes (vocab 8192)
<cosmos_N> ... </cosmos>               # 8-frame spatial video tokens (vocab 64000)
<listen> <snac_N> ... </listen>        # SNAC audio codec tokens, "heard" role
<speak> <snac_N> ... </speak>          # SNAC audio codec tokens, model-generated "spoken" role
<caption> ... </caption>               # inline visual caption text
<agent> <fps_30> <pelvis> ... </agent> # 3D human pose, 17 H36M joints

Progress vs. previous versions (v2 → v6)

The project trained v3/v4/v5 on a window=24 architecture (longer temporal chunks, different cosmos token cost, doc-packing, adjusted cosmos dropout). None of v3/v4/v5 beat v2 on any measured axis (perplexity, modality-persistence over time, or caption accuracy) — see the project's full eval writeup. v6 abandons window=24 and rebuilds on v2's window=8 recipe instead, only adding new data sources on top. Result: test PPL recovers to 5.98–6.36 across the 3 seq_length variants, close to v2's 5.77 and far better than v3 (27.58) / v4 (15.78) / v5 (16.75). This seq8192 checkpoint had the best test PPL of the three and produced all 5 trained modalities in a single full-chain generation test.

Cosmos-continuity: this checkpoint returned to <cosmos> and generated 3 consecutive real chunks in one sampled-decoding run — the best cosmos-persistence observed across the whole v2→v6 lineage for a multi-chunk (not just single-return) case.

Known limitations

• Instruction-following gap, unchanged since v2: given a genuinely novel activity description not seen in training (e.g. "A person is running in a park"), this model's self-generated caption is topically wrong (confabulates an unrelated activity), exactly like every prior version (v2 through v5). This is the project's highest-priority open problem — diagnosed as a missing instruction-tuning data stage (see the project's AnyGPT/AnyInstruct-inspired "VLA-Instruct" proposal), not something any pretraining-recipe change (window size, dropout, seq_length, mix ratio) has fixed so far.
• Modality persistence, improved but only single-seed-verified for this specific checkpoint: the sibling seq4096 variant achieved 4 consecutive non-frozen <agent> blocks in one run (best in project history), but only verified on 1 of 4 standard eval seeds — treat any single-seed persistence claim (including this checkpoint's cosmos-continuity result above) as encouraging, not proven-robust.
• avc_lm tokens are essentially unused — discarded at the data-flatten stage before training (to control token count), so the model rarely if ever produces them.
• seed2→image reconstruction is generative, not a deterministic round-trip (see tools/decode/decode_seed2.py — conditions a diffusion img2img pipeline on the token embeddings, unlike cosmos/snac's lossy-but-deterministic codec decoders).
• Evaluation so far is qualitative (manual inspection of generated tokens/decoded media, plus PPL) — no MPJPE, BLEU/CIDEr, or closed-loop task-success metric has been run yet.
• Two other seq_length variants exist internally (seq4096, test PPL 6.36; seq16384, test PPL 6.12) but were not chosen for public release — seq8192 had the best overall balance of PPL and multi-modality coverage in the project's eval suite.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "EmpathicRobotics/vla-1.7b-qwen3-v6",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("EmpathicRobotics/vla-1.7b-qwen3-v6")

prompt = (
    "### Context: Person raises both arms above head.\n"
    "<seed2_3758> <seed2_2157> <cosmos_58567> "
    "<fps_30> <pelvis> <pelvis_t_0> <pelvis_x_128> <pelvis_y_128> <pelvis_z_128>"
)
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(model.device)
output = model.generate(
    input_ids, max_new_tokens=500,
    do_sample=True, temperature=0.8, top_p=0.9, repetition_penalty=1.3,
)
print(tokenizer.decode(output[0]))

Encoding real media into tokens (so you can actually prompt the model)

The ## Usage prompt above uses pre-picked token ids as a demo. To send the model real media -- e.g. "here's a photo, continue the scene" or "here's a real motion clip, keep going" -- encode it first with the 4 encoders below. Bundled in this repo the same way as the decoders (tools/encode/), no separate git clone needed.

# Image -> <seed2_N> tokens (32 ids, auto-downloads the Q-Former checkpoint
# from ontocord/seed2 if not cached locally)
python tools/encode/encode_seed2.py --image photo.jpg

# 8 video frames -> <cosmos_N> tokens (200 ids -- window=8/square-crop
# convention; the window=24/aspect-preserving convention (896 tokens) was NOT
# used for this model)
python tools/encode/encode_cosmos.py --frames f0.png f1.png f2.png f3.png f4.png f5.png f6.png f7.png

# Audio/video file -> <snac_N> tokens, wrapped in <listen> (this model's
# "heard" convention, replacing v2's bare <snac>)
python tools/encode/encode_snac.py --input clip.wav

# Real 3D pose (8 frames x 17 joints x xyz, metres, root-centred) -> <agent>
# tokens -- for "give the model a real motion capture / pose-pipeline output,
# have it continue"
python tools/encode/encode_agent.py --input pose.npy   # shape (8, 17, 3)

Splice the printed token block into your prompt (e.g. after ### Context: ...) the same way the ## Usage example does, then call model.generate() as shown there.

Decoding generated tokens back to media

The decoder scripts + their vendored dependencies are bundled directly in this repo (tools/) -- one snapshot_download gets everything, no separate git clone needed.

python -c "
from huggingface_hub import snapshot_download
snapshot_download('EmpathicRobotics/vla-1.7b-qwen3-v6', allow_patterns=['tools/*', 'tools/**/*'])
"
pip install scipy numpy torch torchvision imageio-ffmpeg soundfile snac huggingface_hub
cd <snapshot-download-cache-dir-printed-above>

Agent tokens -> 3D pose (pure Python, no extra downloads):

python tools/eval/decode_agent_tokens.py --input generated_tokens.txt --output poses.json

Cosmos tokens -> video (auto-downloads the ~350MB decoder checkpoint from nvidia/Cosmos-Tokenizer-DV8x16x16 on first run):

python tools/decode/decode_cosmos.py --tokens 58345,57843,... --output out.mp4
# this model's cosmos chunks are exactly 200 raw ids each (8 frames, 160x160,
# square-cropped) -- the window=24/aspect-preserving convention (896 tokens)
# does NOT apply to this model.

SNAC tokens -> audio (auto-downloads hubertsiuzdak/snac_24khz from HF):

python tools/decode/decode_snac.py --tokens 130911,134940,... --format listen --output out.wav
# use --format listen for <listen>-wrapped tokens (input/"heard" role) or
# --format speak for <speak>-wrapped tokens (model-generated "spoken" role) --
# this model was trained on both, unlike v2 which only ever saw bare <snac>.

Seed2 tokens -> image (auto-downloads the ~2.6GB Q-Former checkpoint from the tokenizer's own public repo, ontocord/seed2, plus a ~5GB diffusion img2img pipeline on first run -- this one is a generative reconstruction, not a deterministic decode, so expect run-to-run and prompt-to-prompt variation in the exact pixels even for the same tokens):

python tools/decode/decode_seed2.py --tokens 6750,680,2472,... --output out.png
# exactly 32 raw ids per image (Seed2Tokenizer's fixed Q-former query length)

Training details

Loss curve

Config

• Batch: GBS 1024, seq_len 8192, micro_batch_size 2 → 33.83B tokens trained (exactly 1 epoch of the w8_new corpus)
• Infrastructure: 64 nodes × 4 GH200 GPUs (256 total), ~303 TFLOP/s/GPU, ~20,650 tok/s/GPU
• Framework: Megatron-LM via oellm-autoexp

Data mix

Citation

@misc{empathicrobotics2026vlaqwen3v6,
  title={VLA 1.7B Qwen3 v6: Window=8 Architecture Rebuild with Expanded Multimodal Sources},
  author={EmpathicRobotics},
  year={2026},
  url={https://huggingface.co/EmpathicRobotics/vla-1.7b-qwen3-v6}
}