holodeck-parser-qwen35-ft

holodeck-parser-qwen35-ft

Fine-tuned LoRA adapter for the Holodeck VR voice command parser.

Built on top of Qwen/Qwen3.5-4B, this model takes a voice transcript and a structured scene context and outputs a single JSON command for a 3D virtual environment engine.

⚠️ Ollama compatibility note: Qwen3.5-4B uses a hybrid SSM+Transformer architecture with a Multi-Token Prediction (MTP) head. Ollama ≤ 0.24.0 cannot load llama.cpp-converted GGUFs of this model due to a GGUF parsing limitation for hybrid architectures. The trained LoRA weights in this repo are fully valid. For local inference, see the GGUF export note below.

Usage via PEFT

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-4B")
model = PeftModel.from_pretrained(base, "trans-realities-lab/holodeck-parser-qwen35-ft")

Input format

Send messages in this structure (system prompt is handled by the tokenizer's chat template):

Transcript: "move the red chair a bit to the left"
User context: {"id": "user_1", "position": {"x": 0, "y": 1.7, "z": 3}, "look_direction": {"x": 0, "y": 0, "z": -1}}
Voice context: {"lockedObjects": [], "fovObjects": [{"nodeId": "server_abc123", "meshName": "Red Chair", "type": "chair", "position": {"x": 2, "y": 0, "z": 1}, "rotation": {"x": 0, "y": 0, "z": 0, "w": 1}, "scale": {"x": 1, "y": 1, "z": 1}}], "raycastHit": null}

Expected output:

{"command": "edit", "id": "server_abc123", "changes": {"position_relative": {"direction": "left", "units": 1}}}

Output schema

Edit supports both absolute (position) and relative (position_relative) moves. Relative directions (left, right, forward, back) are relative to the user's look direction; up/down are world-space.

GGUF export for local inference

To convert for use with llama.cpp or Ollama (requires stripping the MTP layer):

# Merge LoRA into base weights first (run inside the finetuning venv)
python scripts/export_gguf.py --model qwen35 --no-gguf  # saves merged safetensors

# Convert with MTP layer stripped so Ollama can load it
python ~/.unsloth/llama.cpp/convert_hf_to_gguf.py models/qwen35-ft-gguf \
    --outtype f16 --outfile models/qwen35-ft-nomtp-f16.gguf --no-mtp

# Quantize
llama-quantize models/qwen35-ft-nomtp-f16.gguf models/qwen35-ft-q4km.gguf Q4_K_M

# Register with Ollama
bash scripts/register_ollama.sh qwen35

Training details