holodeck-parser-llama32-ft

holodeck-parser-llama32-ft

Fine-tuned LoRA adapter for the Holodeck VR voice command parser.

Built on top of Llama 3.2-3B-Instruct, this model takes a voice transcript and a structured scene context and outputs a single JSON command for a 3D virtual environment engine.

Usage via Ollama (recommended)

ollama pull eyyrone/holodeck-parser-llama32-ft

The system prompt is baked into the Ollama model. Send user messages in this format:

Transcript: "move the red chair a bit to the left"
User context: {"id": "user_1", "position": {"x": 0, "y": 1.7, "z": 3}, "look_direction": {"x": 0, "y": 0, "z": -1}}
Voice context: {"lockedObjects": [], "fovObjects": [{"nodeId": "server_abc123", "meshName": "Red Chair", "type": "chair", "position": {"x": 2, "y": 0, "z": 1}, "rotation": {"x": 0, "y": 0, "z": 0, "w": 1}, "scale": {"x": 1, "y": 1, "z": 1}}], "raycastHit": null}

Expected output:

{"command": "edit", "id": "server_abc123", "changes": {"position_relative": {"direction": "left", "units": 1}}}

Note: Strip trailing $ and take text before the first \n\n to get clean JSON from the raw response.

Usage via PEFT (this repo)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
model = PeftModel.from_pretrained(base, "trans-realities-lab/holodeck-parser-llama32-ft")

Output schema

Edit supports both absolute (position) and relative (position_relative) moves. Relative directions (left, right, forward, back) are relative to the user's look direction; up/down are world-space.

Training details