vr-guardian-segmentation

🥽 Meta Quest VR Guardian Grid Segmentation (U-Net ResNet-34)

A specialized high-resolution semantic segmentation model developed to detect, isolate, and mask the safety boundary grids (Guardian Grid — glowing blue, purple, or red lines and dot patterns) in video gameplay recorded directly from Meta Quest (Quest 2, Quest 3, Quest Pro) VR headsets.

Originally built as the perception module for the GenAIVideoStab / VR Guardian Remover automated inpainting pipeline.

---

🎯 Problem Statement & Engineering Journey

When recording gameplay or mixed reality footage on Meta Quest headsets, approaching the physical room boundaries triggers the headset's safety grid. These bright, semi-transparent lines permanently ruin video captures.

Automating the removal of these grids required solving three distinct computer vision challenges:

[Phase 1: YOLOv8 Bounding Boxes] ──> [Phase 2: YOLOv8 Polygon Seg] ──> [Phase 3: High-Res U-Net (1024x1024)]
   (Too much background cut)           (Lost fine lines & dots)           (Sub-pixel accuracy — PRODUCTION)

1. Phase 1 (BBox Detection): Rectangular bounding boxes cut away huge areas of the game world, forcing the inpainter to hallucinate large parts of the scene.
2. Phase 2 (Polygon Instance Seg): Polygon approximations failed to capture fine, interrupted dots and thin lines under acute perspective angles, leaving ugly artifact remnants.
3. Phase 3 (Semantic U-Net 1024x1024 — Final Solution): A high-resolution U-Net with ResNet-34 backbone trained with a combined $\text{BCEWithLogitsLoss} + \text{DiceLoss}$, achieving sub-pixel precision across diverse gaming environments and lighting conditions.

---

🔬 Morphological Post-Processing: Bloom Removal

Neural networks detect the high-contrast core of the grid lines, but physical displays and optical lenses create a subtle colored halo (light bloom/glow).

This model is paired with a deterministic two-stage morphological post-processing pipeline:

1. Closing (cv2.MORPH_CLOSE, 9x9 kernel): Connects separated dots and broken lines into solid geometry.
2. Dilation (cv2.dilate, 11x11 ellipse kernel, 3 iterations): Safely expands the mask boundaries to swallow all surrounding optical bloom before passing the mask to inpainting engines (e.g. LaMa / ProPainter).

---

⚡ Sparse Skipping for Video Acceleration

In typical VR recordings, the Guardian grid only appears during 10%–30% of the video duration. Using this model's fast logical check:

has_grid = np.any(raw_mask > 0)

clean frames bypass heavy inpainting pipelines entirely. This delivers an up to 3x total video processing speedup with zero degradation to pristine footage.

---

🚀 Quick Start (Inference)

Option 1: ONNX Runtime (Recommended — No PyTorch needed)

pip install onnxruntime opencv-python numpy huggingface_hub

import cv2
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download

# 1. Download ONNX model
model_path = hf_hub_download(repo_id="BiernyVR/vr-guardian-segmentation", filename="vr_guardian_unet.onnx")

# 2. Preprocess input image to 1024x1024 RGB
img_bgr = cv2.imread("vr_gameplay_frame.jpg")
h, w = img_bgr.shape[:2]

resized = cv2.resize(cv2.cvtColor(img_bgr, cv2.COLOR_BGR2RGB), (1024, 1024))
tensor = ((resized.astype(np.float32) / 255.0 - [0.485, 0.456, 0.406]) / [0.229, 0.224, 0.225]).transpose(2, 0, 1)
tensor = np.expand_dims(tensor, axis=0).astype(np.float32)

# 3. Run Inference
session = ort.InferenceSession(model_path, providers=["CPUExecutionProvider"])
logits = session.run(None, {"input": tensor})[0][0][0]
probs = 1.0 / (1.0 + np.exp(-logits))
raw_mask = (probs > 0.5).astype(np.uint8) * 255

# 4. Morphological Refinement (Closing + Dilation to swallow bloom)
mask_full = cv2.resize(raw_mask, (w, h), interpolation=cv2.INTER_NEAREST)
close_k = cv2.getStructuringElement(cv2.MORPH_RECT, (9, 9))
dilate_k = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (11, 11))
refined_mask = cv2.morphologyEx(mask_full, cv2.MORPH_CLOSE, close_k)
refined_mask = cv2.dilate(refined_mask, dilate_k, iterations=3)

cv2.imwrite("guardian_mask.png", refined_mask)
print("Mask saved! Pass this mask directly to LaMa or another inpainting tool.")

Option 2: Standalone CLI

python infer.py --image sample_frame.jpg

---

📦 Files in this Repository