Bridging 2D Canvas and 3D Spatial Rendering

Bridging 2D Canvas and 3D Spatial Rendering

StickerNest uses a parallel rendering architecture where 2D DOM canvas and 3D WebGL scene coexist. Understanding this bridge is critical for building features that work across desktop, VR, and AR.

Core Architecture

┌─────────────────────────────────────────────────────────────┐
│  CanvasPage.tsx                                             │
│  ┌─────────────────────┐    ┌─────────────────────────────┐ │
│  │  CanvasRenderer     │    │  SpatialCanvas              │ │
│  │  (DOM/​2D)           │    │  (WebGL/​3D)                 │ │
│  │                     │    │                             │ │
│  │  - HTML elements    │    │  - Three.js scene           │ │
│  │  - CSS positioning  │    │  - 3D meshes & materials    │ │
│  │  - React components │    │  - WebXR sessions           │ │
│  │                     │    │                             │ │
│  │  visible: desktop   │    │  visible: vr/​ar             │ │
│  └─────────────────────┘    └─────────────────────────────┘ │
│                                                             │
│  ┌─────────────────────────────────────────────────────────┐│
│  │  Shared State (Zustand stores)                          ││
│  │  - useCanvasStore (widgets, stickers, positions)        ││
│  │  - useSpatialModeStore (desktop/​vr/​ar mode)             ││
│  └─────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────┘

Key principle: Both renderers read from the same state. Position a widget in 2D, and it appears in the correct 3D location automatically.

Spatial Modes

import { useActiveSpatialMode, useIsDesktopMode } from '@/​state/​useSpatialModeStore';

type SpatialMode = 'desktop' | 'vr' | 'ar';

// In components:
const spatialMode = useActiveSpatialMode();
const isDesktopMode = spatialMode === 'desktop';

Coordinate Conversion

The bridge between 2D and 3D is coordinate conversion. Use these utilities from src/​utils/​spatialCoordinates.ts:

Constants

import {
  PIXELS_PER_METER,    // 100 - conversion factor
  DEFAULT_WIDGET_Z,    // -2 meters (in front of user)
  DEFAULT_EYE_HEIGHT,  // 1.6 meters (standing user)
} from '@/​utils/​spatialCoordinates';

2D to 3D Conversion

import { toSpatialPosition, toSpatialSize, toSpatialRotation } from '@/​utils/​spatialCoordinates';

// Position: pixels → meters
const pos3D = toSpatialPosition({ x: 500, y: 300 });
// Returns: [5, 1.1, -2] (x in meters, y adjusted for eye height, z = default depth)

// Size: pixels → meters
const size3D = toSpatialSize({ width: 200, height: 150 });
// Returns: { width: 2, height: 1.5 }

// Rotation: degrees → radians (around Z axis)
const rot3D = toSpatialRotation(45);
// Returns: [0, 0, -0.785] (Euler angles)

3D to 2D Conversion

import { toDOMPosition, toDOMSize } from '@/​utils/​spatialCoordinates';

// Position: meters → pixels
const pos2D = toDOMPosition([5, 1.1, -2]);
// Returns: { x: 500, y: 300 }

// Size: meters → pixels
const size2D = toDOMSize({ width: 2, height: 1.5 });
// Returns: { width: 200, height: 150 }

Full Transform

import { toSpatialTransform } from '@/​utils/​spatialCoordinates';

const transform = toSpatialTransform({
  x: 500,
  y: 300,
  width: 200,
  height: 150,
  rotation: 45,
  scale: 1,
  z: -3, // optional custom depth
});
// Returns: { position: [x,y,z], rotation: [rx,ry,rz], scale: [s,s,s] }

Y-Axis Inversion

Critical: DOM Y grows downward, 3D Y grows upward. The conversion handles this:

DOM:     (0,0) ──────► X        3D:      Y ▲
              │                           │
              │                           │
              ▼ Y                         └──────► X

Widget Rendering Across Modes

The Html Component Problem

The <Html> component from @react-three/​drei renders DOM content in 3D space. BUT it creates DOM overlays that break immersive WebXR:

// This breaks immersive VR! DOM overlays appear as flat screen
<Html transform position={[0, 0, 0]}>
  <div>Widget content</​div>
</​Html>

XR Session Detection

Always check if in an XR session before rendering Html:

import { useXR } from '@react-three/​xr';

function SpatialWidget({ widget }) {
  // Detect active XR session
  const session = useXR((state) => state.session);
  const isPresenting = !!session;

  return (
    <group position={position3D}>
      {/​* 3D panel mesh - always renders */​}
      <mesh>
        <planeGeometry args={[width, height]} />
        <meshStandardMaterial color="#1e1b4b" />
      </​mesh>

      {/​* Html content - ONLY when NOT in XR session */​}
      {!isPresenting && (
        <Html transform center>
          <WidgetContent />
        </​Html>
      )}

      {/​* 3D placeholder - ONLY when IN XR session */​}
      {isPresenting && (
        <Text position={[0, 0, 0.01]} fontSize={0.05}>
          {widget.name}
        </​Text>
      )}
    </​group>
  );
}

Decision Tree: Html vs Pure 3D

Is this for XR (VR/​AR)?
├─ YES: Use pure Three.js
│       - <mesh> with materials
│       - <Text> from drei for labels
│       - Textures for images
│       - NO <Html> components
│
└─ NO (desktop/​preview): Can use Html
        - <Html> for complex React UI
        - iframes for widget sandboxing
        - Full CSS styling

Mode-Aware Component Pattern

import { useActiveSpatialMode } from '@/​state/​useSpatialModeStore';
import { useXR } from '@react-three/​xr';

function ModeAwareWidget({ widget }) {
  const spatialMode = useActiveSpatialMode();
  const session = useXR((state) => state.session);
  const isXRActive = !!session;

  // Desktop mode: use CanvasRenderer (not this component)
  // This component only runs in SpatialCanvas (vr/​ar modes)

  if (isXRActive) {
    // TRUE XR: Pure WebGL only
    return <PureWebGLWidget widget={widget} />;
  }

  // Preview mode (vr/​ar without XR session): Can use Html
  return <HtmlBasedWidget widget={widget} />;
}

Parallel Rendering in CanvasPage

// src/​pages/​CanvasPage.tsx structure
function CanvasPage() {
  const spatialMode = useActiveSpatialMode();
  const isDesktopMode = spatialMode === 'desktop';

  return (
    <>
      {/​* DOM Renderer - visible only in desktop mode */​}
      {isDesktopMode && (
        <CanvasRenderer
          widgets={widgets}
          // ... DOM-based rendering
        />
      )}

      {/​* WebGL/​XR Renderer - visible in VR/​AR modes */​}
      <SpatialCanvas active={!isDesktopMode} />
    </>
  );
}

Common Patterns

Pattern: Conditional Widget Content

function WidgetPanel({ widget, isPresenting }) {
  const size3D = toSpatialSize({ width: widget.width, height: widget.height });

  return (
    <group>
      {/​* Base panel - always visible */​}
      <mesh>
        <planeGeometry args={[size3D.width, size3D.height]} />
        <meshStandardMaterial color="#1e1b4b" transparent opacity={0.95} />
      </​mesh>

      {/​* Content layer */​}
      {isPresenting ? (
        // XR mode: 3D placeholder
        <group position={[0, 0, 0.01]}>
          <Text fontSize={0.05} color="white" anchorX="center">
            {widget.name || 'Widget'}
          </​Text>
          <Text position={[0, -0.08, 0]} fontSize={0.03} color="#888">
            {widget.widgetDefId}
          </​Text>
        </​group>
      ) : (
        // Preview mode: Full HTML content
        <Html transform center distanceFactor={1.5}>
          <WidgetIframe widget={widget} />
        </​Html>
      )}
    </​group>
  );
}

Pattern: Syncing 3D Changes Back to 2D

When users move widgets in VR, update the 2D canvas state:

function handleWidgetMove(widgetId: string, newPos3D: [number, number, number]) {
  // Convert 3D position back to 2D
  const pos2D = toDOMPosition(newPos3D);

  // Update the shared canvas store
  useCanvasStore.getState().updateWidget(widgetId, {
    x: pos2D.x,
    y: pos2D.y,
  });
}

Pattern: Resolution Scaling for VR

HTML content can look pixelated in VR. Use resolution scaling:

const VR_RESOLUTION_SCALE = 2.5;

function getWidgetResolutionScale(width: number, height: number): number {
  const maxDimension = Math.max(width, height);
  if (maxDimension <= 600) return VR_RESOLUTION_SCALE;
  // Scale down for large widgets to save memory
  return Math.max(1.5, VR_RESOLUTION_SCALE * (600 / maxDimension));
}

// Render at higher resolution, scale down visually
const scaledWidth = widget.width * resolutionScale;
const scaledHeight = widget.height * resolutionScale;
const inverseScale = 1 / resolutionScale;

<Html scale={inverseScale} style={{ width: scaledWidth, height: scaledHeight }}>
  {/​* Content renders at higher resolution */​}
</​Html>

Reference Files

Troubleshooting

Issue: VR shows flat screen instead of immersive 3D

Cause: <Html> components rendering in XR session Fix: Check isPresenting and skip Html when true

Issue: Widget positions don't match between 2D and 3D

Cause: Missing Y-axis inversion or eye height offset Fix: Use toSpatialPosition() / toDOMPosition() consistently

Issue: Widgets look pixelated in VR

Cause: Default resolution too low for VR displays Fix: Apply VR_RESOLUTION_SCALE multiplier to Html content

Issue: Widget interactions don't work in VR

Cause: Pointer events not reaching 3D meshes Fix: Ensure pointerEvents: 'auto' on Html or use onClick on meshes