Building a Meta Quest VR Viewer with Next.js + React Three Fiber: @react-three/xr v6 Implementation Notes

Background

A VR viewer for viewing IIIF (International Image Interoperability Framework) images at full scale was built with A-Frame in a previous article. That was later rewritten using Next.js 16 + React Three Fiber (r3f) 9, and while the 3D rendering was working, the app still couldn't deliver a proper VR experience on Meta Quest.

@react-three/xr was installed but never actually used anywhere in the codebase.

This article documents the problems encountered — including dead ends — while adding WebXR (Meta Quest) support to an existing r3f app.

Tech Stack

Item	Version
Next.js	16.2.1
React	19.2.4
React Three Fiber	9.5.0
@react-three/xr	6.6.29
Three.js	0.183.2
Target device	Meta Quest 3

Step 1: Basic Setup for @react-three/xr v6

In v6, the <VRButton> component from v5 and earlier is deprecated. A store-based API is used instead.

// src/lib/xrStore.ts
import { createXRStore } from "@react-three/xr";
export const xrStore = createXRStore();

Wrap the Canvas contents in <XR> and start a VR session by calling xrStore.enterVR().

// Scene.tsx
<Canvas>
  <XR store={xrStore}>
    <Suspense fallback={null}>
      <SceneContent />
    </Suspense>
  </XR>
</Canvas>

// VRButton.tsx (DOM side)
<button onClick={() => xrStore.enterVR()}>VR</button>

The VR button is shown only on WebXR-capable devices, based on navigator.xr?.isSessionSupported("immersive-vr").

Step 2: Controller Locomotion

The target control scheme mirrors VRChat:

Left stick: move
Right stick: 30° snap turn

@react-three/xr v6 provides a useXRControllerLocomotion hook that accepts a reference to the target Object3D to move.

// VRLocomotion.tsx
import { useXRControllerLocomotion } from "@react-three/xr";

export default function VRLocomotion({ originRef }) {
  useXRControllerLocomotion(
    originRef,         // movement target: ref to XROrigin
    { speed: 2 },      // left stick → move
    { type: "snap", degrees: 30, deadZone: 0.5 }, // right stick → rotate
    "left",            // left hand handles movement
  );
  return null;
}

<XROrigin> is a component provided by r3f/xr that represents the player's "floor anchor." The XR camera is added as a child of this group, so moving the group moves the camera (i.e., the viewpoint).

// Inside SceneContent in Scene.tsx
<XROrigin ref={originRef} />
<VRLocomotion originRef={originRef} />

Pitfall 1: CameraRig Hijacks the XR Camera

Symptom

Stick input is received. XROrigin position is changing. But nothing moves in VR space.

Cause

The debug log revealed the answer:

Before VR: children=[PerspectiveCamera]  ← XR camera is a child of XROrigin
During VR: children=[]                   ← XR camera is gone!

The culprit was the CameraRig component. It was originally written to reparent the camera into its own group for non-VR use:

// CameraRig.tsx (problematic code)
const { camera } = useThree();

useEffect(() => {
  rigRef.current.add(camera); // ← when the XR session starts, camera is replaced by the XR camera
  // → this steals the XR camera away from XROrigin!
}, [camera]);

r3f's useThree() swaps camera for the XR camera (gl.xr.getCamera()) when an XR session starts. Since this is in the useEffect dependency array, CameraRig calls add() on the XR camera the moment a session begins, removing it from XROrigin.

In Three.js, an object can only have one parent. Calling add() automatically detaches it from its previous parent.

Fix

Skip reparenting during an XR session.

useEffect(() => {
  if (!rigRef.current) return;
  if (xrStore.getState().session) return; // skip during VR
  rigRef.current.add(camera);
  // ...
}, [camera, scene, startPosition]);

Pitfall 2: The callback vs. ref Form of useXRControllerLocomotion

useXRControllerLocomotion accepts either a ref or a callback function as the target. Using the callback form for debugging caused the avatar to instantly fly hundreds of meters away.

Inspecting the library source revealed the cause:

// Internal implementation of @pmndrs/xr (summarized)

// ref form: deltaTime applied internally, XZ plane only
target.position.x += velocity.x * delta;
target.position.z += velocity.z * delta;

// callback form: velocity is passed raw (deltaTime passed separately as the 3rd argument)
target(vectorHelper, rotationY, deltaTime, state, frame);

	ref form	callback form
deltaTime	Applied internally	Passed as 3rd argument — must be multiplied manually
Y-axis movement	Excluded automatically	Y component included based on camera orientation
Use case	Normal locomotion	Custom movement logic

Lesson: Use the ref form for normal locomotion. The callback form is for cases requiring custom movement logic.

Pitfall 3: position Prop Resets on Re-render

Initially, XROrigin was given a position prop directly.

// Bad: position prop resets on every React re-render
<XROrigin ref={originRef} position={camStartPos} />

Whenever a store state change caused SceneContent to re-render, the position prop was reapplied, rolling back any locomotion movement.

The fix is to set the initial position once using a ref callback.

const originInitialized = useRef(false);

<XROrigin
  ref={(node) => {
    if (node && !originInitialized.current) {
      node.position.set(...camStartPos);
      originInitialized.current = true;
    }
    originRef.current = node;
  }}
/>

Quest Device Debugging

Browser DevTools are not available inside the Quest's VR environment. USB-based remote debugging is possible but cumbersome.

The approach adopted here was to send logs to the PC terminal via an API route.

// src/app/api/vr-debug/route.ts
export async function POST(req: NextRequest) {
  if (process.env.NODE_ENV !== "development") {
    return NextResponse.json({ error: "not available" }, { status: 404 });
  }
  const body = await req.json();
  console.log("[VR]", body.msg);
  return NextResponse.json({ ok: true });
}

// VR component side
function sendDebug(msg: string) {
  fetch("/api/vr-debug", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ msg }),
  }).catch(() => {});
}

Logs stream to the PC terminal in real time:

[VR] idle pos=(0.00,0.00,3.00) children=[PerspectiveCamera]
[VR] MOVED pos=(-0.75,0.00,2.42) children=[] rot=0.00

The children=[] output was what revealed the CameraRig problem. In production builds, the process.env.NODE_ENV check disables the endpoint.

Accessing the Dev Server from Quest

Accessing the PC development server from the Quest browser requires a few setup steps.

1. Hostname configuration

npm run dev -- --hostname 0.0.0.0 --experimental-https

--hostname 0.0.0.0 allows connections from all network interfaces. WebXR requires a secure context, so --experimental-https is also needed.

2. Cross-origin permission

In Next.js 16, HMR WebSocket connections from external origins are blocked by default.

// next.config.ts
const nextConfig: NextConfig = {
  allowedDevOrigins: ["192.168.11.58"], // PC's IP address
};

3. Accessing from Quest

Open https://192.168.11.58:3000 in the Quest browser. Accept the self-signed certificate warning via "Advanced → Proceed to unsafe page."

Three.js 0.183 Deprecation Warnings

Three.js 0.183 deprecated THREE.Clock and PCFSoftShadowMap.

PCFSoftShadowMap

r3f's <Canvas shadows> uses PCFSoftShadowMap when shadows={true}. Specifying "percentage" switches to PCFShadowMap.

<Canvas shadows="percentage">

THREE.Clock

@react-three/fiber internally calls new THREE.Clock(), which is outside the app's control. The console.warn in the constructor was silenced with patch-package:

npm install --save-dev patch-package
# Edit node_modules/three/build/three.cjs and three.core.js
npx patch-package three

Adding "postinstall": "patch-package" to package.json applies the patch automatically after each npm install.

Final Architecture

Canvas
 └─ <XR store={xrStore}>
     └─ <Suspense>
         └─ <SceneContent>
             ├─ Sky, Lighting, Room, Garden
             ├─ IIIF ImagePlanes
             ├─ <XROrigin ref={originRef}>  ← parent of VR camera
             ├─ <VRLocomotion>              ← stick-based movement
             ├─ <CameraRig>                 ← non-VR camera control
             ├─ <Avatar>
             └─ <PlayerControls>            ← non-VR WASD/mouse

VR mode: XROrigin is the parent of the XR camera. VRLocomotion moves XROrigin based on stick input. CameraRig and PlayerControls are skipped via xrStore.getState().session checks.
Non-VR mode: CameraRig manages the camera. PlayerControls handles keyboard, mouse, and touch input. useXRControllerLocomotion in VRLocomotion silently does nothing when no controllers are found.

Summary

What was done	Key point
Integrated `<XR>` + `<XROrigin>`	v6 is store-based: `createXRStore()` → `<XR store>`
Controller locomotion	Use the ref form of `useXRControllerLocomotion`
CameraRig fix	Skip camera reparenting during XR sessions
position prop fix	Set initial position once using a ref callback
Quest debugging	Send logs to PC terminal via API route
HTTPS + cross-origin	`--experimental-https` + `allowedDevOrigins`
Playwright video capture	Read coordinates from the live scene and automate recording

The biggest stumbling block was CameraRig hijacking the XR camera. The fundamental Three.js rule that an object can only have one parent surfaces in unexpected ways when combining r3f and XR.

The key to debugging was remote logging via API route. Since DevTools are unavailable inside a VR headset, finding other ways to visualize state is essential.

Pitfall 4: Playwright VR Video Capture — Coordinate Calculation Errors

Playwright was used to automate the browser for capturing an introductory video of the VR space. The goal was to film the camera viewing a region specified by an IIIF xywh parameter (e.g., xywh=14550,18540,300,344), but the correct location never appeared on screen no matter how many attempts were made.

Failure 1: Pixel-to-world coordinate conversion doesn't match

The images are placed as meshes on the floor. The formula for converting pixel coordinates to world coordinates is:

// Derived from the tile placement logic in IiifImagePlane.tsx
worldX = centerX - widthM / 2 + (pixelX / pxW) * widthM;
worldZ = centerZ - heightM / 2 + (pixelY / pxH) * heightM;

The formula itself was correct, but the values being substituted were wrong.

Failure 2: Offline calculation vs. actual scene

The first approach fetched the IIIF Collection API directly from Playwright and computed image dimensions offline.

// Offline calculation (wrong)
widthM = 5;  // fallback
heightM = 5 * (pxH / pxW);

In the actual app, however, images that include a physicalScale service in the IIIF manifest are placed at true physical scale.

	Offline calculation	Actual scene
widthM	5.00 m	6.216 m
heightM	2.855 m	3.549 m
centerZ	2.871	0.833
startZ	6.04	4.28

Nearly every value was wrong. The cause was that Playwright's simplified implementation missed the physicalScale parsing from the IIIF manifest.

Failure 3: Camera pitch not accounted for

Even with correct coordinates, there is another problem. Standing directly above a target point and looking straight ahead means the camera is looking at the floor ahead, not at the player's feet.

sensitivity = 0.0025 rad/px
drag(0, 200) → pitch = -0.5 rad (28° downward)

Eye height 1.52m, looking 28° down:
  visible distance = 1.52 / tan(28°) ≈ 2.8m ahead

The camera was looking at a point 2.8m ahead on the floor, not directly below.

Fix: Read coordinates directly from the live scene

Offline calculation was abandoned in favor of exposing scene data from the running React app via window.

// Scene.tsx — debug window exposure
useEffect(() => {
  if (iiifImages.length > 0) {
    (window as any).__VR_DEBUG = {
      images: iiifImages.map((img) => ({
        centerX: img.centerX,
        centerZ: img.centerZ,
        widthM: img.data.widthM,
        heightM: img.data.heightM,
        pxW: img.data.pxW,
        pxH: img.data.pxH,
      })),
      roomHalfW, roomHalfD,
      startZ: roomHalfD - 1,
    };
  }
}, [iiifImages, roomHalfW, roomHalfD]);

Reading the data from Playwright:

await page.waitForFunction(() => window.__VR_DEBUG?.images?.length > 0);
const debug = await page.evaluate(() => window.__VR_DEBUG);

// Coordinate conversion using actual values
const nearest = debug.images[2]; // nearest image
const worldX = nearest.centerX - nearest.widthM / 2
             + (TARGET_PX_X / nearest.pxW) * nearest.widthM;
const worldZ = nearest.centerZ - nearest.heightM / 2
             + (TARGET_PX_Y / nearest.pxH) * nearest.heightM;

Playwright Capture Script Structure

The final script follows this flow:

// 1. Load page and retrieve scene data
await page.goto(url);
const debug = await page.evaluate(() => window.__VR_DEBUG);

// 2. Compute angle and distance to target
const dx = worldX - 0;
const dz = worldZ - startZ;
const angle = Math.atan2(-dx, -dz);
const dist = Math.sqrt(dx * dx + dz * dz);

// 3. Face the target (yaw rotation)
const yawPixels = Math.round(angle / 0.0025);
await drag(-yawPixels, 0, 1500);

// 4. Look slightly down and walk forward
await drag(0, 150, 800);
await hold("KeyW", dist / 2 * 1000); // speed = 2m/s

// 5. Look straight down to confirm target
await drag(0, 400, 1200);

// 6. Crouch → prone → stand
await page.keyboard.press("KeyC"); // crouch
await page.keyboard.press("KeyC"); // prone
await page.keyboard.press("KeyC"); // stand

recordVideo records automatically; calling page.close() saves the video file.

Lessons

Read 3D scene coordinates at runtime. Re-implementing IIIF metadata parsing is error-prone; exposing data via window and reading it from Playwright is more reliable.
Include camera pitch in the calculation. Player position ≠ where the line of sight lands on the floor.
Playwright recordVideo requires headless: false + --use-gl=angle. WebGL rendering does not work in headless mode.

🥽Building a Meta Quest VR Viewer with Next.js + React Three Fiber: @react-three/xr v6 Implementation Notes

Background

Tech Stack

Step 1: Basic Setup for @react-three/xr v6

Step 2: Controller Locomotion

Pitfall 1: CameraRig Hijacks the XR Camera

Symptom

Cause

Fix

Pitfall 2: The callback vs. ref Form of useXRControllerLocomotion

Pitfall 3: position Prop Resets on Re-render

Quest Device Debugging

Accessing the Dev Server from Quest

1. Hostname configuration

2. Cross-origin permission

3. Accessing from Quest

Three.js 0.183 Deprecation Warnings

PCFSoftShadowMap

THREE.Clock

Final Architecture

Summary

Pitfall 4: Playwright VR Video Capture — Coordinate Calculation Errors

Failure 1: Pixel-to-world coordinate conversion doesn't match

Failure 2: Offline calculation vs. actual scene

Failure 3: Camera pitch not accounted for

Fix: Read coordinates directly from the live scene

Playwright Capture Script Structure

Lessons

🚶Retargeting Mixamo Walk Animation onto an Avaturn Avatar: From Blender CLI to Three.js Runtime

🏯IIIF VR Viewer: Experiencing Cultural Resources at Real Scale with WebXR and A-Frame

🗂Using @react-three/fiber and drei with Next 15 (Using React 19)

Comments

Background

Tech Stack

Step 1: Basic Setup for @react-three/xr v6

Step 2: Controller Locomotion

Pitfall 1: CameraRig Hijacks the XR Camera

Symptom

Cause

Fix

Pitfall 2: The callback vs. ref Form of useXRControllerLocomotion

Pitfall 3: position Prop Resets on Re-render

Quest Device Debugging

Accessing the Dev Server from Quest

1. Hostname configuration

2. Cross-origin permission

3. Accessing from Quest

Three.js 0.183 Deprecation Warnings

PCFSoftShadowMap

THREE.Clock

Final Architecture

Summary

Pitfall 4: Playwright VR Video Capture — Coordinate Calculation Errors

Failure 1: Pixel-to-world coordinate conversion doesn't match

Failure 2: Offline calculation vs. actual scene

Failure 3: Camera pitch not accounted for

Fix: Read coordinates directly from the live scene

Playwright Capture Script Structure

Lessons

Related Articles

🚶Retargeting Mixamo Walk Animation onto an Avaturn Avatar: From Blender CLI to Three.js Runtime

🏯IIIF VR Viewer: Experiencing Cultural Resources at Real Scale with WebXR and A-Frame

🗂Using @react-three/fiber and drei with Next 15 (Using React 19)

Comments