Post-Processing with WebGPURenderer

Summary: This guide explains how to use the new post-processing stack built into WebGPURenderer, including MRT support, effect composition via TSL nodes, tone mapping control, and advanced MRT packing techniques.

Overview

The previous post-processing for `WebGLRenderer` had many conceptual issues. Making use of Multiple Render Targets (MRT) was cumbersome due to the limited support in the renderer and there was no automatic pass/effect combination to improve the overall performance.

The new post-processing stack for `WebGPURenderer` was designed to support these use cases right from the beginning.

- `WebGPURenderer` comes with full, built-in MRT support.
- The system combines effects if possible which reduces the overall number of render passes.
- The effect chain is expressed as a node composition which allows a more flexible effect setup.

Let's find out how to integrate post-processing in three.js applications.

Basics

First, please read the instructions in the guide about [WebGPURenderer](webgpurenderer) to correctly configure your imports. After that, you can create an instance of the render pipleine module like so:

The instance of `RenderPipeline` replaces the previous instance of `EffectComposer`. To make sure you actually use the output of the module, you have to update your animation loop like so:

Many post-processing setups start with a so called "scene pass" or "beauty pass" that represents the image of you rendered scene. This image should be subsequently enhanced by different effects like Bloom, Depth-of-Field or SSR. Start by importing the `pass()` TSL function from the TSL namespace and use it to create the pass.

The basic idea of the node system is to represent materials or post-processing effects as node compositions. To configure a basic Dotscreen and RGB shift effect, you create effect nodes with TSL functions and compose them together.

When you are done, you can simply assign the final node to the `RenderPipeline` instance.

-  renderer.render( scene, camera );
+  renderPipeline.render();

import { pass } from 'three/tsl';

// in your init routine

const scenePass = pass( scene, camera );

import { pass } from 'three/tsl';
+  import { dotScreen } from 'three/addons/tsl/display/DotScreenNode.js';
+  import { rgbShift } from 'three/addons/tsl/display/RGBShiftNode.js';

// in your init routine

const scenePass = pass( scene, camera );

+  const dotScreenPass = dotScreen( scenePass );
+  const rgbShiftPass = rgbShift( dotScreenPass );

Tone Mapping and Color Spaces

When using post-processing, tone mapping and color space conversion are automatically applied at the end of your effect chain. Sometimes you want full control over how and when these steps are executed though. For example if you want to apply FXAA with `FXAANode` or color grading with `Lut3DNode`, you can disable automatic tone mapping and color space conversion and apply it via `renderOutput()` by yourself.

It is not mandatory to use `renderOutput()`, you can also implement a custom tone mapping and color space conversion based on your requirements.

import { pass, renderOutput } from 'three/tsl';
import { fxaa } from 'three/addons/tsl/display/FXAANode.js';

// in your init routine

const renderPipeline = new THREE.RenderPipeline( renderer );
renderPipeline.outputColorTransform = false; // disable default output color transform

const scenePass = pass( scene, camera );
const outputPass = renderOutput( scenePass ); // apply tone mapping and color space conversion here

// FXAA must be computed in sRGB color space

const fxaaPass = fxaa( outputPass );
renderPipeline.outputNode = fxaaPass;

MRT

The new post-processing stack has built-in Multiple Render Targets (MRT) support which is crucial for more advanced setups. MRT allows you to produce multiple outputs in a single render pass. So for example when rendering your scene with TRAA, you need below setup to prepare the inputs for the anti-aliasing.

The configuration object you assign to the `mrt()` TSL function describes the different outputs of the pass. In this case, we save the default output (the scene's beauty) and scene's velocity since we want to setup a TRAA. If you also require the scene's depth, there is no need to configure it as a MRT output. You get it for free in your default output pass if you request it in your app. If you know want to use these outputs in subsequent effects, you can query them as texture nodes.

The MRT configuration varies depending on your setup. There are many different TSL objects like `output`, `velocity`, `normalView` or `emissive` than you can use to save per-fragment data in MRT attachments. To improve performance and avoid hitting memory restrictions, it's important to pack and optimize your data in complex MRT setups. By default all attachments are RGBA16 (Half-Float) in precision which is not necessary for all types of data. As an example, below code queries the `diffuseColor` attachment and sets its format to RGBA8 which cuts down the memory and bandwidth by half.

Below setup for Scree-Space Reflections (SSR) converts the default FP16 normals into RGBA8 colors and packs metalness/roughness into a single attachment. The usage of the `sample()` TSL functions allows to implement custom unpacking. In this instance, it converts the color back to a (normalized) direction vector.

We want to further improve the packing/unpacking features in the future to offer more ways to pack/unpack MRT data. In the meanwhile, please have a look at the [official examples](https://threejs.org/examples/?q=webgpu%20postprocessing) to get an overview about the existing effects and setups.

import { pass, mrt, output, velocity } from 'three/tsl';

// in your init routine

const scenePass = pass( scene, camera );
scenePass.setMRT( mrt( {
  output: output,
  velocity: velocity
} ) );

import { traa } from 'three/addons/tsl/display/TRAANode.js';

// in your init routine

const scenePassColor = scenePass.getTextureNode( 'output' );
const scenePassDepth = scenePass.getTextureNode( 'depth' );
const scenePassVelocity = scenePass.getTextureNode( 'velocity' );

const traaPass = traa( scenePassColor, scenePassDepth, scenePassVelocity, camera );
renderPipeline.outputNode = traaPass;

const diffuseTexture = scenePass.getTexture( 'diffuseColor' );
diffuseTexture.type = THREE.UnsignedByteType;

scenePass.setMRT( mrt( {
  output: output,
  normal: packNormalToRGB( normalView ),
  metalrough: vec2( metalness, roughness )
} ) );

// use RGBA8 instead of RGBA16

const normalTexture = scenePass.getTexture( 'normal' );
normalTexture.type = THREE.UnsignedByteType;

const metalRoughTexture = scenePass.getTexture( 'metalrough' );
metalRoughTexture.type = THREE.UnsignedByteType;

// custom unpacking. use the resulting "sceneNormal" instead of "scenePassNormal"
// in subsequent effects

const sceneNormal = sample( ( uv ) => {

  return unpackRGBToNormal( scenePassNormal.sample( uv ) );

} );


VR

Summary: A guide to building VR applications with three.js using WebXR, covering setup, device considerations, and supporting both VR and non-VR modes.

Introduction

Making a VR app in three.js is pretty simple. You basically just have to tell three.js you want to use WebXR. If you think about it a few things about WebXR should be clear. Which way the camera is pointing is supplied by the VR system itself since the user turns their head to choose a direction to look. Similarly the field of view and aspect will be supplied by the VR system since each system has a different field of view and display aspect.

Let's take an example from the article on making a responsive webpage and make it support VR.

Requirements

Before we get started you're going to need a VR capable device like an Android smartphone, Google Daydream, Oculus Go, Oculus Rift, Vive, Samsung Gear VR., an iPhone with a WebXR browser.

Next, if you are running locally you need to run a simple web server like is covered in the article on setting up.

If the device you are using to view VR is not the same computer you're running on you need to serve your webpage via https or else the browser will not allow using the WebXR API. The server mentioned in the article on setting up called Servez has an option to use https. Check it and start the server.

The note the URLs. You need the one that is your computer's local ipaddress. It will usually start with `192`, `172` or `10`. Type that full address, including the `https://` part into your VR device's browser. Note: Your computer and your VR device need to be on the same local network or WiFi and you probably need to be on a home network. note: Many cafes are setup to disallow this kind of machine to machine connection.

You'll be greeted with an error something like the one below. Click "advanced" and then click *proceed*.

Now you can run your examples.

If you're really going to do WebXR development another thing you should learn about is remote debugging so that you can see console warnings, errors, and of course actually debug your code.

If you just want to see the code work below you can just run the code from this site.

Basic Setup

The first thing we need to do is include the VR support after including three.js.

Then we need to enable three.js's WebXR support and add its VR button to our page.

import * as THREE from 'three';
import {VRButton} from 'three/addons/webxr/VRButton.js';

Render Loop

We need to let three.js run our render loop. Until now we have used a `requestAnimationFrame` loop but to support VR we need to let three.js handle our render loop for us. We can do that by calling `WebGLRenderer.setAnimationLoop` and passing a function to call for the loop.

function render(time) {
  time *= 0.001;

  if (resizeRendererToDisplaySize(renderer)) {
    const canvas = renderer.domElement;
    camera.aspect = canvas.clientWidth / canvas.clientHeight;
    camera.updateProjectionMatrix();
  }

  cubes.forEach((cube, ndx) => {
    const speed = 1 + ndx * .1;
    const rot = time * speed;
    cube.rotation.x = rot;
    cube.rotation.y = rot;
  });

  renderer.render(scene, camera);

  requestAnimationFrame(render);
}

requestAnimationFrame(render);
renderer.setAnimationLoop(render);

Camera Positioning and Units in VR

There is one more detail. We should probably set a camera height that's kind of average for a standing user.

and move the cubes up to be in front of the camera

We set them to `z = -2` since the camera will now be at `z = 0` and camera defaults to looking down the -z axis.

This brings up an extremely important point. **Units in VR are in meters**. In other words **One Unit = One Meter**. This means the camera is 1.6 meters above 0. The cube's centers are 2 meters in front of the camera. Each cube is 1x1x1 meter large. This is important because VR needs to adjust things to the user *in the real world*. That means we need the units used in three.js to match the user's own movements.

And with that we should get 3 spinning cubes in front of the camera with a button to enter VR.

[click here to open in a separate window](/manual/examples/webxr-basic.html)

const cube = new THREE.Mesh(geometry, material);
scene.add(cube);

cube.position.x = x;
cube.position.y = 1.6;
cube.position.z = -2;

Adding a Background

I find that VR works better if we have something surrounding the camera like room for reference so let's add a simple grid cubemap like we covered in the article on backgrounds. We'll just use the same grid texture for each side of the cube which will give as a grid room.

That's better.

[click here to open in a separate window](/manual/examples/webxr-basic-w-background.html)

Note: To actually see VR you will need a WebXR compatible device. I believe most Android Phones can support WebXR using Chrome or Firefox. For iOS you might be able to use this WebXR App though in general WebXR support on iOS is unsupported as of May 2019.

const scene = new THREE.Scene();
{
  const loader = new THREE.CubeTextureLoader();
  const texture = loader.load([
    'resources/images/grid-1024.png',
    'resources/images/grid-1024.png',
    'resources/images/grid-1024.png',
    'resources/images/grid-1024.png',
    'resources/images/grid-1024.png',
    'resources/images/grid-1024.png',
  ]);
  scene.background = texture;
}

VR Device Considerations

To use WebXR on Android or iPhone you'll need a *VR Headset* for phones. You can get them for anywhere from $5 for one made of cardboard to $100. Unfortunately I don't know which ones to recommend. I've purchased 6 of them over the years and they are all of varying quality. I've never paid more than about $25.

Just to mention some of the issues

- **Do they fit your phone**

Phones come in a variety of sizes and so the VR headsets need to match. Many headsets claim to match a large variety of sizes. My experience is the more sizes they match the worse they actually are since instead of being designed for a specific size they have to make compromises to match more sizes. Unfortunately multi-size headsets are the most common type.

- **Can they focus for your face**

Some devices have more adjustments than others. Generally there are at most 2 adjustments. How far the lenses are from your eyes and how far apart the lenses are.

- **Are they too reflective**

Many headsets of a cone of plastic from your eye to the phone. If that plastic is shinny or reflective then it will act like a mirror reflecting the screen and be very distracting.

Few if any of the reviews seem to cover this issue.

- **Are the comfortable on your face.**

Most of the devices rest on your nose like a pair of glasses. That can hurt after a few minutes. Some have straps that go around your head. Others have a 3rd strap that goes over your head. These may or may not help keep the device at the right place.

It turns out for most (all?) devices, you eyes need to be centered with the lenses. If the lenses are slightly above or below your eyes the image gets out of focus. This can be very frustrating as things might start in focus but 45-60 seconds later the device has shifted up or down 1 millimeter and you suddenly realize you've been struggling to focus on a blurry image.

- **Can they support your glasses.**

If you wear eye glasses then you'll need to read the reviews to see if a particular headset works well with eye glasses.

I really can't make any recommendations unfortunately. Google has some cheap recommendations made from cardboard some of them as low as $5 so maybe start there and if you enjoy it then consider upgrading. $5 is like the price of 1 coffee so seriously, give it try!

Types of VR Devices

There are also 3 basic types of devices.

- **3 degrees of freedom (3dof), no input device**

This is generally the phone style although sometimes you can buy a 3rd party input device. The 3 degrees of freedom mean you can look up/down (1), left/right(2) and you can tilt your head left and right (3).

- **3 degrees of freedom (3dof) with 1 input device (3dof)**

This is basically Google Daydream and Oculus GO

These also allow 3 degrees of freedom and include a small controller that acts like a laser pointer inside VR. The laser pointer also only has 3 degrees of freedom. The system can tell which way the input device is pointing but it can not tell where the device is.

- **6 degrees of freedom (6dof) with input devices (6dof)**

These are *the real deal* haha. 6 degrees of freedom means not only do these device know which way you are looking but they also know where your head actually is. That means if you move from left to right or forward and back or stand up / sit down the devices can register this and everything in VR moves accordingly. It's spookily and amazingly real feeling. With a good demo you'll be blown away or at least I was and still am.

Further these devices usually include 2 controllers, one for each hand and the system can tell exactly where your hands are and which way they are oriented and so you can manipulate things in VR by just reaching out, touching, pushing, twisting, etc...

6 degree of freedom devices include the Vive and Vive Pro, the Oculus Rift and Quest, and I believe all of the Windows MR devices.

With all that covered I don't for sure know which devices will work with WebXR. I'm 99% sure that most Android phones will work when running Chrome. You may need to turn on WebXR support in `about:flags`. I also know Google Daydream will also work and similarly you need to enable WebXR support in `about:flags`. Oculus Rift, Vive, and Vive Pro will work via Chrome or Firefox. I'm less sure about Oculus Go and Oculus Quest as both of them use custom OSes but according to the internet they both appear to work.

Supporting both VR and Non-VR

Okay, after that long detour about VR Devices and WebXR there's some things to cover

AFAICT, at least as of r112, there is no easy way to support both VR and non-VR modes with three.js. Ideally if not in VR mode you'd be able to control the camera using whatever means you want, for example the `OrbitControls`, and you'd get some kind of event when switching into and out of VR mode so that you could turn the controls on/off.

If three.js adds some support to do both I'll try to update this article. Until then you might need 2 versions of your site OR pass in a flag in the URL, something like

Then we could add some links in to switch modes

and some CSS to position them

in your code you could use that parameter like this

Whether that's good or bad I don't know. I have a feeling the differences between what's needed for VR and what's needed for non-VR are often very different so for all but the most simple things maybe 2 separate pages are better? You'll have to decide.

Note for various reasons this will not work in the live editor on this site so if you want to check it out click here. It should start in non-VR mode and you can use the mouse or fingers to move the camera. Clicking "Allow VR" should switch to allow VR mode and you should be able to click "Enter VR" if you're on a VR device.

  <canvas id="c"></canvas>

    Allow VR
    Use Non-VR Mode

body {
    margin: 0;
}
#c {
    width: 100%;
    height: 100%;
    display: block;
}
.mode {
  position: absolute;
  right: 1em;
  top: 1em;
}

function main() {
  const canvas = document.querySelector('#c');
  const renderer = new THREE.WebGLRenderer({antialias: true, canvas});
  renderer.xr.enabled = true;
  document.body.appendChild(VRButton.createButton(renderer));

  const fov = 75;
  const aspect = 2;  // the canvas default
  const near = 0.1;
  const far = 5;
  const camera = new THREE.PerspectiveCamera(fov, aspect, near, far);
  camera.position.set(0, 1.6, 0);

  const params = (new URL(document.location)).searchParams;
  const allowvr = params.get('allowvr') === 'true';
  if (allowvr) {
    renderer.xr.enabled = true;
    document.body.appendChild(VRButton.createButton(renderer));
    document.querySelector('#vr').style.display = 'none';
  } else {
    // no VR, add some controls
    const controls = new OrbitControls(camera, canvas);
    controls.target.set(0, 1.6, -2);
    controls.update();
    document.querySelector('#nonvr').style.display = 'none';
  }

Deciding on the Level of VR Support

Above we covered 3 types of VR devices.

- 3DOF no input
- 3DOF + 3DOF input
- 6DOF + 6DOF input

You need to decide how much effort you're willing to put in to support each type of device.

For example the simplest device has no input. The best you can generally do is make it so there are some buttons or objects in the user's view and if the user aligns some marker in the center of the display on those objects for 1/2 a second or so then that button is clicked. A common UX is to display a small timer that will appear over the object indicating if you keep the marker there for a moment the object/button will be selected.

Since there is no other input that's about the best you can do

The next level up you have one 3DOF input device. Generally it can point at things and the user has at least 2 buttons. The Daydream also has a touchpad which provides normal touch inputs.

In any case if a user has this type of device it's far more comfortable for the user to by able to point at things with their controller than it is to make them do it with their head by looking at things.

A similar level to that might be 3DOF or 6DOF device with a game console controller. You'll have to decide what to do here. I suspect the most common thing is the user still has to look to point and the controller is just used for buttons.

The last level is a user with a 6DOF headset and 2 6DOF controllers. Those users will find an experience that is only 3DOF to often be frustrating. Similarly they usually expect to be able to virtually manipulate things with their hands in VR so you'll have to decide if you want to support that or not.

As you can see getting started in VR is pretty easy but actually making something shippable in VR will require lots of decision making and design.

This was a pretty brief intro into VR with three.js. We'll cover some of the input methods in future articles.


VR - Look to Select

Summary: A tutorial on implementing a 'look to select' interaction pattern for Google Cardboard-style VR in Three.js, where users select objects by pointing their head at them. Covers building a PickHelper, creating a visual selection gauge using texture offset animation, and pairing objects for selection feedback.

Introduction

**NOTE: The examples on this page require a VR capable device. Without one they won't work.**

In the previous article we went over a very simple VR example using three.js and we discussed the various kinds of VR systems.

The simplest and possibly most common is the Google Cardboard style of VR which is basically a phone put into a $5 - $50 face mask. This kind of VR has no controller so people have to come up with creative solutions for allowing user input.

The most common solution is "look to select" where if the user points their head at something for a moment it gets selected.

Let's implement "look to select"! We'll start with an example from the previous article and to do it we'll add the `PickHelper` we made in the article on picking.

Adding PickHelper for Basic Object Picking

For an explanation of that code see the article on picking.

To use it we just need to create an instance and call it in our render loop.

In the original picking example we converted the mouse coordinates from CSS pixels into normalized coordinates that go from -1 to +1 across the canvas.

In this case though we will always pick where the camera is facing which is the center of the screen so we pass in `0` for both `x` and `y` which is the center in normalized coordinates.

And with that objects will flash when we look at them.

class PickHelper {
  constructor() {
    this.raycaster = new THREE.Raycaster();
    this.pickedObject = null;
    this.pickedObjectSavedColor = 0;
  }
  pick(normalizedPosition, scene, camera, time) {
    // restore the color if there is a picked object
    if (this.pickedObject) {
      this.pickedObject.material.emissive.setHex(this.pickedObjectSavedColor);
      this.pickedObject = undefined;
    }

    // cast a ray through the frustum
    this.raycaster.setFromCamera(normalizedPosition, camera);
    // get the list of objects the ray intersected
    const intersectedObjects = this.raycaster.intersectObjects(scene.children);
    if (intersectedObjects.length) {
      // pick the first object. It's the closest one
      this.pickedObject = intersectedObjects[0].object;
      // save its color
      this.pickedObjectSavedColor = this.pickedObject.material.emissive.getHex();
      // set its emissive color to flashing red/yellow
      this.pickedObject.material.emissive.setHex((time * 8) % 2 > 1 ? 0xFFFF00 : 0xFF0000);
    }
  }
}

+const pickHelper = new PickHelper();

...
function render(time) {
  time *= 0.001;

  ...

+  // 0, 0 is the center of the view in normalized coordinates.
+  pickHelper.pick({x: 0, y: 0}, scene, camera, time);

Creating a Selection Timer Gauge

Typically we don't want selection to be immediate. Instead we require the user to keep the camera on the thing they want to select for a few moments to give them a chance not to select something by accident.

To do that we need some kind of meter or gauge or some way to convey that the user must keep looking and for how long.

One easy way we could do that is to make a 2 color texture and use a texture offset to slide the texture across a model.

Let's do this by itself to see it work before we add it to the VR example.

First we make an `OrthographicCamera`.

And of course update it if the canvas changes size.

We now have a camera that shows 2 units above and below the center and aspect units left and right.

Next let's make a 2 color texture. We'll use a `DataTexture` which we've used a few other places.

We'll then use that texture on a `TorusGeometry`.

`THREE.MathUtils.mapLinear` takes a value that goes between `fromStart` and `fromEnd` and maps it to a value between `toStart` and `toEnd`. In the case above we're taking `time % 2` which means a value that goes from 0 to 2 and maps that to a value that goes from -0.5 to 0.5.

Textures are mapped to geometry using normalized texture coordinates that go from 0 to 1. That means our 2x1 pixel image, set to the default wrapping mode of `THREE.ClampToEdge`, if we adjust the texture coordinates by -0.5 then the entire mesh will be the first color and if we adjust the texture coordinates by +0.5 the entire mesh will be the second color. In between with the filtering set to `THREE.NearestFilter` we'll be able to move the transition between the 2 colors through the geometry.

const left = -2;    // Use values for left
const right = 2;    // right, top and bottom
const top = 1;      // that match the default
const bottom = -1;  // canvas size.
const near = -1;
const far = 1;
const camera = new THREE.OrthographicCamera(left, right, top, bottom, near, far);

function render(time) {
  time *= 0.001;

  if (resizeRendererToDisplaySize(renderer)) {
    const canvas = renderer.domElement;
    const aspect = canvas.clientWidth / canvas.clientHeight;
+    camera.left = -aspect;
+    camera.right = aspect;
    camera.updateProjectionMatrix();
  }
  ...

function makeDataTexture(data, width, height) {
  const texture = new THREE.DataTexture(data, width, height, THREE.RGBAFormat);
  texture.minFilter = THREE.NearestFilter;
  texture.magFilter = THREE.NearestFilter;
  texture.needsUpdate = true;
  return texture;
}

const cursorColors = new Uint8Array([
  64, 64, 64, 64,       // dark gray
  255, 255, 255, 255,   // white
]);
const cursorTexture = makeDataTexture(cursorColors, 2, 1);

const ringRadius = 0.4;
const tubeRadius = 0.1;
const tubeSegments = 4;
const ringSegments = 64;
const cursorGeometry = new THREE.TorusGeometry(
    ringRadius, tubeRadius, tubeSegments, ringSegments);

const cursorMaterial = new THREE.MeshBasicMaterial({
  color: 'white',
  map: cursorTexture,
  transparent: true,
  blending: THREE.CustomBlending,
  blendSrc: THREE.OneMinusDstColorFactor,
  blendDst: THREE.OneMinusSrcColorFactor,
});
const cursor = new THREE.Mesh(cursorGeometry, cursorMaterial);
scene.add(cursor);

function render(time) {
  time *= 0.001;

  if (resizeRendererToDisplaySize(renderer)) {
    const canvas = renderer.domElement;
    const aspect = canvas.clientWidth / canvas.clientHeight;
    camera.left = -aspect;
    camera.right = aspect;
    camera.updateProjectionMatrix();
  }

+  const fromStart = 0;
+  const fromEnd = 2;
+  const toStart = -0.5;
+  const toEnd = 0.5;
+  cursorTexture.offset.x = THREE.MathUtils.mapLinear(
+      time % 2,
+      fromStart, fromEnd,
+      toStart, toEnd);

  renderer.render(scene, camera);
}

+const backgroundColors = new Uint8Array([
+    0,   0,   0, 255,  // black
+   90,  38,  38, 255,  // dark red
+  100, 175, 103, 255,  // medium green
+  255, 239, 151, 255,  // light yellow
+]);
+const backgroundTexture = makeDataTexture(backgroundColors, 2, 2);
+backgroundTexture.wrapS = THREE.RepeatWrapping;
+backgroundTexture.wrapT = THREE.RepeatWrapping;
+backgroundTexture.repeat.set(4, 4);

const scene = new THREE.Scene();
+scene.background = backgroundTexture;

Notes on Cursor Implementation

A few things to notice **and try**.

- We set the `cursorMaterial`'s `blending`, `blendSrc` and `blendDst` properties as follows:

  ```
    blending: THREE.CustomBlending,
    blendSrc: THREE.OneMinusDstColorFactor,
    blendDst: THREE.OneMinusSrcColorFactor,
  ```

  This gives as an *inverse* type of effect. Comment out those 3 lines and you'll see the difference. I'm just guessing the inverse effect is best here as that way we can hopefully see the cursor regardless of the colors it is over.

- We use a `TorusGeometry` and not a `RingGeometry`.

  For whatever reason the `RingGeometry` uses a flat UV mapping scheme. Because of this if we use a `RingGeometry` the texture slides horizontally across the ring instead of around it like it does above.

  Try it out, change the `TorusGeometry` to a `RingGeometry` (it's just commented out in the example above) and you'll see what I mean.

  The *proper* thing to do (for some definition of *proper*) would be to either use the `RingGeometry` but fix the texture coordinates so they go around the ring. Or else, generate our own ring geometry. But, the torus works just fine. Placed directly in front of the camera with a `MeshBasicMaterial` it will look exactly like a ring and the texture coordinates go around the ring so it works for our needs.

Integrating the Cursor into VR Code

Let's integrate it with our VR code above.

You can see the code above we added all the code to create the cursor geometry, texture, and material and we added it as a child of the camera so it will always be in front of the camera. Note we need to add the camera to the scene otherwise the cursor won't be rendered.

We then check if the thing we're picking this time is the same as it was last time. If so we add the elapsed time to a timer and if the timer reaches its limit we return the selected item.

class PickHelper {
-  constructor() {
+  constructor(camera) {
    this.raycaster = new THREE.Raycaster();
    this.pickedObject = null;
-    this.pickedObjectSavedColor = 0;

+    const cursorColors = new Uint8Array([
+      64, 64, 64, 64,       // dark gray
+      255, 255, 255, 255,   // white
+    ]);
+    this.cursorTexture = makeDataTexture(cursorColors, 2, 1);
+
+    const ringRadius = 0.4;
+    const tubeRadius = 0.1;
+    const tubeSegments = 4;
+    const ringSegments = 64;
+    const cursorGeometry = new THREE.TorusGeometry(
+        ringRadius, tubeRadius, tubeSegments, ringSegments);
+
+    const cursorMaterial = new THREE.MeshBasicMaterial({
+      color: 'white',
+      map: this.cursorTexture,
+      transparent: true,
+      blending: THREE.CustomBlending,
+      blendSrc: THREE.OneMinusDstColorFactor,
+      blendDst: THREE.OneMinusSrcColorFactor,
+    });
+    const cursor = new THREE.Mesh(cursorGeometry, cursorMaterial);
+    // add the cursor as a child of the camera
+    camera.add(cursor);
+    // and move it in front of the camera
+    cursor.position.z = -1;
+    const scale = 0.05;
+    cursor.scale.set(scale, scale, scale);
+    this.cursor = cursor;
+
+    this.selectTimer = 0;
+    this.selectDuration = 2;
+    this.lastTime = 0;
  }
  pick(normalizedPosition, scene, camera, time) {
+    const elapsedTime = time - this.lastTime;
+    this.lastTime = time;

-    // restore the color if there is a picked object
-    if (this.pickedObject) {
-      this.pickedObject.material.emissive.setHex(this.pickedObjectSavedColor);
-      this.pickedObject = undefined;
-    }

+    const lastPickedObject = this.pickedObject;
+    this.pickedObject = undefined;

    // cast a ray through the frustum
    this.raycaster.setFromCamera(normalizedPosition, camera);
    // get the list of objects the ray intersected
    const intersectedObjects = this.raycaster.intersectObjects(scene.children);
    if (intersectedObjects.length) {
      // pick the first object. It's the closest one
      this.pickedObject = intersectedObjects[0].object;
-      // save its color
-      this.pickedObjectSavedColor = this.pickedObject.material.emissive.getHex();
-      // set its emissive color to flashing red/yellow
-      this.pickedObject.material.emissive.setHex((time * 8) % 2 > 1 ? 0xFFFF00 : 0xFF0000);
    }

+    // show the cursor only if it's hitting something
+    this.cursor.visible = this.pickedObject ? true : false;
+
+    let selected = false;
+
+    // if we're looking at the same object as before
+    // increment time select timer
+    if (this.pickedObject && lastPickedObject === this.pickedObject) {
+      this.selectTimer += elapsedTime;
+      if (this.selectTimer >= this.selectDuration) {
+        this.selectTimer = 0;
+        selected = true;
+      }
+    } else {
+      this.selectTimer = 0;
+    }
+
+    // set cursor material to show the timer state
+    const fromStart = 0;
+    const fromEnd = this.selectDuration;
+    const toStart = -0.5;
+    const toEnd = 0.5;
+    this.cursorTexture.offset.x = THREE.MathUtils.mapLinear(
+        this.selectTimer,
+        fromStart, fromEnd,
+        toStart, toEnd);
+
+    return selected ? this.pickedObject : undefined;
  }
}

Creating Paired Box and Sphere Meshes

Now let's use that to pick the cubes. As a simple example we'll add 3 spheres as well. When a cube is picked with hide the cube and un-hide the corresponding sphere.

So first we'll make a sphere geometry.

Then let's create 3 pairs of box and sphere meshes. We'll use a `Map` so that we can associate each `Mesh` with its partner.

In `render` where we rotate the cubes we need to iterate over `meshToMeshMap` instead of `cubes`.

const boxWidth = 1;
const boxHeight = 1;
const boxDepth = 1;
-const geometry = new THREE.BoxGeometry(boxWidth, boxHeight, boxDepth);
+const boxGeometry = new THREE.BoxGeometry(boxWidth, boxHeight, boxDepth);
+
+const sphereRadius = 0.5;
+const sphereGeometry = new THREE.SphereGeometry(sphereRadius);

-const cubes = [
-  makeInstance(geometry, 0x44aa88,  0),
-  makeInstance(geometry, 0x8844aa, -2),
-  makeInstance(geometry, 0xaa8844,  2),
-];
+const meshToMeshMap = new Map();
+[
+  { x:  0, boxColor: 0x44aa88, sphereColor: 0xFF4444, },
+  { x:  2, boxColor: 0x8844aa, sphereColor: 0x44FF44, },
+  { x: -2, boxColor: 0xaa8844, sphereColor: 0x4444FF, },
+].forEach((info) => {
+  const {x, boxColor, sphereColor} = info;
+  const sphere = makeInstance(sphereGeometry, sphereColor, x);
+  const box = makeInstance(boxGeometry, boxColor, x);
+  // hide the sphere
+  sphere.visible = false;
+  // map the sphere to the box
+  meshToMeshMap.set(box, sphere);
+  // map the box to the sphere
+  meshToMeshMap.set(sphere, box);
+});

-cubes.forEach((cube, ndx) => {
+let ndx = 0;
+for (const mesh of meshToMeshMap.keys()) {
  const speed = 1 + ndx * .1;
  const rot = time * speed;
-  cube.rotation.x = rot;
-  cube.rotation.y = rot;
-});
+  mesh.rotation.x = rot;
+  mesh.rotation.y = rot;
+  ++ndx;
+}

Final Selection Logic

And now we can use our new `PickHelper` implementation to select one of the objects. When selected we hide that object and un-hide its partner.

And with that we should have a pretty decent *look to select* implementation.

// 0, 0 is the center of the view in normalized coordinates.
-pickHelper.pick({x: 0, y: 0}, scene, camera, time);
+const selectedObject = pickHelper.pick({x: 0, y: 0}, scene, camera, time);
+if (selectedObject) {
+  selectedObject.visible = false;
+  const partnerObject = meshToMeshMap.get(selectedObject);
+  partnerObject.visible = true;
+}

Conclusion

I hope this example gave some ideas of how to implement a "look to select" type of Google Cardboard level UX. Sliding textures using texture coordinates offsets is also a commonly useful technique.

Next up let's allow the user that has a VR controller to point at and move things.


VR - 3DOF Point to Select

Summary: A tutorial on implementing point-to-select functionality using VR controllers in Three.js, covering raycasting from controller positions, event dispatching, highlighting, and moving objects via controller interaction.

VR - 3DOF Point to Select

**NOTE: The examples on this page require a VR capable device with a pointing device. Without one they won't work. See this article as to why**

In the previous article we went over a very simple VR example where we let the user choose things by pointing via looking. In this article we will take it one step further and let the user choose with a pointing device

Three.js makes is relatively easy by providing 2 controller objects in VR and tries to handle both cases of a single 3DOF controller and two 6DOF controllers. Each of the controllers are Object3D objects which give the orientation and position of that controller. They also provide `selectstart`, `select` and `selectend` events when the user starts pressing, is pressing, and stops pressing (ends) the "main" button on the controller.

Starting with the last example from the previous article let's change the `PickHelper` into a `ControllerPickHelper`.

Our new implementation will emit a `select` event that gives us the object that was picked so to use it we'll just need to do this.

Remember from our previous code `meshToMeshMap` maps our boxes and spheres to each other so if we have one we can look up its partner through `meshToMeshMap` so here we're just hiding the selected object and un-hiding its partner.

As for the actual implementation of `ControllerPickHelper`, first we need to add the VR controller objects to the scene and to those add some 3D lines we can use to display where the user is pointing. We save off both the controllers and their lines.

Without doing anything else this alone would give us 1 or 2 lines in the scene showing where the user's pointing devices are and which way they are pointing.

One problem we have though, we don't want have our `RayCaster` pick the line itself so an easy solution is separate the objects we wanted to be able to pick from the objects we don't by parenting them under another Object3D.

Next let's add some code to pick from the controllers. This is the first time we've picked with something not the camera. In our article on picking the user uses the mouse or finger to pick which means picking comes from the camera into the screen. In the previous article we were picking based on which way the user is looking so again that comes from the camera. This time though we're picking from the position of the controllers so we're not using the camera.

Like before we use a Raycaster but this time we take the ray from the controller. Our previous `PickHelper` there was only one thing picking but here we have up to 2 controllers, one for each hand. We save off which object each controller is looking at in `controllerToObjectMap`. We also save off the original emissive color in `objectToColorMap` and we make the line long enough to touch whatever it's pointing at.

We need to add some code to reset these settings every frame.

Next we want to emit a `select` event when the user clicks the controller. To do that we can extend three.js's EventDispatcher and then we'll check when we get a `select` event from the controller, then if that controller is pointing at something we emit what that controller is pointing at as our own `select` event.

All that is left is to call `update` in our render loop

and assuming you have a VR device with a controller you should be able to use the controllers to pick things.

And what if we wanted to be able to move the objects?

That's relatively easy. Let's move our controller 'select' listener code out into a function so we can use it for more than one thing.

Then let's use it for both `selectstart` and `select`

and let's also pass on the `selectend` event which three.js sends out when you user lets of the button on the controller.

Now let's change the code so when we get a `selectstart` event we'll remove the selected object from the scene and make it a child of the controller. This means it will move with the controller. When we get a `selectend` event we'll put it back in the scene.

When an object is selected we save off that object and its original parent. When the user is done we can put the object back.

We use the Object3D.attach to re-parent the selected objects. These functions let us change the parent of an object without changing its orientation and position in the scene.

And with that we should be able to move the objects around with a 6DOF controller or at least change their orientation with a 3DOF controller

To be honest I'm not 100% sure this `ControllerPickHelper` is the best way to organize the code but it's useful to demonstrating the various parts of getting something simple working in VR in three.js

const pickHelper = new ControllerPickHelper(scene);
pickHelper.addEventListener('select', (event) => {
  event.selectedObject.visible = false;
  const partnerObject = meshToMeshMap.get(event.selectedObject);
  partnerObject.visible = true;
});

class ControllerPickHelper {
  constructor(scene) {
    const pointerGeometry = new THREE.BufferGeometry().setFromPoints([
      new THREE.Vector3(0, 0, 0),
      new THREE.Vector3(0, 0, -1),
    ]);

    this.controllers = [];
    for (let i = 0; i < 2; ++i) {
      const controller = renderer.xr.getController(i);
      scene.add(controller);

      const line = new THREE.Line(pointerGeometry);
      line.scale.z = 5;
      controller.add(line);
      this.controllers.push({controller, line});
    }
  }
}

const scene = new THREE.Scene();
+// object to put pickable objects on so we can easily
+// separate them from non-pickable objects
+const pickRoot = new THREE.Object3D();
+scene.add(pickRoot);

...

function makeInstance(geometry, color, x) {
  const material = new THREE.MeshPhongMaterial({color});

  const cube = new THREE.Mesh(geometry, material);
-  scene.add(cube);
+  pickRoot.add(cube);

...

class ControllerPickHelper {
  constructor(scene) {
+    this.raycaster = new THREE.Raycaster();
+    this.objectToColorMap = new Map();
+    this.controllerToObjectMap = new Map();
+    this.tempMatrix = new THREE.Matrix4();

    const pointerGeometry = new THREE.BufferGeometry().setFromPoints([
      new THREE.Vector3(0, 0, 0),
      new THREE.Vector3(0, 0, -1),
    ]);

    this.controllers = [];
    for (let i = 0; i < 2; ++i) {
      const controller = renderer.xr.getController(i);
      scene.add(controller);

      const line = new THREE.Line(pointerGeometry);
      line.scale.z = 5;
      controller.add(line);
      this.controllers.push({controller, line});
    }
  }
+  update(pickablesParent, time) {
+    this.reset();
+    for (const {controller, line} of this.controllers) {
+      // cast a ray through the from the controller
+      this.tempMatrix.identity().extractRotation(controller.matrixWorld);
+      this.raycaster.ray.origin.setFromMatrixPosition(controller.matrixWorld);
+      this.raycaster.ray.direction.set(0, 0, -1).applyMatrix4(this.tempMatrix);
+      // get the list of objects the ray intersected
+      const intersections = this.raycaster.intersectObjects(pickablesParent.children);
+      if (intersections.length) {
+        const intersection = intersections[0];
+        // make the line touch the object
+        line.scale.z = intersection.distance;
+        // pick the first object. It's the closest one
+        const pickedObject = intersection.object;
+        // save which object this controller picked
+        this.controllerToObjectMap.set(controller, pickedObject);
+        // highlight the object if we haven't already
+        if (this.objectToColorMap.get(pickedObject) === undefined) {
+          // save its color
+          this.objectToColorMap.set(pickedObject, pickedObject.material.emissive.getHex());
+          // set its emissive color to flashing red/yellow
+          pickedObject.material.emissive.setHex((time * 8) % 2 > 1 ? 0xFF2000 : 0xFF0000);
+        }
+      } else {
+        line.scale.z = 5;
+      }
+    }
+  }
}

class ControllerPickHelper {

  ...

+  _reset() {
+    // restore the colors
+    this.objectToColorMap.forEach((color, object) => {
+      object.material.emissive.setHex(color);
+    });
+    this.objectToColorMap.clear();
+    this.controllerToObjectMap.clear();
+  }
  update(pickablesParent, time) {
+    this._reset();

    ...

}

-class ControllerPickHelper {
+class ControllerPickHelper extends THREE.EventDispatcher {
  constructor(scene) {
+    super();
    this.raycaster = new THREE.Raycaster();
    this.objectToColorMap = new Map();  // object to save color and picked object
    this.controllerToObjectMap = new Map();
    this.tempMatrix = new THREE.Matrix4();

    const pointerGeometry = new THREE.BufferGeometry().setFromPoints([
      new THREE.Vector3(0, 0, 0),
      new THREE.Vector3(0, 0, -1),
    ]);

    this.controllers = [];
    for (let i = 0; i < 2; ++i) {
      const controller = renderer.xr.getController(i);
+      controller.addEventListener('select', (event) => {
+        const controller = event.target;
+        const selectedObject = this.controllerToObjectMap.get(controller);
+        if (selectedObject) {
+          this.dispatchEvent({type: 'select', controller, selectedObject});
+        }
+      });
      scene.add(controller);

      const line = new THREE.Line(pointerGeometry);
      line.scale.z = 5;
      controller.add(line);
      this.controllers.push({controller, line});
    }
  }
}

function render(time) {

  ...

+  pickHelper.update(pickablesParent, time);

  renderer.render(scene, camera);
}

class ControllerPickHelper extends THREE.EventDispatcher {
  constructor(scene) {
    super();

    ...

    this.controllers = [];

+    const selectListener = (event) => {
+      const controller = event.target;
+      const selectedObject = this.controllerToObjectMap.get(event.target);
+      if (selectedObject) {
+        this.dispatchEvent({type: 'select', controller, selectedObject});
+      }
+    };

    for (let i = 0; i < 2; ++i) {
      const controller = renderer.xr.getController(i);
-      controller.addEventListener('select', (event) => {
-        const controller = event.target;
-        const selectedObject = this.controllerToObjectMap.get(event.target);
-        if (selectedObject) {
-          this.dispatchEvent({type: 'select', controller, selectedObject});
-        }
-      });
+      controller.addEventListener('select', selectListener);

       ...

class ControllerPickHelper extends THREE.EventDispatcher {
  constructor(scene) {
    super();

    ...

    this.controllers = [];

    const selectListener = (event) => {
      const controller = event.target;
      const selectedObject = this.controllerToObjectMap.get(event.target);
      if (selectedObject) {
-        this.dispatchEvent({type: 'select', controller, selectedObject});
+        this.dispatchEvent({type: event.type, controller, selectedObject});
      }
    };

    for (let i = 0; i < 2; ++i) {
      const controller = renderer.xr.getController(i);
      controller.addEventListener('select', selectListener);
      controller.addEventListener('selectstart', selectListener);

       ...

class ControllerPickHelper extends THREE.EventDispatcher {
  constructor(scene) {
    super();

    ...

    this.controllers = [];

    const selectListener = (event) => {
      const controller = event.target;
      const selectedObject = this.controllerToObjectMap.get(event.target);
      if (selectedObject) {
        this.dispatchEvent({type: event.type, controller, selectedObject});
      }
    };

+    const endListener = (event) => {
+      const controller = event.target;
+      this.dispatchEvent({type: event.type, controller});
+    };

    for (let i = 0; i < 2; ++i) {
      const controller = renderer.xr.getController(i);
      controller.addEventListener('select', selectListener);
      controller.addEventListener('selectstart', selectListener);
+      controller.addEventListener('selectend', endListener);

       ...

const pickHelper = new ControllerPickHelper(scene);
-pickHelper.addEventListener('select', (event) => {
-  event.selectedObject.visible = false;
-  const partnerObject = meshToMeshMap.get(event.selectedObject);
-  partnerObject.visible = true;
-});

+const controllerToSelection = new Map();
+pickHelper.addEventListener('selectstart', (event) => {
+  const {controller, selectedObject} = event;
+  const existingSelection = controllerToSelection.get(controller);
+  if (!existingSelection) {
+    controllerToSelection.set(controller, {
+      object: selectedObject,
+      parent: selectedObject.parent,
+    });
+    controller.attach(selectedObject);
+  }
+});
+
+pickHelper.addEventListener('selectend', (event) => {
+  const {controller} = event;
+  const selection = controllerToSelection.get(controller);
+  if (selection) {
+    controllerToSelection.delete(controller);
+    selection.parent.attach(selection.object);
+  }
+});


Material Feature Table

Summary: A reference table showing which properties and features are supported by each of the five main Mesh materials in three.js (Basic, Lambert, Phong, Standard, Physical).

Material Feature Table

The most common materials in three.js are the Mesh materials. Here is a table showing which material support which features.

The table below compares feature support across the five main Mesh materials. Each column header links to the corresponding material documentation; each table cell with a bullet (•) links to that property's documentation page.

Materials compared: Basic, Lambert, Phong, Standard, Physical.

Feature comparison (• indicates support):

| Feature | Basic | Lambert | Phong | Standard | Physical |
|---|---|---|---|---|---|
| alphaMap | • | • | • | • | • |
| anisotropy |  |  |  |  | • |
| anisotropyMap |  |  |  |  | • |
| anisotropyRotation |  |  |  |  | • |
| aoMap | • | • | • | • | • |
| aoMapIntensity | • | • | • | • | • |
| attenuationColor |  |  |  |  | • |
| attenuationDistance |  |  |  |  | • |
| bumpMap |  | • | • | • | • |
| bumpScale |  | • | • | • | • |
| clearcoat |  |  |  |  | • |
| clearcoatMap |  |  |  |  | • |
| clearcoatNormalMap |  |  |  |  | • |
| clearcoatNormalScale |  |  |  |  | • |
| clearcoatRoughness |  |  |  |  | • |
| clearcoatRoughnessMap |  |  |  |  | • |
| color | • | • | • | • | • |
| combine | • | • | • |  |  |
| displacementBias |  | • | • | • | • |
| displacementMap |  | • | • | • | • |
| displacementScale |  | • | • | • | • |
| emissive |  | • | • | • | • |
| emissiveIntensity |  | • | • | • | • |
| emissiveMap |  | • | • | • | • |
| envMap | • | • | • | • | • |
| envMapIntensity |  |  |  | • | • |
| envMapRotation | • | • | • | • | • |
| flatShading |  | • | • | • | • |
| fog | • | • | • | • | • |
| ior |  |  |  |  | • |
| iridescence |  |  |  |  | • |
| iridescenceIOR |  |  |  |  | • |
| iridescenceMap |  |  |  |  | • |
| iridescenceThicknessMap |  |  |  |  | • |
| iridescenceThicknessRange |  |  |  |  | • |
| lightMap | • | • | • | • | • |
| lightMapIntensity | • | • | • | • | • |
| map | • | • | • | • | • |
| metalness |  |  |  | • | • |
| metalnessMap |  |  |  | • | • |
| normalMap |  | • | • | • | • |
| normalMapType |  | • | • | • | • |
| normalScale |  | • | • | • | • |
| reflectivity | • | • | • |  | • |
| refractionRatio | • | • | • |  |  |
| roughness |  |  |  | • | • |
| roughnessMap |  |  |  | • | • |
| sheen |  |  |  |  | • |
| sheenColor |  |  |  |  | • |
| sheenColorMap |  |  |  |  | • |
| sheenRoughness |  |  |  |  | • |
| sheenRoughnessMap |  |  |  |  | • |
| shininess |  |  | • |  |  |
| specular |  |  | • |  |  |
| specularColor |  |  |  |  | • |
| specularColorMap |  |  |  |  | • |
| specularIntensity |  |  |  |  | • |
| specularIntensityMap |  |  |  |  | • |
| specularMap | • | • | • |  |  |
| thickness |  |  |  |  | • |
| thicknessMap |  |  |  |  | • |
| transmission |  |  |  |  | • |
| transmissionMap |  |  |  |  | • |
| wireframe | • | • | • | • | • |
| wireframeLinecap | • | • | • | • | • |
| wireframeLinejoin | • | • | • | • | • |
| wireframeLinewidth | • | • | • | • | • |