Loading a .OBJ File

Summary: A guide on loading and displaying .OBJ 3D models in three.js, covering the OBJLoader and MTLLoader, handling materials and textures, automatically framing the camera to fit loaded models, and troubleshooting common loading issues.

Loading a .OBJ File

One of the most common things people want to do with three.js is to load and display 3D models. A common format is the .OBJ 3D format so let's try loading one.

Searching the net I found this CC-BY-NC 3.0 windmill 3D model by ahedov. I downloaded the .blend file from that site, loaded it into Blender and exported it as an .OBJ file.

> Note: If you've never used Blender you might be in for a surprise in that Blender does things differently than just about every other program you've ever used. Just be aware you might need to set aside some time to read some basic UI navigation for Blender.
>
> Let me also add that 3D programs in general are giant beasts with 1000s of features. They are some of the most complicated software there is. When I first learned 3D Studio Max in 1996 I read through 70% of the 600 page manual spending a few hours a day for around 3 weeks. That paid off in that when I learned Maya a few years later some of the lessons learned before were applicable to Maya. So, just be aware that if you really want to be able to use 3D software to either build 3D assets or to modify existing ones put it on your schedule and clear sometime to really go through some lessons.

In any case I used these export options. Let's try to display it!

I started with the directional lighting example from the lights article and I combined it with the hemispherical lighting example so I ended up with one HemisphereLight and one DirectionalLight. I also removed all the GUI stuff related to adjusting the lights. I also removed the cube and sphere that were being added to the scene.

Setting up the OBJLoader

From that the first thing we need to do is include the OBJLoader loader in our script. Then to load the .OBJ file we create an instance of OBJLoader, pass it the URL of our .OBJ file, and pass in a callback that adds the loaded model to our scene.

If we run that what happens? Well it's close but we're getting errors about materials since we haven't given the scene any materials and .OBJ files don't have material parameters.

The .OBJ loader can be passed an object of name / material pairs. When it loads the .OBJ file, any material name it finds it will look for the corresponding material in the map of materials set on the loader. If it finds a material that matches by name it will use that material. If not it will use the loader's default material.

{
  const objLoader = new OBJLoader();
  objLoader.load('resources/models/windmill/windmill.obj', (root) => {
    scene.add(root);
  });
}

Loading .MTL Materials

Sometimes .OBJ files come with a .MTL file that defines materials. In our case the exporter also created a .MTL file. .MTL format is plain ASCII so it's easy to look at. Looking at it here we can see there are 2 materials referencing 5 jpg textures but where are the texture files?

All we got was an .OBJ file and an .MTL file.

At least for this model it turns out the textures are embedded in the .blend file we downloaded. We can ask blender to export those files to by picking File->External Data->Unpack All Into Files and then choosing Write Files to Current Directory. This ends up writing the files in the same folder as the .blend file in a sub folder called textures. I copied those textures into the same folder I exported the .OBJ file to.

Now that we have the textures available we can load the .MTL file. First we need to include the MTLLoader. Then we first load the .MTL file. When it's finished loading we add the just loaded materials on to the OBJLoader itself via the setMaterials and then load the .OBJ file.

import * as THREE from 'three';
import {OrbitControls} from 'three/addons/controls/OrbitControls.js';
import {OBJLoader} from 'three/addons/loaders/OBJLoader.js';
import {MTLLoader} from 'three/addons/loaders/MTLLoader.js';

{
  const mtlLoader = new MTLLoader();
  mtlLoader.load('resources/models/windmill/windmill.mtl', (mtl) => {
    mtl.preload();
    objLoader.setMaterials(mtl);
    objLoader.load('resources/models/windmill/windmill.obj', (root) => {
      scene.add(root);
    });
  });
}

Fixing Missing Cloth on Blades

Note that if we spin the model around you'll see the windmill cloth disappears. We need the material on the blades to be double sided, something we went over in the article on materials. There is no easy way to fix this in the .MTL file. Off the top of my head I can think of 3 ways to fix this.

- Loop over all the materials after loading them and set them all to double sided. This solution works but ideally we only want materials that need to be double sided to be double sided because drawing double sided is slower than single sided.
- Manually set a specific material. Looking in the .MTL file there are 2 materials. One called "windmill" and the other called "Material". Through trial and error I figured out the blades use the material called "Material" so we could set that one specifically.
- Realizing that the .MTL file is limited we could just not use it and instead create materials ourselves. In this case we'd need to look up the Mesh object after loading the obj file.

Which one you pick is up to you. 1 is easiest. 3 is most flexible. 2 somewhere in between. For now I'll pick 2.

const mtlLoader = new MTLLoader();
 mtlLoader.load('resources/models/windmill/windmill.mtl', (mtl) => {
   mtl.preload();
   for (const material of Object.values(mtl.materials)) {
     material.side = THREE.DoubleSide;
   }
   ...

const mtlLoader = new MTLLoader();
 mtlLoader.load('resources/models/windmill/windmill.mtl', (mtl) => {
   mtl.preload();
   mtl.materials.Material.side = THREE.DoubleSide;
   ...

objLoader.load('resources/models/windmill/windmill.obj', (root) => {
   const materials = {
     Material: new THREE.MeshPhongMaterial({...}),
     windmill: new THREE.MeshPhongMaterial({...}),
   };
   root.traverse(node => {
     const material = materials[node.material?.name];
     if (material) {
       node.material = material;
     }
   })
   scene.add(root);
 });

Fixing Blocky Appearance with Normal Maps

And with that change you should still see the cloth on the blades when looking from behind but there's one more issue. If we zoom in close we see things are turning blocky.

What's going on? Looking at the textures there are 2 textures labelled NOR for NORmal map. And looking at them they look like normal maps. Normal maps are generally purple where as bump maps are black and white. Normal maps represent the direction of the surface where as bump maps represent the height of the surface.

Looking at the source for the MTLLoader it expects the keyword norm for normal maps so let's edit the .MTL file and now when we load it it will be using the normal maps as normal maps and we can see the back of the blades.

Loading a Different Model

Let's load a different file. Searching the net I found this CC-BY-NC windmill 3D model made by Roger Gerzner / GERIZ.3D Art. It had a .OBJ version already available. Let's load it up (note I removed the .MTL loader for now).

Hmmm, nothing appears. What's the problem? I wonder what size the model is? We can ask THREE.js what size the model is and try to set our camera automatically.

objLoader.load('resources/models/windmill_2/windmill.obj', (root) => {
  scene.add(root);

  const box = new THREE.Box3().setFromObject(root);
  const boxSize = box.getSize(new THREE.Vector3()).length();
  const boxCenter = box.getCenter(new THREE.Vector3());
  console.log(boxSize);
  console.log(boxCenter);

Auto-Framing the Camera

Our camera is currently only showing about 100 units with near at 0.1 and far at 100. Our ground plane is only 40 units across so basically this windmill model is so big, 2000 units, that it's surrounding our camera and all parts of it our outside our frustum.

We could manually fix that but we could also make the camera auto frame our scene. Let's try that. We can then use the box we just computed adjust the camera settings to view the entire scene. Note that there is no right answer on where to put the camera. We could be facing the scene from any direction at any altitude so we'll just have to pick something.

As we went over in the article on cameras the camera defines a frustum. That frustum is defined by the field of view (fov) and the near and far settings. We want to know given whatever field of view the camera currently has, how far away does the camera need to be so the box containing the scene fits inside the frustum assuming the frustum extended forever. In other words let's assume near is 0.00000001 and far is infinity.

Since we know the size of the box and we know the field of view we have this triangle. You can see on the left is the camera and the blue frustum is projecting out in front of it. We just computed the box that contains the windmill. We need to compute how far way the camera should be from the box so that the box appears inside the frustum.

Using basic right triangle trigonometry and SOHCAHTOA, given we know the field of view for the frustum and we know the size of the box we can compute the distance.

Based on that diagram the formula for computing distance is:

distance = halfSizeToFitOnScreen / tangent(halfFovY)

Let's translate that to code. First let's make a function that will compute distance and then move the camera that distance units from the center of the box. We'll then point the camera at the center of the box.

We pass in 2 sizes. The boxSize and the sizeToFitOnScreen. If we just passed in boxSize and used that as sizeToFitOnScreen then the math would make the box fit perfectly inside the frustum. We want a little extra space above and below so we'll pass in a slightly larger size.

function frameArea(sizeToFitOnScreen, boxSize, boxCenter, camera) {
  const halfSizeToFitOnScreen = sizeToFitOnScreen * 0.5;
  const halfFovY = THREE.MathUtils.degToRad(camera.fov * .5);
  const distance = halfSizeToFitOnScreen / Math.tan(halfFovY);

  // compute a unit vector that points in the direction the camera is now
  // from the center of the box
  const direction = (new THREE.Vector3()).subVectors(camera.position, boxCenter).normalize();

  // move the camera to a position distance units way from the center
  // in whatever direction the camera was from the center already
  camera.position.copy(direction.multiplyScalar(distance).add(boxCenter));

  // pick some near and far values for the frustum that
  // will contain the box.
  camera.near = boxSize / 100;
  camera.far = boxSize * 100;

  camera.updateProjectionMatrix();

  // point the camera to look at the center of the box
  camera.lookAt(boxCenter.x, boxCenter.y, boxCenter.z);
}

{
  const objLoader = new OBJLoader();
  objLoader.load('resources/models/windmill_2/windmill.obj', (root) => {
    scene.add(root);
    // compute the box that contains all the stuff
    // from root and below
    const box = new THREE.Box3().setFromObject(root);

    const boxSize = box.getSize(new THREE.Vector3()).length();
    const boxCenter = box.getCenter(new THREE.Vector3());

    // set the camera to frame the box
    frameArea(boxSize * 1.2, boxSize, boxCenter, camera);

    // update the Trackball controls to handle the new size
    controls.maxDistance = boxSize * 10;
    controls.target.copy(boxCenter);
    controls.update();
  });
}

Adjusting Camera Direction and Ground Plane

This almost works. Use the mouse to rotate the camera and you should see the windmill. The problem is the windmill is large and the box's center is at about (0, 770, 0). So, when we move the camera from where it starts (0, 10, 20) to distance units way from the center in the direction the camera is relative to the center that's moving the camera almost straight down below the windmill.

Let's change it to move sideways from the center of the box to in whatever direction the camera is from the center. All we need to do to do that is zero out the y component of the vector from the box to the camera. Then, when we normalize that vector it will become a vector parallel to the XZ plane. In other words parallel to the ground.

If you look at the bottom of the windmill you'll see a small square. That is our ground plane. It's only 40x40 units and so is way too small relative to the windmill. Since the windmill is over 2000 units big let's change the size of the ground plane to something more fitting. We also need to adjust the repeat otherwise our checkerboard will be so fine we won't even be able to see it unless we zoom way way in.

// compute a unit vector that points in the direction the camera is now
// in the xz plane from the center of the box
const direction = (new THREE.Vector3())
    .subVectors(camera.position, boxCenter)
    .multiply(new THREE.Vector3(1, 0, 1))
    .normalize();

const planeSize = 4000;

const loader = new THREE.TextureLoader();
const texture = loader.load('resources/images/checker.png');
texture.wrapS = THREE.RepeatWrapping;
texture.wrapT = THREE.RepeatWrapping;
texture.magFilter = THREE.NearestFilter;
const repeats = planeSize / 200;
texture.repeat.set(repeats, repeats);

Converting TGA Textures to JPG

Let's add the materials back. Like before there is a .MTL file that references some textures but looking at the files I quickly see an issue.

There are TARGA (.tga) files and they are giant!

THREE.js actually has a TGA loader but it's arguably wrong to use it for most use cases. If you're making a viewer where you want to allow users to view random 3D files they find on the net then maybe, just maybe, you might want to load TGA files.

One problem with TGA files are they can't be compressed well at all. TGA only supports very simple compression and looking above we can see the files are not compressed at all as the odds of them being all exactly the same size are extremely low. Further they are 12 megabytes each!!! If we used those files the user would have to download 36meg to see the windmill.

Another issue with TGA is the browser itself has no support for them so loading them is likely going to be slower than loading supported formats like .JPG and .PNG

I'm pretty sure for our purposes converting them to .JPG will be the best option. Looking inside I see they are 3 channels each, RGB, there is no alpha channel. JPG only supports 3 channels so that's a good fit. JPG also supports lossy compression so we can make the files much smaller to download.

Loading the files up they were each 2048x2048. That seemed like a waste to me but of course it depends on your use case. I made them each 1024x1024 and saved them at a 50% quality setting in Photoshop.

We went from 36meg to 0.55meg! Of course the artist might not be pleased with this compression so be sure to consult with them to discuss the tradeoffs.

Now, to use the .MTL file we need to edit it to reference the .JPG files instead of the .TGA files. Fortunately it's a simple text file so it's easy to edit.

newmtl blinn1SG
Ka 0.10 0.10 0.10
Kd 0.00 0.00 0.00
Ks 0.00 0.00 0.00
Ke 0.00 0.00 0.00
Ns 0.060000
Ni 1.500000
d 1.000000
Tr 0.000000
Tf 1.000000 1.000000 1.000000
illum 2
map_Kd windmill_diffuse.jpg
map_Ks windmill_spec.jpg
map_bump windmill_normal.jpg
bump windmill_normal.jpg

Fixing MTL File Issues

Now that the .MTL file points to some reasonable size textures we need to load it so we'll just do like we did above, first load the materials and then set them on the OBJLoader.

Before we actually try it out I ran into some issues that rather than show a failure I'm just going to go over them.

Issue #1: The three MTLLoader creates materials that multiply the material's diffuse color by the diffuse texture map.

That's a useful feature but looking a the .MTL file above the line:

Kd 0.00 0.00 0.00

sets the diffuse color to 0. Texture map * 0 = black! It's possible the modeling tool used to make the windmill did not multiply the diffuse texture map by the diffuse color. That's why it worked for the artists that made this windmill.

To fix this we can change the line to:

Kd 1.00 1.00 1.00

since Texture Map * 1 = Texture Map.

Issue #2: The specular color is also black

The line that starts with Ks specifies the specular color. It's likely the modeling software used to make the windmill did something similar as it did with diffuse maps in that it used the specular map's color for specular highlights. Three.js uses only the red channel of a specular map as input to how much of the specular color to reflect but three still needs a specular color set.

Like above we can fix that by editing the .MTL file.

Issue #3: The windmill_normal.jpg is a normal map not a bump map.

Just like above we just need to edit the .MTL file.

{
  const mtlLoader = new MTLLoader();
  mtlLoader.load('resources/models/windmill_2/windmill-fixed.mtl', (mtl) => {
    mtl.preload();
    const objLoader = new OBJLoader();
    objLoader.setMaterials(mtl);
    objLoader.load('resources/models/windmill/windmill.obj', (root) => {
      root.updateMatrixWorld();
      scene.add(root);
      // compute the box that contains all the stuff
      // from root and below
      const box = new THREE.Box3().setFromObject(root);

      const boxSize = box.getSize(new THREE.Vector3()).length();
      const boxCenter = box.getCenter(new THREE.Vector3());

      // set the camera to frame the box
      frameArea(boxSize * 1.2, boxSize, boxCenter, camera);

      // update the Trackball controls to handle the new size
      controls.maxDistance = boxSize * 10;
      controls.target.copy(boxCenter);
      controls.update();
    });
  });
}

Common Loading Issues

Loading models often runs into these kinds of issues. Common issues include:

- Needing to know the size: Above we made the camera try to frame the scene but that's not always the appropriate thing to do. Generally the most appropriate thing to do is to make your own models or download the models, load them up in some 3D software and look at their scale and adjust if need be.
- Orientation Wrong: THREE.js is generally Y = up. Some modeling packages default to Z = up, some Y = up. Some are settable. If you run into this case where you load a model and it's on its side. You can either hack your code to rotate the model after loading (not recommended), or you can load the model into your favorite modeling package or use some command line tools to rotate the object in the orientation you need it to be just like you'd edit an image for your website rather than download it and apply code to adjust it. Blender even has options when you export to change the orientation.
- No .MTL file or wrong materials or incompatible parameters: Above we used a .MTL file above which helped us load materials but there were issues. We manually edited the .MTL file to fix. It's also common to look inside the .OBJ file to see what materials there are, or to load the .OBJ file in THREE.js and walk the scene and print out all the materials. Then, go modify the code to make custom materials and assign them where appropriate either by making a name/material pair object to pass to the loader instead of loading the .MTL file, OR, after the scene has loaded, walking the scene and fixing things.
- Textures too large: Most 3D models are made for either architecture, movies and commercials, or games. For architecture and movies no one really cares about the size of the textures since. For games people care because games have limited memory but most games run locally. Webpages though you want to load as fast as possible and so you need to look at the textures and try to make them as small as possible and still look good. In fact the first windmill we should arguably done something about the textures. They are currently a total of 10meg!!!

Also remember like we mentioned in the article on textures that textures take memory so a 50k JPG that expands to 4096x4096 will download fast but still take a ton of memory.

Spinning Limitation and Next Steps

The last thing I wanted to show is spinning the windmills. Unfortunately, .OBJ files have no hierarchy. That means all parts of each windmill are basically considered 1 single mesh. You can't spin the blades of the mill as they aren't separated from the rest of the building.

This is one of the main reasons why .OBJ is not really a good format. If I was to guess, the reason it's more common than other formats is because it's simple and doesn't support many features it works more often than not. Especially if you're making something still like an architectural image and there's no need to animate anything it's not a bad way to get static props into a scene.

Next up we'll try loading a gLTF scene. The gLTF format supports many more features.


Loading a .GLTF File

Summary: A tutorial on loading and working with .GLTF files in three.js, covering loading the file, inspecting the scene graph, animating objects, creating paths, and adding shadows.

Loading a .GLTF File

In a previous lesson we loaded an .OBJ file. If you haven't read it you might want to check it out first.

As pointed out over there the .OBJ file format is very old and fairly simple. It provides no scene graph so everything loaded is one large mesh. It was designed mostly as a simple way to pass data between 3D editors.

The gLTF format is actually a format designed from the ground up for be used for displaying graphics. 3D formats can be divided into 3 or 4 basic types.

- 3D Editor Formats: This are formats specific to a single app. .blend (Blender), .max (3d Studio Max), .mb and .ma (Maya), etc...

- Exchange formats: These are formats like .OBJ, .DAE (Collada), .FBX. They are designed to help exchange information between 3D editors. As such they are usually much larger than needed with extra info used only inside 3d editors

- App formats: These are usually specific to certain apps, usually games.

- Transmission formats: gLTF might be the first true transmission format. I suppose VRML might be considered one but VRML was actually a pretty poor format.

gLTF is designed to do some things well that all those other formats don't do:

- Be small for transmission: For example this means much of their large data, like vertices, is stored in binary. When you download a .gLTF file that data can be uploaded to the GPU with zero processing. It's ready as is. This is in contrast to say VRML, .OBJ, or .DAE where vertices are stored as text and have to be parsed. Text vertex positions can easily be 3x to 5x larger than binary.

- Be ready to render: This again is different from other formats except maybe App formats. The data in a glTF file is mean to be rendered, not edited. Data that's not important to rendering has generally been removed. Polygons have been converted to triangles. Materials have known values that are supposed to work everywhere.

gLTF was specifically designed so you should be able to download a glTF file and display it with a minimum of trouble. Let's cross our fingers that's truly the case as none of the other formats have been able to do this.

Loading a gLTF Model

I wasn't really sure what I should show. At some level loading and displaying a gLTF file is simpler than an .OBJ file. Unlike a .OBJ file materials are directly part of the format. That said I thought I should at least load one up and I think going over the issues I ran into might provide some good info.

Searching the net I found this low-poly city by antonmoek which seemed like if we're lucky might make a good example.

Starting with an example from the .OBJ article I removed the code for loading .OBJ and replaced it with code for loading .GLTF

const mtlLoader = new MTLLoader();
mtlLoader.loadMtl('resources/models/windmill/windmill-fixed.mtl', (mtl) => {
  mtl.preload();
  mtl.materials.Material.side = THREE.DoubleSide;
  objLoader.setMaterials(mtl);
  objLoader.load('resources/models/windmill/windmill.obj', (event) => {
    const root = event.detail.loaderRootNode;
    scene.add(root);
    ...
  });
});

{
  const gltfLoader = new GLTFLoader();
  const url = 'resources/models/cartoon_lowpoly_small_city_free_pack/scene.gltf';
  gltfLoader.load(url, (gltf) => {
    const root = gltf.scene;
    scene.add(root);
    ...
  });
}

-import {LoaderSupport} from 'three/addons/loaders/LoaderSupport.js';
-import {OBJLoader} from 'three/addons/loaders/OBJLoader.js';
-import {MTLLoader} from 'three/addons/loaders/MTLLoader.js';
+import {GLTFLoader} from 'three/addons/loaders/GLTFLoader.js';

Inspecting the Scene Graph

I kept the auto framing code as before. We also need to include the GLTFLoader and we can get rid of the OBJLoader. Magic! It just works, textures and all.

Next I wanted to see if I could animate the cars driving around so I needed to check if the scene had the cars as separate entities and if they were setup in a way I could use them.

I wrote some code to dump put the scenegraph to the JavaScript console. Here's the code to print out the scenegraph.

And I just called it right after loading the scene. Running that I got this listing:

function dumpObject(obj, lines = [], isLast = true, prefix = '') {
  const localPrefix = isLast ? '└─' : '├─';
  lines.push(${prefix}${prefix ? localPrefix : ''}${obj.name || '*no-name*'} [${obj.type}]);
  const newPrefix = prefix + (isLast ? '  ' : '│ ');
  const lastNdx = obj.children.length - 1;
  obj.children.forEach((child, ndx) => {
    const isLast = ndx === lastNdx;
    dumpObject(child, lines, isLast, newPrefix);
  });
  return lines;
}

const gltfLoader = new GLTFLoader();
gltfLoader.load('resources/models/cartoon_lowpoly_small_city_free_pack/scene.gltf', (gltf) => {
  const root = gltf.scene;
  scene.add(root);
  console.log(dumpObject(root).join('\n'));

OSG_Scene [Scene]
  └─RootNode_(gltf_orientation_matrix) [Object3D]
    └─RootNode_(model_correction_matrix) [Object3D]
      └─4d4100bcb1c640e69699a87140df79d7fbx [Object3D]
        └─RootNode [Object3D]
          │ ...
          ├─Cars [Object3D]
          │ ├─CAR_03_1 [Object3D]
          │ │ └─CAR_03_1_World_ap_0 [Mesh]
          │ ├─CAR_03 [Object3D]
          │ │ └─CAR_03_World_ap_0 [Mesh]
          │ ├─Car_04 [Object3D]
          │ │ └─Car_04_World_ap_0 [Mesh]
          │ ├─CAR_03_2 [Object3D]
          │ │ └─CAR_03_2_World_ap_0 [Mesh]
          │ ├─Car_04_1 [Object3D]
          │ │ └─Car_04_1_World_ap_0 [Mesh]
          │ ├─Car_04_2 [Object3D]
          │ │ └─Car_04_2_World_ap_0 [Mesh]
          │ ├─Car_04_3 [Object3D]
          │ │ └─Car_04_3_World_ap_0 [Mesh]
          │ ├─Car_04_4 [Object3D]
          │ │ └─Car_04_4_World_ap_0 [Mesh]
          │ ├─Car_08_4 [Object3D]
          │ │ └─Car_08_4_World_ap8_0 [Mesh]
          │ ├─Car_08_3 [Object3D]
          │ │ └─Car_08_3_World_ap9_0 [Mesh]
          │ ├─Car_04_1_2 [Object3D]
          │ │ └─Car_04_1_2_World_ap_0 [Mesh]
          │ ├─Car_08_2 [Object3D]
          │ │ └─Car_08_2_World_ap11_0 [Mesh]
          │ ├─CAR_03_1_2 [Object3D]
          │ │ └─CAR_03_1_2_World_ap_0 [Mesh]
          │ ├─CAR_03_2_2 [Object3D]
          │ │ └─CAR_03_2_2_World_ap_0 [Mesh]
          │ ├─Car_04_2_2 [Object3D]
          │ │ └─Car_04_2_2_World_ap_0 [Mesh]
          ...

Rotating the Cars

From that we can see all the cars happen to be under a parent called "Cars".

So as a simple test I thought I would just try rotating all the children of the "Cars" node around their Y axis.

I looked up the "Cars" node after loading the scene and saved the result.

Then in the render function we can just set the rotation of each child of cars.

And we get... Hmmm, it looks like unfortunately this scene wasn't designed to animate the cars as their origins are not setup for that purpose. The trucks are rotating in the wrong direction.

This brings up an important point which is if you're going to do something in 3D you need to plan ahead and design your assets so they have their origins in the correct places, so they are the correct scale, etc.

+let cars;
{
  const gltfLoader = new GLTFLoader();
  gltfLoader.load('resources/models/cartoon_lowpoly_small_city_free_pack/scene.gltf', (gltf) => {
    const root = gltf.scene;
    scene.add(root);
+    cars = root.getObjectByName('Cars');

+function render(time) {
+  time *= 0.001;  // convert to seconds

  if (resizeRendererToDisplaySize(renderer)) {
    const canvas = renderer.domElement;
    camera.aspect = canvas.clientWidth / canvas.clientHeight;
    camera.updateProjectionMatrix();
  }

+  if (cars) {
+    for (const car of cars.children) {
+      car.rotation.y = time;
+    }
+  }

  renderer.render(scene, camera);

  requestAnimationFrame(render);
}

Fixing Car Orientation

Since I'm not an artist and I don't know blender that well I will hack this example. We'll take each car and parent it to another Object3D. We will then move those Object3D objects to move the cars but separately we can set the car's original Object3D to re-orient it so it's about where we really need it.

Looking back at the scene graph listing it looks like there are really only 3 types of cars, "Car_08", "CAR_03", and "Car_04". Hopefully each type of car will work with the same adjustments.

I wrote this code to go through each car, parent it to a new Object3D, parent that new Object3D to the scene, and apply some per car type settings to fix its orientation, and add the new Object3D a cars array.

This fixes the orientation of the cars.

-let cars;
+const cars = [];
{
  const gltfLoader = new GLTFLoader();
  gltfLoader.load('resources/models/cartoon_lowpoly_small_city_free_pack/scene.gltf', (gltf) => {
    const root = gltf.scene;
    scene.add(root);

-    cars = root.getObjectByName('Cars');
+    const loadedCars = root.getObjectByName('Cars');
+    const fixes = [
+      { prefix: 'Car_08', rot: [Math.PI * .5, 0, Math.PI * .5], },
+      { prefix: 'CAR_03', rot: [0, Math.PI, 0], },
+      { prefix: 'Car_04', rot: [0, Math.PI, 0], },
+    ];
+
+    root.updateMatrixWorld();
+    for (const car of loadedCars.children.slice()) {
+      const fix = fixes.find(fix => car.name.startsWith(fix.prefix));
+      const obj = new THREE.Object3D();
+      car.getWorldPosition(obj.position);
+      car.position.set(0, 0, 0);
+      car.rotation.set(...fix.rot);
+      obj.add(car);
+      scene.add(obj);
+      cars.push(obj);
+    }
     ...

Creating a Driving Path

Now let's drive them around.

Making even a simple driving system is too much for this post but it seems instead we could just make one convoluted path that drives down all the roads and then put the cars on the path. Here's a picture from Blender about half way through building the path.

I needed a way to get the data for that path out of Blender. Fortunately I was able to select just my path and export .OBJ checking "write nurbs".

Opening the .OBJ file I was able to get a list of points which I formatted into this control points array.

THREE.js has some curve classes. The CatmullRomCurve3 seemed like it might work. The thing about that kind of curve is it tries to make a smooth curve going through the points.

In fact putting those points in directly will generate a curve like this (before image), but we want sharper corners. It seemed like if we computed some extra points we could get what we want. For each pair of points we'll compute a point 10% of the way between the 2 points and another 90% of the way between the 2 points and pass the result to CatmullRomCurve3.

This will give us a curve like this (after image).

Here's the code to make the curve. The first part of that code makes a curve. The second part of that code generates 250 points from the curve and then creates an object to display the lines made by connecting those 250 points.

Running the example I didn't see the curve. To make it visible I made it ignore the depth test and render last. And that's when I discovered it was way too small.

Checking the hierarchy in Blender I found out that the artist had scaled the node all the cars are parented to.

Scaling is bad for real time 3D apps. It causes all kinds of issues and ends up being no end of frustration when doing real time 3D. Artists often don't know this because it's so easy to scale an entire scene in a 3D editing program but if you decide to make a real time 3D app I suggest you request your artists to never scale anything. If they change the scale they should find a way to apply that scale to the vertices so that when it ends up making it to your app you can ignore scale.

And, not just scale, in this case the cars are rotated and offset by their parent, the Cars node. This will make it hard at runtime to move the cars around in world space. To be clear, in this case we want cars to drive around in world space which is why these issues are coming up. If something that is meant to be manipulated in a local space, like the moon revolving around the earth this is less of an issue.

curveObject = new THREE.Line(geometry, material);
+    material.depthTest = false;
+    curveObject.renderOrder = 1;

Debugging Scene Graph with Transforms

Going back to the function we wrote above to dump the scene graph, let's dump the position, rotation, and scale of each node.

And the result from running it shows us that Cars in the original scene has had its rotation and scale removed and applied to its children. That suggests either whatever exporter was used to create the .GLTF file did some special work here or more likely the artist exported a different version of the file than the corresponding .blend file, which is why things don't match.

The moral of that is I should have probably downloaded the .blend file and exported myself. Before exporting I should have inspected all the major nodes and removed any transformations.

All these nodes at the top of the hierarchy are also a waste.

Ideally the scene would consist of a single "root" node with no position, rotation, or scale. At runtime I could then pull all the children out of that root and parent them to the scene itself. There might be children of the root like "Cars" which would help me find all the cars but ideally it would also have no translation, rotation, or scale so I could re-parent the cars to the scene with the minimal amount of work.

+function dumpVec3(v3, precision = 3) {
+  return ${v3.x.toFixed(precision)}, ${v3.y.toFixed(precision)}, ${v3.z.toFixed(precision)};
+}

function dumpObject(obj, lines, isLast = true, prefix = '') {
  const localPrefix = isLast ? '└─' : '├─';
  lines.push(${prefix}${prefix ? localPrefix : ''}${obj.name || '*no-name*'} [${obj.type}]);
+  const dataPrefix = obj.children.length
+     ? (isLast ? '  │ ' : '│ │ ')
+     : (isLast ? '    ' : '│   ');
+  lines.push(${prefix}${dataPrefix}  pos: ${dumpVec3(obj.position)});
+  lines.push(${prefix}${dataPrefix}  rot: ${dumpVec3(obj.rotation)});
+  lines.push(${prefix}${dataPrefix}  scl: ${dumpVec3(obj.scale)});
  const newPrefix = prefix + (isLast ? '  ' : '│ ');
  const lastNdx = obj.children.length - 1;
  obj.children.forEach((child, ndx) => {
    const isLast = ndx === lastNdx;
    dumpObject(child, lines, isLast, newPrefix);
  });
  return lines;
}

Moving Cars Along the Path

In any case the quickest though maybe not the best fix is to just adjust the object we're using to view the curve.

Here's what I ended up with. First I adjusted the position of the curve and found values that seemed to work. I then hid it.

Then I wrote code to move the cars along the curve. For each car we pick a position from 0 to 1 along the curve and compute a point in world space using the curveObject to transform the point. We then pick another point slightly further down the curve. We set the car's orientation using lookAt and put the car at the mid point between the 2 points.

and when I ran it I found out for each type of car, their height above their origins are not consistently set and so I needed to offset each one a little.

And the result. Not bad for a few minutes work.

{
  const points = curve.getPoints(250);
  const geometry = new THREE.BufferGeometry().setFromPoints(points);
  const material = new THREE.LineBasicMaterial({color: 0xff0000});
  curveObject = new THREE.Line(geometry, material);
+  curveObject.scale.set(100, 100, 100);
+  curveObject.position.y = -621;
+  curveObject.visible = false;
  material.depthTest = false;
  curveObject.renderOrder = 1;
  scene.add(curveObject);
}

// create 2 Vector3s we can use for path calculations
const carPosition = new THREE.Vector3();
const carTarget = new THREE.Vector3();

function render(time) {
  ...

-  for (const car of cars) {
-    car.rotation.y = time;
-  }

+  {
+    const pathTime = time * .01;
+    const targetOffset = 0.01;
+    cars.forEach((car, ndx) => {
+      // a number between 0 and 1 to evenly space the cars
+      const u = pathTime + ndx / cars.length;
+
+      // get the first point
+      curve.getPointAt(u % 1, carPosition);
+      carPosition.applyMatrix4(curveObject.matrixWorld);
+
+      // get a second point slightly further down the curve
+      curve.getPointAt((u + targetOffset) % 1, carTarget);
+      carTarget.applyMatrix4(curveObject.matrixWorld);
+
+      // put the car at the first point (temporarily)
+      car.position.copy(carPosition);
+      // point the car the second point
+      car.lookAt(carTarget);
+
+      // put the car between the 2 points
+      car.position.lerpVectors(carPosition, carTarget, 0.5);
+    });
+  }

const loadedCars = root.getObjectByName('Cars');
const fixes = [
-  { prefix: 'Car_08', rot: [Math.PI * .5, 0, Math.PI * .5], },
-  { prefix: 'CAR_03', rot: [0, Math.PI, 0], },
-  { prefix: 'Car_04', rot: [0, Math.PI, 0], },
+  { prefix: 'Car_08', y: 0,  rot: [Math.PI * .5, 0, Math.PI * .5], },
+  { prefix: 'CAR_03', y: 33, rot: [0, Math.PI, 0], },
+  { prefix: 'Car_04', y: 40, rot: [0, Math.PI, 0], },
];

root.updateMatrixWorld();
for (const car of loadedCars.children.slice()) {
  const fix = fixes.find(fix => car.name.startsWith(fix.prefix));
  const obj = new THREE.Object3D();
  car.getWorldPosition(obj.position);
-  car.position.set(0, 0, 0);
+  car.position.set(0, fix.y, 0);
  car.rotation.set(...fix.rot);
  obj.add(car);
  scene.add(obj);
  cars.push(obj);
}

Adding Shadows

The last thing I wanted to do is turn on shadows.

To do this I grabbed all the GUI code from the DirectionalLight shadows example in the article on shadows and pasted it into our latest code.

Then, after loading, we need to turn on shadows on all the objects.

I then spent nearly 4 hours trying to figure out why the shadow helpers were not working. It was because I forgot to enable shadows with renderer.shadowMap.enabled = true.

I then adjusted the values until our DirectionLight's shadow camera had a frustum that covered the entire scene. These are the settings I ended up with.

and I set the background color to light blue.

And ... shadows!

{
  const gltfLoader = new GLTFLoader();
  gltfLoader.load('resources/models/cartoon_lowpoly_small_city_free_pack/scene.gltf', (gltf) => {
    const root = gltf.scene;
    scene.add(root);

+    root.traverse((obj) => {
+      if (obj.castShadow !== undefined) {
+        obj.castShadow = true;
+        obj.receiveShadow = true;
+      }
+    });

{
  const color = 0xFFFFFF;
  const intensity = 1;
  const light = new THREE.DirectionalLight(color, intensity);
+  light.castShadow = true;
*  light.position.set(-250, 800, -850);
*  light.target.position.set(-550, 40, -450);

+  light.shadow.bias = -0.004;
+  light.shadow.mapSize.width = 2048;
+  light.shadow.mapSize.height = 2048;

  scene.add(light);
  scene.add(light.target);
+  const cam = light.shadow.camera;
+  cam.near = 1;
+  cam.far = 2000;
+  cam.left = -1500;
+  cam.right = 1500;
+  cam.top = 1500;
+  cam.bottom = -1500;
...

const scene = new THREE.Scene();
-scene.background = new THREE.Color('black');
+scene.background = new THREE.Color('#DEFEFF');

Conclusion

I hope walking through this project was useful and showed some good examples of working though some of the issues of loading a file with a scenegraph.

One interesting thing is that comparing the .blend file to the .gltf file, the .blend file has several lights but they are not lights after being loaded into the scene. A .GLTF file is just a JSON file so you can easily look inside. It consists of several arrays of things and each item in an array is referenced by index else where. While there are extensions in the works they point to a problem with almost all 3d formats. They can never cover every case.

There is always a need for more data. For example we manually exported a path for the cars to follow. Ideally that info could have been in the .GLTF file but to do that we'd need to write our own exporter and some how mark nodes for how we want them exported or use a naming scheme or something along those lines to get data from whatever tool we're using to create the data into our app.

All of that is left as an exercise to the reader.


Backgrounds and Skyboxes

Summary: Extraction fallback content. [stop]

Backgrounds and Skyboxes

##### Backgrounds and Skyboxes

Most of the articles here use a solid color for a background.

Adding as static background can be as simple as setting some CSS. Taking
an example from [the article on making THREE.js responsive](responsive.html)
we only need to change 2 things.

We need to add some CSS to our canvas to set its background to an image

and we need to tell the [`WebGLRenderer`](/docs/#api/en/renderers/WebGLRenderer) to use `alpha` so places we are not
drawing anything are transparent.

And we get a background.

[click here to open in a separate window](/manual/examples/background-css.html)

If we want the background to be able to be affected by [post processing
effects](post-processing.html) then we need to draw the background using
THREE.js.

THREE.js makes this some what simple. We can just set the background of the scene to
a texture.

which gives us

[click here to open in a separate window](/manual/examples/background-scene-background.html)

This gets us a background image but its stretched to fit the screen.

We can solve this issue by setting the `repeat` and `offset` properties of
the texture to show only a portion of image.

and now THREE.js drawing the background. There is no visible difference from
the CSS version at the top but now if we used a [post processing
effect](post-processing.html) the background would be affected too.

[click here to open in a separate window](/manual/examples/background-scene-background-fixed-aspect.html)

Of course a static background is not usually what we want in a 3D scene. Instead
we usually want some kind of  *skybox* . A skybox is just that, box with the sky
draw on it. We put the camera inside the box and it looks like there is a sky in
the background.

The most common way to implement a skybox is to make a cube, apply a texture to
it, draw it from the inside. On each side of the cube put a texture (using
texture coordinates) that looks like some image of the horizon. It's also often
common to use a sky sphere or a sky dome with a texture drawn on it. You can
probably figure that one out on your own. Just make a cube or sphere,
[apply a texture](textures.html), mark it as `THREE.BackSide` so we
render the inside instead of the outside, and either put it in your scene directly
or like above, or, make 2 scenes, a special one to draw the skybox/sphere/dome and the
normal one to draw everything else. You'd use your normal [`PerspectiveCamera`](/docs/#api/en/cameras/PerspectiveCamera) to
draw. No need for the [`OrthographicCamera`](/docs/#api/en/cameras/OrthographicCamera).

Another solution is to use a  *Cubemap* . A Cubemap is a special kind of texture
that has 6 sides, the sides of a cube. Instead of using standard texture
coordinates it uses a direction from the center pointing outward to decide where
to get a color.

Here are the 6 images of a cubemap from the computer history museum in Mountain
View, California.

To use them we use [`CubeTextureLoader`](/docs/#api/en/loaders/CubeTextureLoader) to load them and then use that as a the
scene's background.

At render time we don't need to adjust the texture like we did above

Let's add some controls in so we can rotate the camera.

and try it out. Drag on the example to rotate the camera and see the cubemap
surrounds us.

[click here to open in a separate window](/manual/examples/background-cubemap.html)

Another option is to use an Equirectangular map. This is the kind of picture a
[360 camera](https://google.com/search?q=360+camera) takes.

[Here's one](https://hdrihaven.com/hdri/?h=tears_of_steel_bridge) I found from
[this site](https://hdrihaven.com).

And that's all there is to it.

[click here to open in a separate window](/manual/examples/background-equirectangularmap.html)

Rather than do it at load time you can also convert an equirectangular image
to a cubemap beforehand. [Here's a site that will do it for you](https://matheowis.github.io/HDRI-to-CubeMap/).

<style>
body {
    margin: 0;
}
#c {
    width: 100%;
    height: 100%;
    display: block;
+    background: url(resources/images/daikanyama.jpg) no-repeat center center;
+    background-size: cover;
}
</style>

function main() {
  const canvas = document.querySelector('#c');
-  const renderer = new THREE.WebGLRenderer({antialias: true, canvas});
+  const renderer = new THREE.WebGLRenderer({
+    antialias: true,
+    canvas,
+    alpha: true,
+  });

const loader = new THREE.TextureLoader();
const bgTexture = loader.load('resources/images/daikanyama.jpg');
bgTexture.colorSpace = THREE.SRGBColorSpace;
scene.background = bgTexture;

function render(time) {

   ...

+  // Set the repeat and offset properties of the background texture
+  // to keep the image's aspect correct.
+  // Note the image may not have loaded yet.
+  const canvasAspect = canvas.clientWidth / canvas.clientHeight;
+  const imageAspect = bgTexture.image ? bgTexture.image.width / bgTexture.image.height : 1;
+  const aspect = imageAspect / canvasAspect;
+
+  bgTexture.offset.x = aspect > 1 ? (1 - 1 / aspect) / 2 : 0;
+  bgTexture.repeat.x = aspect > 1 ? 1 / aspect : 1;
+
+  bgTexture.offset.y = aspect > 1 ? 0 : (1 - aspect) / 2;
+  bgTexture.repeat.y = aspect > 1 ? 1 : aspect;

  ...

  renderer.render(scene, camera);

  requestAnimationFrame(render);
}

{
  const loader = new THREE.CubeTextureLoader();
  const texture = loader.load([
    'resources/images/cubemaps/computer-history-museum/pos-x.jpg',
    'resources/images/cubemaps/computer-history-museum/neg-x.jpg',
    'resources/images/cubemaps/computer-history-museum/pos-y.jpg',
    'resources/images/cubemaps/computer-history-museum/neg-y.jpg',
    'resources/images/cubemaps/computer-history-museum/pos-z.jpg',
    'resources/images/cubemaps/computer-history-museum/neg-z.jpg',
  ]);
  scene.background = texture;
}

function render(time) {

   ...

-  // Set the repeat and offset properties of the background texture
-  // to keep the image's aspect correct.
-  // Note the image may not have loaded yet.
-  const canvasAspect = canvas.clientWidth / canvas.clientHeight;
-  const imageAspect = bgTexture.image ? bgTexture.image.width / bgTexture.image.height : 1;
-  const aspect = imageAspect / canvasAspect;
-
-  bgTexture.offset.x = aspect > 1 ? (1 - 1 / aspect) / 2 : 0;
-  bgTexture.repeat.x = aspect > 1 ? 1 / aspect : 1;
-
-  bgTexture.offset.y = aspect > 1 ? 0 : (1 - aspect) / 2;
-  bgTexture.repeat.y = aspect > 1 ? 1 : aspect;

  ...

  renderer.render(scene, camera);

  requestAnimationFrame(render);
}

import {OrbitControls} from 'three/addons/controls/OrbitControls.js';

const fov = 75;
const aspect = 2;  // the canvas default
const near = 0.1;
-const far = 5;
+const far = 100;
const camera = new THREE.PerspectiveCamera(fov, aspect, near, far);
-camera.position.z = 2;
+camera.position.z = 3;

+const controls = new OrbitControls(camera, canvas);
+controls.target.set(0, 0, 0);
+controls.update();

{
-  const loader = new THREE.CubeTextureLoader();
-  const texture = loader.load([
-    'resources/images/cubemaps/computer-history-museum/pos-x.jpg',
-    'resources/images/cubemaps/computer-history-museum/neg-x.jpg',
-    'resources/images/cubemaps/computer-history-museum/pos-y.jpg',
-    'resources/images/cubemaps/computer-history-museum/neg-y.jpg',
-    'resources/images/cubemaps/computer-history-museum/pos-z.jpg',
-    'resources/images/cubemaps/computer-history-museum/neg-z.jpg',
-  ]);
-  scene.background = texture;
+  const loader = new THREE.TextureLoader();
+  const texture = loader.load(
+    'resources/images/equirectangularmaps/tears_of_steel_bridge_2k.jpg',
+    () => {
+      texture.mapping = THREE.EquirectangularReflectionMapping;
+      texture.colorSpace = THREE.SRGBColorSpace;
+      scene.background = texture;
+    });
}


Transparency

Summary: This guide explains how to work with transparency in three.js, covering basic transparency settings, back-face rendering issues, depth sorting problems, and various solutions like DoubleSide, alpha testing, and geometry splitting techniques.

Introduction

Transparency in three.js is both easy and hard.

First we'll go over the easy part. Let's make a scene with 8 cubes placed in a 2x2x2 grid.

We'll start with the example from the article on rendering on demand which had 3 cubes and modify it to have 8. First let's change our `makeInstance` function to take an x, y, and z

-function makeInstance(geometry, color) {
+function makeInstance(geometry, color, x, y, z) {
  const material = new THREE.MeshPhongMaterial({color});

  const cube = new THREE.Mesh(geometry, material);
  scene.add(cube);

-  cube.position.x = x;
+  cube.position.set(x, y, z);

  return cube;
}

Creating 8 Cubes

Then we can create 8 cubes

+function hsl(h, s, l) {
+  return (new THREE.Color()).setHSL(h, s, l);
+}

-makeInstance(geometry, 0x44aa88,  0);
-makeInstance(geometry, 0x8844aa, -2);
-makeInstance(geometry, 0xaa8844,  2);

+{
+  const d = 0.8;
+  makeInstance(geometry, hsl(0 / 8, 1, .5), -d, -d, -d);
+  makeInstance(geometry, hsl(1 / 8, 1, .5),  d, -d, -d);
+  makeInstance(geometry, hsl(2 / 8, 1, .5), -d,  d, -d);
+  makeInstance(geometry, hsl(3 / 8, 1, .5),  d,  d, -d);
+  makeInstance(geometry, hsl(4 / 8, 1, .5), -d, -d,  d);
+  makeInstance(geometry, hsl(5 / 8, 1, .5),  d, -d,  d);
+  makeInstance(geometry, hsl(6 / 8, 1, .5), -d,  d,  d);
+  makeInstance(geometry, hsl(7 / 8, 1, .5),  d,  d,  d);
+}

Camera Adjustments

I also adjusted the camera

const fov = 75;
const aspect = 2;  // the canvas default
const near = 0.1;
-const far = 5;
+const far = 25;
const camera = new THREE.PerspectiveCamera(fov, aspect, near, far);
-camera.position.z = 4;
+camera.position.z = 2;

Background and Lighting

Set the background to white

const scene = new THREE.Scene();
+scene.background = new THREE.Color('white');

Adding a Second Light

And added a second light so all sides of the cubes get some lighting.

-{
+function addLight(...pos) {
  const color = 0xFFFFFF;
  const intensity = 1;
  const light = new THREE.DirectionalLight(color, intensity);
-  light.position.set(-1, 2, 4);
+  light.position.set(...pos);
  scene.add(light);
}
+addLight(-1, 2, 4);
+addLight( 1, -1, -2);

Making Cubes Transparent

To make the cubes transparent we just need to set the `transparent` flag and to set an `opacity` level with 1 being completely opaque and 0 being completely transparent.

function makeInstance(geometry, color, x, y, z) {
-  const material = new THREE.MeshPhongMaterial({color});
+  const material = new THREE.MeshPhongMaterial({
+    color,
+    opacity: 0.5,
+    transparent: true,
+  });

  const cube = new THREE.Mesh(geometry, material);
  scene.add(cube);

  cube.position.set(x, y, z);

  return cube;
}

The Missing Backs Problem

and with that we get 8 transparent cubes

Drag on the example to rotate the view.

So it seems easy but ... look closer. The cubes are missing their backs.

We learned about the `side` material property in the article on materials. So, let's set it to `THREE.DoubleSide` to get both sides of each cube to be drawn.

const material = new THREE.MeshPhongMaterial({
  color,
  map: loader.load(url),
  opacity: 0.5,
  transparent: true,
+  side: THREE.DoubleSide,
});

Depth Sorting Issues

And we get

Give it a spin. It kind of looks like it's working as we can see backs except on closer inspection sometimes we can't.

This happens because of the way 3D objects are generally drawn. For each geometry each triangle is drawn one at a time. When each pixel of the triangle is drawn 2 things are recorded. One, the color for that pixel and two, the depth of that pixel. When the next triangle is drawn, for each pixel if the depth is deeper than the previously recorded depth no pixel is drawn.

This works great for opaque things but it fails for transparent things.

The solution is to sort transparent things and draw the stuff in back before drawing the stuff in front. THREE.js does this for objects like `Mesh` otherwise the very first example would have failed between cubes with some cubes blocking out others. Unfortunately for individual triangles shorting would be extremely slow.

The cube has 12 triangles, 2 for each face, and the order they are drawn is the same order they are built in the geometry so depending on which direction we are looking the triangles closer to the camera might get drawn first. In that case the triangles in the back aren't drawn. This is why sometimes we don't see the backs.

BackSide and FrontSide Hack

For a convex object like a sphere or a cube one kind of solution is to add every cube to the scene twice. Once with a material that draws only the back facing triangles and another with a material that only draws the front facing triangles.

function makeInstance(geometry, color, x, y, z) {
+  [THREE.BackSide, THREE.FrontSide].forEach((side) => {
    const material = new THREE.MeshPhongMaterial({
      color,
      opacity: 0.5,
      transparent: true,
+      side,
    });

    const cube = new THREE.Mesh(geometry, material);
    scene.add(cube);

    cube.position.set(x, y, z);
+  });
}

Draw Order Assumption

Any with that it *seems* to work.

It assumes that the three.js's sorting is stable. Meaning that because we added the `side: THREE.BackSide` mesh first and because it's at the exact same position that it will be drawn before the `side: THREE.FrontSide` mesh.

Intersecting Planes

Let's make 2 intersecting planes (after deleting all the code related to cubes). We'll add a texture to each plane.

const planeWidth = 1;
const planeHeight = 1;
const geometry = new THREE.PlaneGeometry(planeWidth, planeHeight);

const loader = new THREE.TextureLoader();

function makeInstance(geometry, color, rotY, url) {
  const texture = loader.load(url, render);
  const material = new THREE.MeshPhongMaterial({
    color,
    map: texture,
    opacity: 0.5,
    transparent: true,
    side: THREE.DoubleSide,
  });

  const mesh = new THREE.Mesh(geometry, material);
  scene.add(mesh);

  mesh.rotation.y = rotY;
}

makeInstance(geometry, 'pink',       0,             'resources/images/happyface.png');
makeInstance(geometry, 'lightblue',  Math.PI * 0.5, 'resources/images/hmmmface.png');

Texture Rendering on Demand

This time we can use `side: THREE.DoubleSide` since we can only ever see one side of a plane at a time. Also note we pass our `render` function to the texture loading function so that when the texture finishes loading we re-render the scene. This is because this sample is rendering on demand instead of rendering continuously.

And again we see a similar issue.

Splitting Planes Solution

The solution here is to manually split the each pane into 2 panes so that there really is no intersection.

function makeInstance(geometry, color, rotY, url) {
+  const base = new THREE.Object3D();
+  scene.add(base);
+  base.rotation.y = rotY;

+  [-1, 1].forEach((x) => {
    const texture = loader.load(url, render);
+    texture.offset.x = x < 0 ? 0 : 0.5;
+    texture.repeat.x = .5;
    const material = new THREE.MeshPhongMaterial({
      color,
      map: texture,
      opacity: 0.5,
      transparent: true,
      side: THREE.DoubleSide,
    });

    const mesh = new THREE.Mesh(geometry, material);
-    scene.add(mesh);
+    base.add(mesh);

-    mesh.rotation.y = rotY;
+    mesh.position.x = x * .25;
  });
}

Texture Coordinate Manipulation

How you accomplish that is up to you. If I was using modeling package like Blender I'd probably do this manually by adjusting texture coordinates. Here though we're using `PlaneGeometry` which by default stretches the texture across the plane. Like we covered before By setting the `texture.repeat` and `texture.offset` we can scale and move the texture to get the correct half of the face texture on each plane.

The code above also makes a `Object3D` and parents the 2 planes to it. It seemed easier to rotate a parent `Object3D` than to do the math required do it without.

This solution really only works for simple things like 2 planes that are not changing their intersection position.

Alpha Test Solution

For textured objects one more solution is to set an alpha test.

An alpha test is a level of *alpha* below which three.js will not draw the pixel. If we don't draw a pixel at all then the depth issues mentioned above disappear. For relatively sharp edged textures this works pretty well. Examples include leaf textures on a plant or tree or often a patch of grass.

Let's try on the 2 planes. First let's use different textures. The textures above were 100% opaque. These 2 use transparency.

Going back to the 2 planes that intersect (before we split them) let's use these textures and set an `alphaTest`.

function makeInstance(geometry, color, rotY, url) {
  const texture = loader.load(url, render);
  const material = new THREE.MeshPhongMaterial({
    color,
    map: texture,
-    opacity: 0.5,
    transparent: true,
+    alphaTest: 0.5,
    side: THREE.DoubleSide,
  });

  const mesh = new THREE.Mesh(geometry, material);
  scene.add(mesh);

  mesh.rotation.y = rotY;
}

-makeInstance(geometry, 'pink',       0,             'resources/images/happyface.png');
-makeInstance(geometry, 'lightblue',  Math.PI * 0.5, 'resources/images/hmmmface.png');
+makeInstance(geometry, 'white', 0,             'resources/images/tree-01.png');
+makeInstance(geometry, 'white', Math.PI * 0.5, 'resources/images/tree-02.png');

Adding lil-gui Controls

Before we run this let's add a small UI so we can more easily play with the `alphaTest` and `transparent` settings. We'll use lil-gui like we introduced in the article on three.js's scenegraph.

First we'll make a helper for lil-gui that sets every material in the scene to a value

class AllMaterialPropertyGUIHelper {
  constructor(prop, scene) {
    this.prop = prop;
    this.scene = scene;
  }
  get value() {
    const {scene, prop} = this;
    let v;
    scene.traverse((obj) => {
      if (obj.material && obj.material[prop] !== undefined) {
        v = obj.material[prop];
      }
    });
    return v;
  }
  set value(v) {
    const {scene, prop} = this;
    scene.traverse((obj) => {
      if (obj.material && obj.material[prop] !== undefined) {
        obj.material[prop] = v;
        obj.material.needsUpdate = true;
      }
    });
  }
}

const gui = new GUI();
gui.add(new AllMaterialPropertyGUIHelper('alphaTest', scene), 'value', 0, 1)
    .name('alphaTest')
    .onChange(requestRenderIfNotRequested);
gui.add(new AllMaterialPropertyGUIHelper('transparent', scene), 'value')
    .name('transparent')
    .onChange(requestRenderIfNotRequested);

import * as THREE from 'three';
import {OrbitControls} from 'three/addons/controls/OrbitControls.js';
+import {GUI} from 'three/addons/libs/lil-gui.module.min.js';

Alpha Test Limitations

You can see it works but zoom in and you'll see one plane has white lines.

This is the same depth issue from before. That plane was drawn first so the plane behind is not drawn. There is no perfect solution. Adjust the `alphaTest` and/or turn off `transparent` to find a solution that fits your use case.

Conclusion

The take way from this article is perfect transparency is hard. There are issues and trade offs and workarounds.

For example say you have a car. Cars usually have windshields on all 4 sides. If you want to avoid the sorting issues above you'd have to make each window its own object so that three.js can sort the windows and draw them in the correct order.

If you are making some plants or grass the alpha test solution is common.

Which solution you pick depends on your needs.


Multiple Canvases Multiple Scenes

Summary: A guide explaining how to use THREE.js with multiple virtual scenes rendered into a single canvas, addressing the browser's WebGL context limit and resource sharing issues. It covers the scissor test approach, syncing with scroll, making the system generic, using HTML dataset, adding controls, and an alternative using off-screen canvas copying.

Multiple Canvases Multiple Scenes

A common question is how to use THREE.js with multiple canvases. Let's say you want to make an e-commerce site or you want to make a page with lots of 3D diagrams. At first glance it appears easy. Just make a canvas every where you want a diagram. For each canvas make a Renderer. You'll quickly find though that you run into problems.

The browser limits how many WebGL contexts you can have. Typically that limit is around 8 of them. As soon as you create the 9th context the oldest one will be lost.

WebGL resources can not be shared across contexts. That means if you want to load a 10 meg model into 2 canvases and that model uses 20 meg of textures your 10 meg model will have to be loaded twice and your textures will also be loaded twice. Nothing can be shared across contexts. This also means things have to be initialized twice, shaders compiled twice, etc. It gets worse as there are more canvases.

So what's the solution? The solution is one canvas that fills the viewport in the background and some other element to represent each "virtual" canvas. We make a single Renderer and then one Scene for each virtual canvas. We'll then check the positions of the virtual canvas elements and if they are on the screen we'll tell THREE.js to draw their scene at the correct place.

With this solution there is only 1 canvas so we solve both problem 1 and 2 above. We won't run into the WebGL context limit because we will only be using one context. We also won't run into the sharing issues for the same reasons.

Let's start with a simple example with just 2 scenes. First we'll make the HTML

Then we can setup the CSS maybe something like this

We set the canvas to fill the screen and we set its z-index to -1 to make it appear behind other elements. We also need to specify some kind of width and height for our virtual canvas elements since there is nothing inside to give them any size.

Now we'll make 2 scenes each with a light and a camera. To one scene we'll add a cube and to another a diamond.

And then we'll make a function to render each scene only if the element is on the screen. We can tell THREE.js to only render to part of the canvas by turning on the scissor test with Renderer.setScissorTest and then setting both the scissor and the viewport with Renderer.setViewport and Renderer.setScissor.

And then our render function will just first clear the screen and then render each scene.

And here it is

You can see where the first <span> is there's a red cube and where the second span is there's a blue diamond.

<canvas id="c"></canvas>

  I love boxes. Presents come in boxes.
  When I find a new box I'm always excited to find out what's inside.

  When I was a kid I dreamed of going on an expedition inside a pyramid
  and finding a undiscovered tomb full of mummies and treasure.

#c {
  position: fixed;
  left: 0;
  top: 0;
  width: 100%;
  height: 100%;
  display: block;
  z-index: -1;
}
.diagram {
  display: inline-block;
  width: 5em;
  height: 3em;
  border: 1px solid black;
}
.left {
  float: left;
  margin-right: .25em;
}
.right {
  float: right;
  margin-left: .25em;
}

function makeScene(elem) {
  const scene = new THREE.Scene();

  const fov = 45;
  const aspect = 2;  // the canvas default
  const near = 0.1;
  const far = 5;
  const camera = new THREE.PerspectiveCamera(fov, aspect, near, far);
  camera.position.z = 2;
  camera.position.set(0, 1, 2);
  camera.lookAt(0, 0, 0);

  {
    const color = 0xFFFFFF;
    const intensity = 1;
    const light = new THREE.DirectionalLight(color, intensity);
    light.position.set(-1, 2, 4);
    scene.add(light);
  }

  return {scene, camera, elem};
}

function setupScene1() {
  const sceneInfo = makeScene(document.querySelector('#box'));
  const geometry = new THREE.BoxGeometry(1, 1, 1);
  const material = new THREE.MeshPhongMaterial({color: 'red'});
  const mesh = new THREE.Mesh(geometry, material);
  sceneInfo.scene.add(mesh);
  sceneInfo.mesh = mesh;
  return sceneInfo;
}

function setupScene2() {
  const sceneInfo = makeScene(document.querySelector('#pyramid'));
  const radius = .8;
  const widthSegments = 4;
  const heightSegments = 2;
  const geometry = new THREE.SphereGeometry(radius, widthSegments, heightSegments);
  const material = new THREE.MeshPhongMaterial({
    color: 'blue',
    flatShading: true,
  });
  const mesh = new THREE.Mesh(geometry, material);
  sceneInfo.scene.add(mesh);
  sceneInfo.mesh = mesh;
  return sceneInfo;
}

const sceneInfo1 = setupScene1();
const sceneInfo2 = setupScene2();

function renderSceneInfo(sceneInfo) {
  const {scene, camera, elem} = sceneInfo;

  // get the viewport relative position of this element
  const {left, right, top, bottom, width, height} =
      elem.getBoundingClientRect();

  const isOffscreen =
      bottom < 0 ||
      top > renderer.domElement.clientHeight ||
      right < 0 ||
      left > renderer.domElement.clientWidth;

  if (isOffscreen) {
    return;
  }

  camera.aspect = width / height;
  camera.updateProjectionMatrix();

  const positiveYUpBottom = canvasRect.height - bottom;
  renderer.setScissor(left, positiveYUpBottom, width, height);
  renderer.setViewport(left, positiveYUpBottom, width, height);

  renderer.render(scene, camera);
}

function render(time) {
  time *= 0.001;

  resizeRendererToDisplaySize(renderer);

  renderer.setScissorTest(false);
  renderer.clear(true, true);
  renderer.setScissorTest(true);

  sceneInfo1.mesh.rotation.y = time * .1;
  sceneInfo2.mesh.rotation.y = time * .1;

  renderSceneInfo(sceneInfo1);
  renderSceneInfo(sceneInfo2);

  requestAnimationFrame(render);
}

Syncing up

The code above works but there is one minor issue. If your scenes are complicated or if for whatever reason it takes too long to render, the position of the scenes drawn into the canvas will lag behind the rest of the page.

If we give each area a border

And we set the background of each scene

And if we quickly scroll up and down we'll see the issue. Here's an animation of scrolling slowed down by 10x.

We can switch to a different method which has a different tradeoff. We'll switch the canvas's CSS from position: fixed to position: absolute.

Then we'll set the canvas's transform to move it so the top of the canvas is at the top of whatever part the page is currently scrolled to.

position: fixed kept the canvas from scrolling at all while the rest of the page scrolled over it. position: absolute will let the canvas scroll with the rest of the page which means whatever we draw will stick with the page as it scrolls even if we're too slow to render. When we finally get a chance to render then we move the canvas so it matches where the page has been scrolled and then we re-render. This means only the edges of the window will show some un-rendered bits for a moment but the stuff in the middle of the page should match up and not slide. Here's a view of the results of the new method slowed down 10x.

.diagram {
  display: inline-block;
  width: 5em;
  height: 3em;
+  border: 1px solid black;
}

const scene = new THREE.Scene();
+scene.background = new THREE.Color('red');

#c {
-  position: fixed;
+  position: absolute;

function render(time) {
  ...

  const transform = translateY(${window.scrollY}px);
  renderer.domElement.style.transform = transform;

Making it more Generic

Now that we've gotten multiple scenes working let's make this just slightly more generic.

We could make it so the main render function, the one managing the canvas, just has a list of elements and their associated render function. For each element it would check if the element is on screen and if so call the corresponding render function. In this way we'd have a generic system where individual scenes aren't really aware they are being rendered in some smaller space.

Here's the main render function

You can see it loops over sceneElements which it expects is an array of objects each of which have an elem and fn property.

It checks if the element is on screen. If it is it calls fn and passes it the current time and its rectangle.

Now the setup code for each scene just adds itself to the list of scenes

With that we no longer need sceneInfo1 and sceneInfo2 and the code that was rotating the meshes is now specific to each scene.

const sceneElements = [];
function addScene(elem, fn) {
  sceneElements.push({elem, fn});
}

function render(time) {
  time *= 0.001;

  resizeRendererToDisplaySize(renderer);

  renderer.setScissorTest(false);
  renderer.setClearColor(clearColor, 0);
  renderer.clear(true, true);
  renderer.setScissorTest(true);

  const transform = translateY(${window.scrollY}px);
  renderer.domElement.style.transform = transform;

  for (const {elem, fn} of sceneElements) {
    // get the viewport relative position of this element
    const rect = elem.getBoundingClientRect();
    const {left, right, top, bottom, width, height} = rect;

    const isOffscreen =
        bottom < 0 ||
        top > renderer.domElement.clientHeight ||
        right < 0 ||
        left > renderer.domElement.clientWidth;

    if (!isOffscreen) {
      const positiveYUpBottom = renderer.domElement.clientHeight - bottom;
      renderer.setScissor(left, positiveYUpBottom, width, height);
      renderer.setViewport(left, positiveYUpBottom, width, height);

      fn(time, rect);
    }
  }

  requestAnimationFrame(render);
}

{
  const elem = document.querySelector('#box');
  const {scene, camera} = makeScene();
  const geometry = new THREE.BoxGeometry(1, 1, 1);
  const material = new THREE.MeshPhongMaterial({color: 'red'});
  const mesh = new THREE.Mesh(geometry, material);
  scene.add(mesh);
  addScene(elem, (time, rect) => {
    camera.aspect = rect.width / rect.height;
    camera.updateProjectionMatrix();
    mesh.rotation.y = time * .1;
    renderer.render(scene, camera);
  });
}

{
  const elem = document.querySelector('#pyramid');
  const {scene, camera} = makeScene();
  const radius = .8;
  const widthSegments = 4;
  const heightSegments = 2;
  const geometry = new THREE.SphereGeometry(radius, widthSegments, heightSegments);
  const material = new THREE.MeshPhongMaterial({
    color: 'blue',
    flatShading: true,
  });
  const mesh = new THREE.Mesh(geometry, material);
  scene.add(mesh);
  addScene(elem, (time, rect) => {
    camera.aspect = rect.width / rect.height;
    camera.updateProjectionMatrix();
    mesh.rotation.y = time * .1;
    renderer.render(scene, camera);
  });
}

Using HTML Dataset

One last even more generic thing we can do is use HTML dataset. This is a way to add your own data to an HTML element. Instead of using id="..." we'll use data-diagram="..." like this

We can them change the CSS selector to select for that

We'll change the scene setup code to just be a map of names to scene initialization functions that return a scene render function.

And to init we can just use querySelectorAll to find all the diagrams and call the corresponding init function for that diagram.

No change to the visuals but the code is even more generic.

<canvas id="c"></canvas>

-
+
  I love boxes. Presents come in boxes.
  When I find a new box I'm always excited to find out what's inside.

-
+
  When I was a kid I dreamed of going on an expedition inside a pyramid
  and finding a undiscovered tomb full of mummies and treasure.

-.diagram
+*[data-diagram] {
  display: inline-block;
  width: 5em;
  height: 3em;
}

const sceneInitFunctionsByName = {
  'box': () => {
    const {scene, camera} = makeScene();
    const geometry = new THREE.BoxGeometry(1, 1, 1);
    const material = new THREE.MeshPhongMaterial({color: 'red'});
    const mesh = new THREE.Mesh(geometry, material);
    scene.add(mesh);
    return (time, rect) => {
      mesh.rotation.y = time * .1;
      camera.aspect = rect.width / rect.height;
      camera.updateProjectionMatrix();
      renderer.render(scene, camera);
    };
  },
  'pyramid': () => {
    const {scene, camera} = makeScene();
    const radius = .8;
    const widthSegments = 4;
    const heightSegments = 2;
    const geometry = new THREE.SphereGeometry(radius, widthSegments, heightSegments);
    const material = new THREE.MeshPhongMaterial({
      color: 'blue',
      flatShading: true,
    });
    const mesh = new THREE.Mesh(geometry, material);
    scene.add(mesh);
    return (time, rect) => {
      mesh.rotation.y = time * .1;
      camera.aspect = rect.width / rect.height;
      camera.updateProjectionMatrix();
      renderer.render(scene, camera);
    };
  },
};

document.querySelectorAll('[data-diagram]').forEach((elem) => {
  const sceneName = elem.dataset.diagram;
  const sceneInitFunction = sceneInitFunctionsByName[sceneName];
  const sceneRenderFunction = sceneInitFunction(elem);
  addScene(elem, sceneRenderFunction);
});

Adding Controls to each element

Adding interactively, for example a TrackballControls is just as easy. First we add the script for the control.

And then we can add a TrackballControls to each scene passing in the element associated with that scene.

You'll notice we added the camera to the scene and the light to the camera. This makes the light relative to the camera. Since the TrackballControls are moving the camera this is probably what we want. It keeps the light shining on the side of the object we are looking at.

We need up update those controls in our render functions

And now if you drag the objects they'll rotate.

These techniques are used on this site itself. In particular the article about primitives and the article about materials use this technique to add the various examples throughout the article.

One more solution would be to render to an off screen canvas and copy the result to a 2D canvas at each element. The advantage to this solution is there is no limit on how you can composite each separate area. With the previous solution we and a single canvas in the background. With this solution we have normal HTML elements.

The disadvantage is it's slower because a copy has to happen for each area. How much slower depends on the browser and the GPU.

The changes needed are pretty small

First we'll change HTML as we no longer need a canvas in the page

then we'll change the CSS

We've made all canvases fill their container.

Now let's change the JavaScript. First we no longer look up the canvas. Instead we create one. We also just turn on the scissor test at the beginning.

Then for each scene we create a 2D rendering context and append its canvas to the element for that scene

Then when rendering, if the renderer's canvas is not big enough to render this area we increase its size. As well if this area's canvas is the wrong size we change its size. Finally we set the scissor and viewport, render the scene for this area, then copy the result to the area's canvas.

The result looks the same

One other advantage to this solution is you could potentially use OffscreenCanvas to render from a web worker and still use this technique.

import {TrackballControls} from 'three/addons/controls/TrackballControls.js';

function makeScene(elem) {
  const scene = new THREE.Scene();

  const fov = 45;
  const aspect = 2;  // the canvas default
  const near = 0.1;
  const far = 5;
  const camera = new THREE.PerspectiveCamera(fov, aspect, near, far);
  camera.position.set(0, 1, 2);
  camera.lookAt(0, 0, 0);
  scene.add(camera);

  const controls = new TrackballControls(camera, elem);
  controls.noZoom = true;
  controls.noPan = true;

  {
    const color = 0xFFFFFF;
    const intensity = 1;
    const light = new THREE.DirectionalLight(color, intensity);
    light.position.set(-1, 2, 4);
    camera.add(light);
  }

  return {scene, camera, controls};
}

const sceneInitFunctionsByName = {
 'box': (elem) => {
    const {scene, camera, controls} = makeScene(elem);
    const geometry = new THREE.BoxGeometry(1, 1, 1);
    const material = new THREE.MeshPhongMaterial({color: 'red'});
    const mesh = new THREE.Mesh(geometry, material);
    scene.add(mesh);
    return (time, rect) => {
      mesh.rotation.y = time * .1;
      camera.aspect = rect.width / rect.height;
      camera.updateProjectionMatrix();
      controls.handleResize();
      controls.update();
      renderer.render(scene, camera);
    };
  },
  'pyramid': (elem) => {
    const {scene, camera, controls} = makeScene(elem);
    const radius = .8;
    const widthSegments = 4;
    const heightSegments = 2;
    const geometry = new THREE.SphereGeometry(radius, widthSegments, heightSegments);
    const material = new THREE.MeshPhongMaterial({
      color: 'blue',
      flatShading: true,
    });
    const mesh = new THREE.Mesh(geometry, material);
    scene.add(mesh);
    return (time, rect) => {
      mesh.rotation.y = time * .1;
      camera.aspect = rect.width / rect.height;
      camera.updateProjectionMatrix();
      controls.handleResize();
      controls.update();
      renderer.render(scene, camera);
    };
  },
};

<body>
-  <canvas id="c"></canvas>
  ...
</body>

canvas {
  width: 100%;
  height: 100%;
  display: block;
}
*[data-diagram] {
  display: inline-block;
  width: 5em;
  height: 3em;
}

function main() {
  const canvas = document.createElement('canvas');
  const renderer = new THREE.WebGLRenderer({antialias: true, canvas, alpha: true});
  renderer.setScissorTest(true);

  ...

const sceneElements = [];
function addScene(elem, fn) {
  const ctx = document.createElement('canvas').getContext('2d');
  elem.appendChild(ctx.canvas);
  sceneElements.push({elem, ctx, fn});
}

function render(time) {
  time *= 0.001;

  for (const {elem, fn, ctx} of sceneElements) {
    // get the viewport relative position of this element
    const rect = elem.getBoundingClientRect();
    const {left, right, top, bottom, width, height} = rect;
    const rendererCanvas = renderer.domElement;

    const isOffscreen =
        bottom < 0 ||
        top > window.innerHeight ||
        right < 0 ||
        left > window.innerWidth;

    if (!isOffscreen) {
      // make sure the renderer's canvas is big enough
      if (rendererCanvas.width < width || rendererCanvas.height < height) {
        renderer.setSize(width, height, false);
      }

      // make sure the canvas for this area is the same size as the area
      if (ctx.canvas.width !== width || ctx.canvas.height !== height) {
        ctx.canvas.width = width;
        ctx.canvas.height = height;
      }

      renderer.setScissor(0, 0, width, height);
      renderer.setViewport(0, 0, width, height);

      fn(time, rect);

      // copy the rendered scene to this element's canvas
      ctx.globalCompositeOperation = 'copy';
      ctx.drawImage(
          rendererCanvas,
          0, rendererCanvas.height - height, width, height,  // src rect
          0, 0, width, height);                              // dst rect
    }
  }

  requestAnimationFrame(render);
}


Picking

Summary: This page explains two common methods for implementing picking (determining which object a user clicked) in THREE.js: CPU-based raycasting using the RayCaster class, and GPU-based picking using unique object colors rendered offscreen. It includes working code examples, discusses the tradeoffs of each approach, and demonstrates both implementations with a scene of 100 cubes.

Introduction to Picking

*Picking* refers to the process of figuring out which object a user clicked on or touched. There are tons of ways to implement picking each with their tradeoffs. Let's go over the 2 most common.

Probably the most common way of *picking* is by doing raycasting which means to *cast* a ray from the mouse through the frustum of the scene and computing which objects that ray intersects. Conceptually it's very simple.

First we'd take the position of the mouse. We'd convert that into world space by applying the camera's projection and orientation. We'd compute a ray from the near plane of the camera's frustum to the far plane. Then, for every triangle of every object in the scene we'd check if that ray intersects that triangle. If your scene has 1000 objects and each object has 1000 triangles then 1 million triangles will need to be checked.

A few optimizations would include first checking if the ray intersects with an object's bounding sphere or bounding box, the sphere or box that contains the entire object. If the ray doesn't intersect one of those then we don't have to check the triangles of that object.

THREE.js provides a `RayCaster` class that does exactly this.

Let's make a scene with a 100 objects and try picking them. We'll start with an example from the article on responsive pages.

Setting Up the Scene with a Camera Pole

We'll parent the camera to another object so we can spin that other object and the camera will move around the scene just like a selfie stick.

const fov = 60;
const aspect = 2;  // the canvas default
const near = 0.1;
const far = 200;
const camera = new THREE.PerspectiveCamera(fov, aspect, near, far);
camera.position.z = 30;

const scene = new THREE.Scene();
scene.background = new THREE.Color('white');

// put the camera on a pole (parent it to an object)
// so we can spin the pole to move the camera around the scene
const cameraPole = new THREE.Object3D();
scene.add(cameraPole);
cameraPole.add(camera);

Spinning the Camera Pole

In the `render` function we'll spin the camera pole.

Moving the Light with the Camera

Also let's put the light on the camera so the light moves with it.

Generating 100 Cubes

Let's generate 100 cubes with random colors in random positions, orientations, and scales.

const boxWidth = 1;
const boxHeight = 1;
const boxDepth = 1;
const geometry = new THREE.BoxGeometry(boxWidth, boxHeight, boxDepth);

function rand(min, max) {
  if (max === undefined) {
    max = min;
    min = 0;
  }
  return min + (max - min) * Math.random();
}

function randomColor() {
  return `hsl(${rand(360) | 0}, ${rand(50, 100) | 0}%, 50%)`;
}

const numObjects = 100;
for (let i = 0; i < numObjects; ++i) {
  const material = new THREE.MeshPhongMaterial({
    color: randomColor(),
  });

  const cube = new THREE.Mesh(geometry, material);
  scene.add(cube);

  cube.position.set(rand(-20, 20), rand(-20, 20), rand(-20, 20));
  cube.rotation.set(rand(Math.PI), rand(Math.PI), 0);
  cube.scale.set(rand(3, 6), rand(3, 6), rand(3, 6));
}

Creating the PickHelper Class

Let's make a simple class to manage the picking. You can see we create a `RayCaster` and then we can call the `pick` function to cast a ray through the scene. If the ray hits something we change the color of the first thing it hits.

class PickHelper {
  constructor() {
    this.raycaster = new THREE.Raycaster();
    this.pickedObject = null;
    this.pickedObjectSavedColor = 0;
  }
  pick(normalizedPosition, scene, camera, time) {
    // restore the color if there is a picked object
    if (this.pickedObject) {
      this.pickedObject.material.emissive.setHex(this.pickedObjectSavedColor);
      this.pickedObject = undefined;
    }

    // cast a ray through the frustum
    this.raycaster.setFromCamera(normalizedPosition, camera);
    // get the list of objects the ray intersected
    const intersectedObjects = this.raycaster.intersectObjects(scene.children);
    if (intersectedObjects.length) {
      // pick the first object. It's the closest one
      this.pickedObject = intersectedObjects[0].object;
      // save its color
      this.pickedObjectSavedColor = this.pickedObject.material.emissive.getHex();
      // set its emissive color to flashing red/yellow
      this.pickedObject.material.emissive.setHex((time * 8) % 2 > 1 ? 0xFFFF00 : 0xFF0000);
    }
  }
}

Tracking Mouse Position

Of course we could call this function only when the user pressed the mouse *down* which is probably usually what you want but for this example we'll pick every frame whatever is under the mouse. To do this we first need to track where the mouse is.

Notice we're recording a normalized mouse position. Regardless of the size of the canvas we need a value that goes from -1 on the left to +1 on the right. Similarly we need a value that goes from -1 on the bottom to +1 on the top.

const pickPosition = {x: 0, y: 0};
clearPickPosition();

...

function getCanvasRelativePosition(event) {
  const rect = canvas.getBoundingClientRect();
  return {
    x: (event.clientX - rect.left) * canvas.width  / rect.width,
    y: (event.clientY - rect.top ) * canvas.height / rect.height,
  };
}

function setPickPosition(event) {
  const pos = getCanvasRelativePosition(event);
  pickPosition.x = (pos.x / canvas.width ) *  2 - 1;
  pickPosition.y = (pos.y / canvas.height) * -2 + 1;  // note we flip Y
}

function clearPickPosition() {
  // unlike the mouse which always has a position
  // if the user stops touching the screen we want
  // to stop picking. For now we just pick a value
  // unlikely to pick something
  pickPosition.x = -100000;
  pickPosition.y = -100000;
}

window.addEventListener('mousemove', setPickPosition);
window.addEventListener('mouseout', clearPickPosition);
window.addEventListener('mouseleave', clearPickPosition);

Adding Mobile Touch Support

While we're at it lets support mobile as well.

window.addEventListener('touchstart', (event) => {
  // prevent the window from scrolling
  event.preventDefault();
  setPickPosition(event.touches[0]);
}, {passive: false});

window.addEventListener('touchmove', (event) => {
  setPickPosition(event.touches[0]);
});

window.addEventListener('touchend', clearPickPosition);

Calling PickHelper in the Render Function

And finally in our `render` function we call the `PickHelper`'s `pick` function.

const pickHelper = new PickHelper();

function render(time) {
  time *= 0.001;  // convert to seconds;

  ...

  pickHelper.pick(pickPosition, scene, camera, time);

  renderer.render(scene, camera);

  ...
}

Issues with Raycasting

This appears to work great and it probably does for many use cases but there are several issues.

It's CPU based.

JavaScript is going through each object and checking if the ray intersects that object's bounding box or bounding sphere. If it does then JavaScript has to go through each and every triangle in that object and check if the ray intersects the triangle.

The good part of this is JavaScript can easily compute exactly where the ray intersected the triangle and provide us with that data. For example if you wanted to put a marker where the intersection happened.

The bad part is that's a lot of work for the CPU to do. If you have objects with lots of triangles it might be slow.

It doesn't handle any strange shaders or displacements.

If you have a shader that deforms or morphs the geometry JavaScript has no knowledge of that deformation and so will give the wrong answer. For example AFAIK you can't use this method with skinned objects.

It doesn't handle transparent holes.

As an example let's apply this texture to the cubes.

Demonstrating the Transparency Problem

We'll just make these changes.

const loader = new THREE.TextureLoader();
const texture = loader.load('resources/images/frame.png');

const numObjects = 100;
for (let i = 0; i < numObjects; ++i) {
  const material = new THREE.MeshPhongMaterial({
    color: randomColor(),
    map: texture,
    transparent: true,
    side: THREE.DoubleSide,
    alphaTest: 0.1,
  });

  const cube = new THREE.Mesh(geometry, material);
  scene.add(cube);

  ...
}

Why Transparent Picking Fails

Try to pick something through a box and you can't. This is because JavaScript can't easily look into the textures and materials and figure out if part of your object is really transparent or not.

Introducing GPU-Based Picking

A solution all of these issues is to use GPU based picking. Unfortunately while it is conceptually simple it is more complicated to use than the ray casting method above.

To do GPU picking we render each object in a unique color offscreen. We then look up the color of the pixel corresponding to the mouse position. The color tells us which object was picked.

This can solve issue 2 and 3 above. As for issue 1, speed, it really depends. Every object has to be drawn twice. Once to draw it for viewing and again to draw it for picking. It's possible with fancier solutions maybe both of those could be done at the same time but we're not going to try that.

One thing we can do though is since we're only going to be reading one pixel we can just setup the camera so only that one pixel is drawn. We can do this using `PerspectiveCamera.setViewOffset` which lets us tell THREE.js to compute a camera that just renders a smaller part of a larger rectangle. This should save some time.

To do this type of picking in THREE.js at the moment requires we create 2 scenes. One we will fill with our normal meshes. The other we'll fill with meshes that use our picking material.

Creating the Picking Scene

So, first create a second scene and make sure it clears to black.

const scene = new THREE.Scene();
scene.background = new THREE.Color('white');
const pickingScene = new THREE.Scene();
pickingScene.background = new THREE.Color(0);

Creating Picking Cubes for Each Object

Then, for each cube we place in the main scene we make a corresponding "picking cube" at the same position as the original cube, put it in the `pickingScene`, and set its material to something that will draw the object's id as its color. Also we keep a map of ids to objects so when we look up an id later we can map it back to its corresponding object.

Note that we are abusing the `MeshPhongMaterial` here. By setting its `emissive` to our id and the `color` and `specular` to 0 that will end up rendering the id only where the texture's alpha is greater than `alphaTest`. We also need to set `blending` to `NoBlending` so that the id is not multiplied by alpha.

Note that abusing the `MeshPhongMaterial` might not be the best solution as it will still calculate all our lights when drawing the picking scene even though we don't need those calculations. A more optimized solution would make a custom shader that just writes the id where the texture's alpha is greater than `alphaTest`.

const idToObject = {};
const numObjects = 100;
for (let i = 0; i < numObjects; ++i) {
  const id = i + 1;
  const material = new THREE.MeshPhongMaterial({
    color: randomColor(),
    map: texture,
    transparent: true,
    side: THREE.DoubleSide,
    alphaTest: 0.1,
  });

  const cube = new THREE.Mesh(geometry, material);
  scene.add(cube);
  idToObject[id] = cube;

  cube.position.set(rand(-20, 20), rand(-20, 20), rand(-20, 20));
  cube.rotation.set(rand(Math.PI), rand(Math.PI), 0);
  cube.scale.set(rand(3, 6), rand(3, 6), rand(3, 6));

  const pickingMaterial = new THREE.MeshPhongMaterial({
    emissive: new THREE.Color().setHex(id, THREE.NoColorSpace),
    color: new THREE.Color(0, 0, 0),
    specular: new THREE.Color(0, 0, 0),
    map: texture,
    transparent: true,
    side: THREE.DoubleSide,
    alphaTest: 0.5,
    blending: THREE.NoBlending,
  });
  const pickingCube = new THREE.Mesh(geometry, pickingMaterial);
  pickingScene.add(pickingCube);
  pickingCube.position.copy(cube.position);
  pickingCube.rotation.copy(cube.rotation);
  pickingCube.scale.copy(cube.scale);
}

Updating Pick Position for Pixel-Based Picking

Because we're picking from pixels instead of ray casting we can change the code that sets the pick position to just use pixels.

function setPickPosition(event) {
  const pos = getCanvasRelativePosition(event);
  pickPosition.x = pos.x;
  pickPosition.y = pos.y;
}

Creating the GPUPickHelper Class

Then let's change the `PickHelper` into a `GPUPickHelper`. It will use a `WebGLRenderTarget` like we covered the article on render targets. Our render target here is only a single pixel in size, 1x1.

class GPUPickHelper {
  constructor() {
    // create a 1x1 pixel render target
    this.pickingTexture = new THREE.WebGLRenderTarget(1, 1);
    this.pixelBuffer = new Uint8Array(4);
    this.pickedObject = null;
    this.pickedObjectSavedColor = 0;
  }
  pick(cssPosition, scene, camera, time) {
    const {pickingTexture, pixelBuffer} = this;

    // restore the color if there is a picked object
    if (this.pickedObject) {
      this.pickedObject.material.emissive.setHex(this.pickedObjectSavedColor);
      this.pickedObject = undefined;
    }

    // set the view offset to represent just a single pixel under the mouse
    const pixelRatio = renderer.getPixelRatio();
    camera.setViewOffset(
        renderer.getContext().drawingBufferWidth,   // full width
        renderer.getContext().drawingBufferHeight,  // full top
        cssPosition.x * pixelRatio | 0,             // rect x
        cssPosition.y * pixelRatio | 0,             // rect y
        1,                                          // rect width
        1,                                          // rect height
    );
    // render the scene
    renderer.setRenderTarget(pickingTexture)
    renderer.render(scene, camera);
    renderer.setRenderTarget(null);

    // clear the view offset so rendering returns to normal
    camera.clearViewOffset();
    //read the pixel
    renderer.readRenderTargetPixels(
        pickingTexture,
        0,   // x
        0,   // y
        1,   // width
        1,   // height
        pixelBuffer);

    const id =
        (pixelBuffer[0] << 16) |
        (pixelBuffer[1] <<  8) |
        (pixelBuffer[2]      );

    const intersectedObject = idToObject[id];
    if (intersectedObject) {
      // pick the first object. It's the closest one
      this.pickedObject = intersectedObject;
      // save its color
      this.pickedObjectSavedColor = this.pickedObject.material.emissive.getHex();
      // set its emissive color to flashing red/yellow
      this.pickedObject.material.emissive.setHex((time * 8) % 2 > 1 ? 0xFFFF00 : 0xFF0000);
    }
  }
}

Using the GPUPickHelper

Then we just need to use it, and pass it the `pickScene` instead of the `scene`. And now it should let you pick through the transparent parts.

const pickHelper = new GPUPickHelper();

  pickHelper.pick(pickPosition, pickScene, camera, time);

Conclusion

I hope that gives some idea of how to implement picking. In a future article maybe we can cover how to manipulate objects with the mouse.