Post Processing

Summary: An introduction to post processing in THREE.js, covering the EffectComposer pipeline, built-in passes such as RenderPass, BloomPass, FilmPass, and OutputPass, runtime parameter adjustment via uniforms, and creating custom shader passes with ShaderPass.

Overview

Post processing generally refers to applying some kind of effect or filter to a 2D image. In the case of THREE.js we have a scene with a bunch of meshes in it. We render that scene into a 2D image. Normally that image is rendered directly into the canvas and displayed in the browser but instead we can render it to a render target and then apply some post processing effects to the result before drawing it to the canvas. It's called post processing because it happens after (post) the main scene processing.

Examples of post processing are Instagram like filters, Photoshop filters, etc...

THREE.js has some example classes to help setup a post processing pipeline. The way it works is you create an EffectComposer and to it you add multiple Pass objects. You then call EffectComposer.render and it renders your scene to a render target and then applies each Pass.

Each Pass can be some post processing effect like adding a vignette, blurring, applying a bloom, applying film grain, adjusting the hue, saturation, contrast, etc... and finally rendering the result to the canvas.

It's a little bit important to understand how EffectComposer functions. It creates two render targets. Let's call them rtA and rtB.

Then, you call EffectComposer.addPass to add each pass in the order you want to apply them. The passes are then applied something like this. First the scene you passed into RenderPass is rendered to rtA, then rtA is passed to the next pass, whatever it is. That pass uses rtA as input to do whatever it does and writes the results to rtB. rtB is then passed to the next pass which uses rtB as input and writes back to rtA. This continues through all the passes.

Each Pass has 4 basic options

enabled

Whether or not to use this pass

needsSwap

Whether or not to swap rtA and rtB after finishing this pass

clear

Whether or not to clear before rendering this pass

renderToScreen

Whether or not to render to the canvas instead the current destination render target. In most use cases you do not set this flag explicitly since the last pass in the pass chain is automatically rendered to screen.

Basic Example

Let's put together a basic example. We'll start with the example from the article on responsiveness.

To that first we create an EffectComposer.

Then as the first pass we add a RenderPass that will render our scene with our camera into the first render target.

Next we add a BloomPass. A BloomPass renders its input to a generally smaller render target and blurs the result. It then adds that blurred result on top of the original input. This makes the scene bloom.

Next we had a FilmPass that draws noise and scanlines on top of its input.

Finally we had a OutputPass which performs color space conversion to sRGB and optional tone mapping. This pass is usually the last pass of the pass chain.

To use these classes we need to import a bunch of scripts.

For pretty much any post processing EffectComposer.js, RenderPass.js and OutputPass.js are required.

The last things we need to do are to use EffectComposer.render instead of WebGLRenderer.render and to tell the EffectComposer to match the size of the canvas.

EffectComposer.render takes a deltaTime which is the time in seconds since the last frame was rendered. It passes this to the various effects in case any of them are animated. In this case the FilmPass is animated.

const composer = new EffectComposer(renderer);

const bloomPass = new BloomPass(
    1,    // strength
    25,   // kernel size
    4,    // sigma ?
    256,  // blur render target resolution
);
composer.addPass(bloomPass);

const filmPass = new FilmPass(
    0.5,   // intensity
    false,  // grayscale
);
composer.addPass(filmPass);

const outputPass = new OutputPass();
composer.addPass(outputPass);

import {EffectComposer} from 'three/addons/postprocessing/EffectComposer.js';
import {RenderPass} from 'three/addons/postprocessing/RenderPass.js';
import {BloomPass} from 'three/addons/postprocessing/BloomPass.js';
import {FilmPass} from 'three/addons/postprocessing/FilmPass.js';
import {OutputPass} from 'three/addons/postprocessing/OutputPass.js';

-function render(now) {
-  time *= 0.001;
+let then = 0;
+function render(now) {
+  now *= 0.001;  // convert to seconds
+  const deltaTime = now - then;
+  then = now;

  if (resizeRendererToDisplaySize(renderer)) {
    const canvas = renderer.domElement;
    camera.aspect = canvas.clientWidth / canvas.clientHeight;
    camera.updateProjectionMatrix();
+    composer.setSize(canvas.width, canvas.height);
  }

  cubes.forEach((cube, ndx) => {
    const speed = 1 + ndx * .1;
-    const rot = time * speed;
+    const rot = now * speed;
    cube.rotation.x = rot;
    cube.rotation.y = rot;
  });

-  renderer.render(scene, camera);
+  composer.render(deltaTime);

  requestAnimationFrame(render);
}

Runtime Parameter Adjustment

To change effect parameters at runtime usually requires setting uniform values. Let's add a gui to adjust some of the parameters. Figuring out which values you can easily adjust and how to adjust them requires digging through the code for that effect.

Looking inside BloomPass.js I found this line:

So we can set the strength by setting

Similarly looking in FilmPass.js I found these lines:

So which makes it pretty clear how to set them.

Let's make a quick GUI to set those values

and

and now we can adjust those settings

this.combineUniforms[ 'strength' ].value = strength;

this.uniforms.intensity.value = intensity;
this.uniforms.grayscale.value = grayscale;

import {GUI} from 'three/addons/libs/lil-gui.module.min.js';

const gui = new GUI();
{
  const folder = gui.addFolder('BloomPass');
  folder.add(bloomPass.combineUniforms.strength, 'value', 0, 2).name('strength');
  folder.open();
}
{
  const folder = gui.addFolder('FilmPass');
  folder.add(filmPass.uniforms.grayscale, 'value').name('grayscale');
  folder.add(filmPass.uniforms.intensity, 'value', 0, 1).name('intensity');
  folder.open();
}

Custom Shaders

That was a small step to making our own effect.

Post processing effects use shaders. Shaders are written in a language called GLSL (Graphics Library Shading Language). Going over the entire language is way too large a topic for these articles. A few resources to get start from would be maybe this article and maybe the Book of Shaders.

I think an example to get you started would be helpful though so let's make a simple GLSL post processing shader. We'll make one that lets us multiply the image by a color.

For post processing THREE.js provides a useful helper called the ShaderPass. It takes an object with info defining a vertex shader, a fragment shader, and the default inputs. It will handling setting up which texture to read from to get the previous pass's results and where to render to, either one of the EffectComposer's render target or the canvas.

Here's a simple post processing shader that multiplies the previous pass's result by a color.

Above tDiffuse is the name that ShaderPass uses to pass in the previous pass's result texture so we pretty much always need that. We then declare color as a THREE.js Color.

Next we need a vertex shader. For post processing the vertex shader shown here is pretty much standard and rarely needs to be changed. Without going into too many details (see articles linked above) the variables uv, projectionMatrix, modelViewMatrix and position are all magically added by THREE.js.

Finally we create a fragment shader. In it we get a pixel color from the previous pass with this line

we multiply it by our color and set gl_FragColor to the result

Adding some simple GUI to set the 3 values of the color

Gives us a simple postprocessing effect that multiplies by a color.

const colorShader = {
  uniforms: {
    tDiffuse: { value: null },
    color:    { value: new THREE.Color(0x88CCFF) },
  },
  vertexShader:
    varying vec2 vUv;
    void main() {
      vUv = uv;
      gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1);
    }
  ,
  fragmentShader:
    varying vec2 vUv;
    uniform sampler2D tDiffuse;
    uniform vec3 color;
    void main() {
      vec4 previousPassColor = texture2D(tDiffuse, vUv);
      gl_FragColor = vec4(
          previousPassColor.rgb * color,
          previousPassColor.a);
    }
  ,
};

const gui = new GUI();
gui.add(colorPass.uniforms.color.value, 'r', 0, 4).name('red');
gui.add(colorPass.uniforms.color.value, 'g', 0, 4).name('green');
gui.add(colorPass.uniforms.color.value, 'b', 0, 4).name('blue');

Additional Resources

As mentioned about all the details of how to write GLSL and custom shaders is too much for these articles. If you really want to know how WebGL itself works then check out these articles. Another great resources is just to read through the existing post processing shaders in the THREE.js repo. Some are more complicated than others but if you start with the smaller ones you can hopefully get an idea of how they work.

Most of the post processing effects in the THREE.js repo are unfortunately undocumented so to use them you'll have to read through the examples or the code for the effects themselves. Hopefully these simple example and the article on render targets provide enough context to get started.


Three.js and Shadertoy

Summary: A guide explaining how to integrate Shadertoy shaders into Three.js, covering shader setup, uniform handling, texture inputs, vertex shaders, and procedural textures.

Three.js and Shadertoy

Shadertoy is a famous website hosting amazing shader experiments. People often ask how they can use those shaders with Three.js.

It's important to recognize it's called Shader TOY for a reason. In general shadertoy shaders are not about best practices. Rather they are a fun challenge similar to say dwitter (write code in 140 characters) or js13kGames (make a game in 13k or less).

In the case of Shadertoy the puzzle is, write a function that for a given pixel location outputs a color that draws something interesting. It's a fun challenge and many of the result are amazing. But, it is not best practice.

Compare this amazing shadertoy shader that draws an entire city

Fullscreen on my GPU it runs at about 5 frames a second. Contrast that to a game like Cities: Skylines

This game runs 30-60 frames a second on the same machine because it uses more traditional techniques, drawing buildings made from triangles with textures on them, etc...

Still, let's go over using a Shadertoy shader with three.js.

This is the default shadertoy shader if you pick "New" on shadertoy.com, at least as of January 2019.

One thing important to understand about shaders is they are written in a language called GLSL (Graphics Library Shading Language) designed for 3D math which includes special types. Above we see vec4, vec2, vec3 as 3 such special types. A vec2 has 2 values, a vec3 3, a vec4 4 values. They can be addressed in a bunch of ways. The most common ways are with x, y, z, and w as in

Unlike JavaScript, GLSL is more like C/C++ where variables have to have their type declared so instead of var v = 1.2; it's float v = 1.2; declaring v to be a floating point number.

Explaining GLSL in detail is more than we can do in this article. For a quick overview see this article and maybe follow that up with this series.

It should be noted that, at least as of January 2019, shadertoy.com only concerns itself with fragment shaders. A fragment shader's responsibility is, given a pixel location output a color for that pixel.

Looking at the function above we can see the shader has an out parameter called fragColor. out stands for output. It's a parameter the function is expected to provide a value for. We need to set this to some color.

It also has an in (for input) parameter called fragCoord. This is the pixel coordinate that is about to be drawn. We can use that coordinate to decide on a color. If the canvas we're drawing to is 400x300 pixels then the function will be called 400x300 times or 120,000 times. Each time fragCoord will be a different pixel coordinate.

There are 2 more variables being used that are not defined in the code. One is iResolution. This is set to the resolution of the canvas. If the canvas is 400x300 then iResolution would be 400,300 so as the pixel coordinates change that makes uv go from 0.0 to 1.0 across and up the texture. Working with normalized values often makes things easier and so the majority of shadertoy shaders start with something like this.

The other undefined variable in the shader is iTime. This is the time since the page loaded in seconds.

In shader jargon these global variables are called uniform variables. They are called uniform because they don't change, they stay uniform from one iteration of the shader to the next. It's important to note all of them are specific to shadertoy. They not official GLSL variables. They are variables the makers of shadertoy made up.

The Shadertoy docs define several more. For now let's write something that handles the two being used in the shader above.

The first thing to do is let's make a single plane that fills the canvas. If you haven't read it yet we did this in the article on backgrounds so let's grab that example but remove the cubes. It's pretty short so here's the entire thing

As explained in the backgrounds article an OrthographicCamera with these parameters and a 2 unit plane will fill the canvas. For now all we'll get is a red canvas as our plane is using a red MeshBasicMaterial.

Now that we have something working let's add the shadertoy shader.

Above we declared the 2 uniform variables we talked about. Then we inserted the shader GLSL code from shadertoy. Finally we called mainImage passing it gl_FragColor and gl_FragCoord.xy. gl_FragColor is an official WebGL global variable the shader is responsible for setting to whatever color it wants the current pixel to be. gl_FragCoord is another official WebGL global variable that tells us the coordinate of the pixel we're currently choosing a color for.

We then need to setup three.js uniforms so we can supply values to the shader.

Each uniform in THREE.js has value parameter. That value has to match the type of the uniform.

Then we pass both the fragment shader and uniforms to a ShaderMaterial.

and before rendering we need to set the values of the uniforms

Note: I have no idea why iResolution is a vec3 and what's in the 3rd value is not documented on shadertoy.com. It's not used above so just setting it to 1 for now. ¯\_(ツ)_/¯

This matches what we see on Shadertoy for a new shader, at least as of January 2019 😉. What's the shader above doing?

- uv goes from 0 to 1.
- cos(uv.xyx) gives us 3 cosine values as a vec3. One for uv.x, another for uv.y and another for uv.x again.
- Adding in the time, cos(iTime+uv.xyx) makes them animate.
- Adding in vec3(0,2,4) as in cos(iTime+uv.xyx+vec3(0,2,4)) offsets the cosine waves
- cos goes from -1 to 1 so the 0.5 * 0.5 + cos(...) converts from -1 <-> 1 to 0.0 <-> 1.0
- the results are then used as the RGB color for the current pixel

A minor change will make it easier to see the cosine waves. Right now uv only goes from 0 to 1. A cosine repeats at 2π so let's make it go from 0 to 40 by multiplying by 40.0. That should make it repeat about 6.3 times.

Counting below I see about 6.3 repeats. We can see the blue between the red since it's offset by 4 via the +vec3(0,2,4). Without that the blue and red would overlap perfectly making purple.

Knowing how simple the inputs are and then seeing results like a city canal, a forest, a snail, a mushroom make the challenge all that much more impressive. Hopefully they also make it clear why it's not generally the right approach vs the more traditional ways of making scenes from triangles. The fact that so much math has to be put into computing the color of every pixel means those examples run very slow.

Some shadertoy shaders take textures as inputs like this one.

Passing a texture into a shader is similar to passing one into a normal material but we need to set up the texture on the uniforms.

First we'll add the uniform for the texture to the shader. They're referred to as sampler2D in GLSL.

Then we can load a texture like we covered here and assign the uniform's value.

So far we've been using Shadertoy shaders as they are used on Shadertoy.com, namely drawing to cover the canvas. There's no reason we need to limit it to just that use case though. The important part to remember is the functions people write on shadertoy generally just take a fragCoord input and a iResolution. fragCoord does not have to come from pixel coordinates, we could use something else like texture coordinates instead and could then use them kind of like other textures. This technique of using a function to generate textures is often called a procedural texture.

Let's change the shader above to do this. The simplest thing to do might be to take the texture coordinates that three.js normally supplies, multiply them by iResolution and pass that in for fragCoords.

To do that we add in a varying. A varying is a value passed from the vertex shader to the fragment shader that gets interpolated (or varied) between vertices. To use it in our fragment shader we declare it. Three.js refers to its texture coordinates as uv with the v in front meaning varying.

Then we need to also provide our own vertex shader. Here is a fairly common minimal three.js vertex shader. Three.js declares and will provide values for uv, projectionMatrix, modelViewMatrix, and position.

We need to pass the vertex shader to the ShaderMaterial

We can set the iResolution uniform value at init time since it will no longer change.

and we no longer need to set it at render time

Otherwise I copied back in the original camera and code that sets up 3 rotating cubes from the article on responsiveness. The result:

I hope this at least gets you started on how to use a shadertoy shader with three.js. Again, it's important to remember that most shadertoy shaders are an interesting challenge (draw everything with a single function) rather than the recommended way to actually display things in a performant way. Still, they are amazing, impressive, beautiful, and you can learn a ton by seeing how they work.

function main() {
  const canvas = document.querySelector('#c');
  const renderer = new THREE.WebGLRenderer({antialias: true, canvas});
  renderer.autoClearColor = false;

  const camera = new THREE.OrthographicCamera(
    -1, // left
     1, // right
     1, // top
    -1, // bottom
    -1, // near,
     1, // far,
  );
  const scene = new THREE.Scene();
  const plane = new THREE.PlaneGeometry(2, 2);
  const material = new THREE.MeshBasicMaterial({
      color: 'red',
  });
  scene.add(new THREE.Mesh(plane, material));

  function resizeRendererToDisplaySize(renderer) {
    const canvas = renderer.domElement;
    const width = canvas.clientWidth;
    const height = canvas.clientHeight;
    const needResize = canvas.width !== width || canvas.height !== height;
    if (needResize) {
      renderer.setSize(width, height, false);
    }
    return needResize;
  }

  function render() {
    resizeRendererToDisplaySize(renderer);

    renderer.render(scene, camera);

    requestAnimationFrame(render);
  }

  requestAnimationFrame(render);
}

main();

const fragmentShader =
#include <common>

uniform vec3 iResolution;
uniform float iTime;

// By iq: https://www.shadertoy.com/user/iq
// license: Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.
void mainImage( out vec4 fragColor, in vec2 fragCoord )
{
    // Normalized pixel coordinates (from 0 to 1)
    vec2 uv = fragCoord/iResolution.xy;

    // Time varying pixel color
    vec3 col = 0.5 + 0.5*cos(iTime+uv.xyx+vec3(0,2,4));

    // Output to screen
    fragColor = vec4(col,1.0);
}

void main() {
  mainImage(gl_FragColor, gl_FragCoord.xy);
}
;

const uniforms = {
  iTime: { value: 0 },
  iResolution:  { value: new THREE.Vector3() },
};

-const material = new THREE.MeshBasicMaterial({
-    color: 'red',
-});
+const material = new THREE.ShaderMaterial({
+  fragmentShader,
+  uniforms,
+});

-function render() {
+function render(time) {
+  time *= 0.001;  // convert to seconds

  resizeRendererToDisplaySize(renderer);

+  const canvas = renderer.domElement;
+  uniforms.iResolution.value.set(canvas.width, canvas.height, 1);
+  uniforms.iTime.value = time;

  renderer.render(scene, camera);

  requestAnimationFrame(render);
}

// By Daedelus: https://www.shadertoy.com/user/Daedelus
// license: Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.
#define TIMESCALE 0.25
#define TILES 8
#define COLOR 0.7, 1.6, 2.8

void mainImage( out vec4 fragColor, in vec2 fragCoord )
{
    vec2 uv = fragCoord.xy / iResolution.xy;
    uv.x *= iResolution.x / iResolution.y;

    vec4 noise = texture2D(iChannel0, floor(uv * float(TILES)) / float(TILES));
    float p = 1.0 - mod(noise.r + noise.g + noise.b + iTime * float(TIMESCALE), 1.0);
    p = min(max(p * 3.0 - 1.8, 0.1), 2.0);

    vec2 r = mod(uv * float(TILES), 1.0);
    r = vec2(pow(r.x - 0.5, 2.0), pow(r.y - 0.5, 2.0));
    p *= 1.0 - pow(min(1.0, 12.0 * dot(r, r)), 2.0);

    fragColor = vec4(COLOR, 1.0) * p;
}

const fragmentShader =
#include <common>

uniform vec3 iResolution;
uniform float iTime;
+uniform sampler2D iChannel0;

...

+const loader = new THREE.TextureLoader();
+const texture = loader.load('resources/images/bayer.png');
+texture.minFilter = THREE.NearestFilter;
+texture.magFilter = THREE.NearestFilter;
+texture.wrapS = THREE.RepeatWrapping;
+texture.wrapT = THREE.RepeatWrapping;
const uniforms = {
  iTime: { value: 0 },
  iResolution:  { value: new THREE.Vector3() },
+  iChannel0: { value: texture },
};

+varying vec2 vUv;

void main() {
-  mainImage(gl_FragColor, gl_FragCoord.xy);
+  mainImage(gl_FragColor, vUv * iResolution.xy);
}

const vertexShader =
  varying vec2 vUv;
  void main() {
    vUv = uv;
    gl_Position = projectionMatrix * modelViewMatrix * vec4( position, 1.0 );
  }
`;

const material = new THREE.ShaderMaterial({
  vertexShader,
  fragmentShader,
  uniforms,
});

const uniforms = {
  iTime: { value: 0 },
-  iResolution:  { value: new THREE.Vector3() },
+  iResolution:  { value: new THREE.Vector3(1, 1, 1) },
  iChannel0: { value: texture },
};

-const canvas = renderer.domElement;
-uniforms.iResolution.value.set(canvas.width, canvas.height, 1);
uniforms.iTime.value = time;


Aligning HTML Elements to 3D

Summary: This guide explains how to align HTML elements to 3D objects in a three.js scene, covering basic positioning, handling overlapping labels, frustum checks for off-screen objects, zIndex sorting for correct stacking order, and advanced techniques like dot-product visibility checks for a globe example.

Introduction

This article is part of a series of articles about three.js. The first article is three.js fundamentals. If you haven't read that yet and you're new to three.js you might want to consider starting there.

Sometimes you'd like to display some text in your 3D scene. You have many options each with pluses and minuses.

- Use 3D text

If you look at the primitives article you'll see TextGeometry which makes 3D text. This might be useful for flying logos but probably not so useful for stats, info, or labelling lots of things.

- Use a texture with 2D text drawn into it.

The article on using a Canvas as a texture shows using a canvas as a texture. You can draw text into a canvas and display it as a billboard. The plus here might be that the text is integrated into the 3D scene. For something like a computer terminal shown in a 3D scene this might be perfect.

- Use HTML Elements and position them to match the 3D

The benefits to this approach is you can use all of HTML. Your HTML can have multiple elements. It can by styled with CSS. It can also be selected by the user as it is actual text.

This article will cover this last approach.

Basic Setup

Let's start simple. We'll make a 3D scene with a few primitives and then add a label to each primitive. We'll start with an example from the article on responsive pages.

We'll add some OrbitControls like we did in the article on lighting.

We need to provide an HTML element to contain our label elements.

By putting both the canvas and the `<div id="labels">` inside a parent container we can make them overlap with this CSS.

let's also add some CSS for the labels themselves.

Now into our code we don't have to add too much. We had a function `makeInstance` that we used to generate cubes. Let's make it so it also adds a label element.

As you can see we're adding a `<div>` to the container, one for each cube. We're also returning an object with both the `cube` and the `elem` for the label.

Calling it we need to provide a name for each.

What remains is positioning the label elements at render time.

And with that we have labels aligned to their corresponding objects.

const controls = new OrbitControls(camera, canvas);
controls.target.set(0, 0, 0);
controls.update();

#c {
    width: 100%;  /* let our container decide our size */
    height: 100%;
    display: block;
}
#container {
  position: relative;  /* makes this the origin of its children */
  width: 100%;
  height: 100%;
  overflow: hidden;
}
#labels {
  position: absolute;  /* let us position ourself inside the container */
  left: 0;             /* make our position the top left of the container */
  top: 0;
  color: white;
}

#labels>div {
  position: absolute;  /* let us position them inside the container */
  left: 0;             /* make their default position the top left of the container */
  top: 0;
  cursor: pointer;     /* change the cursor to a hand when over us */
  font-size: large;
  user-select: none;   /* don't let the text get selected */
  text-shadow:         /* create a black outline */
    -1px -1px 0 #000,
     0   -1px 0 #000,
     1px -1px 0 #000,
     1px  0   0 #000,
     1px  1px 0 #000,
     0    1px 0 #000,
    -1px  1px 0 #000,
    -1px  0   0 #000;
}
#labels>div:hover {
  color: red;
}

const labelContainerElem = document.querySelector('#labels');

function makeInstance(geometry, color, x, name) {
  const material = new THREE.MeshPhongMaterial({color});

  const cube = new THREE.Mesh(geometry, material);
  scene.add(cube);

  cube.position.x = x;

  const elem = document.createElement('div');
  elem.textContent = name;
  labelContainerElem.appendChild(elem);

  return {cube, elem};
}

const cubes = [
  makeInstance(geometry, 0x44aa88,  0, 'Aqua'),
  makeInstance(geometry, 0x8844aa, -2, 'Purple'),
  makeInstance(geometry, 0xaa8844,  2, 'Gold'),
];

const tempV = new THREE.Vector3();

...

cubes.forEach((cubeInfo, ndx) => {
  const {cube, elem} = cubeInfo;
  const speed = 1 + ndx * .1;
  const rot = time * speed;
  cube.rotation.x = rot;
  cube.rotation.y = rot;

  // get the position of the center of the cube
  cube.updateWorldMatrix(true, false);
  cube.getWorldPosition(tempV);

  // get the normalized screen coordinate of that position
  // x and y will be in the -1 to +1 range with x = -1 being
  // on the left and y = -1 being on the bottom
  tempV.project(camera);

  // convert the normalized position to CSS coordinates
  const x = (tempV.x *  .5 + .5) * canvas.clientWidth;
  const y = (tempV.y * -.5 + .5) * canvas.clientHeight;

  // move the elem to that position
  elem.style.transform = translate(-50%, -50%) translate(${x}px,${y}px);
});

Handling Overlapping Labels

There are a couple of issues we probably want to deal with.

One is that if we rotate the objects so they overlap all the labels overlap as well.

Another is that if we zoom way out so that the objects go outside the frustum the labels will still appear.

A possible solution to the problem of overlapping objects is to use the picking code from the article on picking. We'll pass in the position of the object on the screen and then ask the RayCaster to tell us which objects were intersected. If our object is not the first one then we are not in the front.

This handles overlapping.

const tempV = new THREE.Vector3();
const raycaster = new THREE.Raycaster();

...

cubes.forEach((cubeInfo, ndx) => {
  const {cube, elem} = cubeInfo;
  const speed = 1 + ndx * .1;
  const rot = time * speed;
  cube.rotation.x = rot;
  cube.rotation.y = rot;

  // get the position of the center of the cube
  cube.updateWorldMatrix(true, false);
  cube.getWorldPosition(tempV);

  // get the normalized screen coordinate of that position
  // x and y will be in the -1 to +1 range with x = -1 being
  // on the left and y = -1 being on the bottom
  tempV.project(camera);

  // ask the raycaster for all the objects that intersect
  // from the eye toward this object's position
  raycaster.setFromCamera(tempV, camera);
  const intersectedObjects = raycaster.intersectObjects(scene.children);
  // We're visible if the first intersection is this object.
  const show = intersectedObjects.length && cube === intersectedObjects[0].object;

  if (!show) {
    // hide the label
    elem.style.display = 'none';
  } else {
    // un-hide the label
    elem.style.display = '';

    // convert the normalized position to CSS coordinates
    const x = (tempV.x *  .5 + .5) * canvas.clientWidth;
    const y = (tempV.y * -.5 + .5) * canvas.clientHeight;

    // move the elem to that position
    elem.style.transform = translate(-50%, -50%) translate(${x}px,${y}px);
  }
});

Handling Labels Outside the Frustum

To handle going outside the frustum we can add this check if the origin of the object is outside the frustum by checking tempV.z.

This *kind of* works because the normalized coordinates we computed include a z value that goes from -1 when at the near part of our camera frustum to +1 when at the far part of our camera frustum.

For the frustum check, the solution above fails as we're only checking the origin of the object. For a large object. That origin might go outside the frustum but half of the object might still be in the frustum.

A more correct solution would be to check if the object itself is in the frustum or not. Unfortunate that check is slow. For 3 cubes it will not be a problem but for many objects it might be.

Three.js provides some functions to check if an object's bounding sphere is in a frustum.

Our current overlapping solution has similar issues. Picking is slow. We could use gpu based picking like we covered in the picking article but that is also not free. Which solution you chose depends on your needs.

if (!show || Math.abs(tempV.z) > 1) {
    // hide the label
    elem.style.display = 'none';

// at init time
const frustum = new THREE.Frustum();
const viewProjection = new THREE.Matrix4();

...

// before checking
camera.updateMatrix();
camera.updateMatrixWorld();
camera.matrixWorldInverse.copy(camera.matrixWorld).invert();

...

// then for each mesh
someMesh.updateMatrix();
someMesh.updateMatrixWorld();

viewProjection.multiplyMatrices(
    camera.projectionMatrix, camera.matrixWorldInverse);
frustum.setFromProjectionMatrix(viewProjection);
const inFrustum = frustum.contains(someMesh));

Sorting Labels with zIndex

Another issue is the order the labels appear. If we change the code to have longer labels and set the CSS so these don't wrap, then we can run into an issue where the purple box is in the back but its label is in front of the aqua box.

We can fix this by setting the zIndex of each element. The projected position has a z value that goes from -1 in front to positive 1 in back. zIndex is required to be an integer and goes the opposite direction meaning for zIndex greater values are in front so the following code should work.

Because of the way the projected z value works we need to pick a large number to spread out the values otherwise many will have the same value. To make sure the labels don't overlap with other parts of the page we can tell the browser to create a new stacking context by setting the z-index of the container of the labels.

And now the labels should always be in the correct order.

const cubes = [
  makeInstance(geometry, 0x44aa88,  0, 'Aqua Colored Box'),
  makeInstance(geometry, 0x8844aa, -2, 'Purple Colored Box'),
  makeInstance(geometry, 0xaa8844,  2, 'Gold Colored Box'),
];

#labels {
  position: absolute;  /* let us position ourself inside the container */
  z-index: 0;          /* make a new stacking context so children don't sort with rest of page */
  left: 0;             /* make our position the top left of the container */
  top: 0;
  color: white;
  z-index: 0;
}

Globe Example - Loading Country Data

While we're at it let's do one more example to show one more issue. Let's draw a globe like Google Maps and label the countries.

I found this data which contains the borders of countries. It's licensed as CC-BY-SA.

I wrote some code to load the data, and generate country outlines and some JSON data with the names of the countries and their locations.

The JSON data is an array of entries something like this where min, max, lat, lon, are all in latitude and longitude degrees.

Let's load it up. The code is based on the examples from optimizing lots of objects though we are not drawing lots of objects we'll be using the same solutions for rendering on demand.

The first thing is to make a sphere and use the outline texture.

Then let's load the JSON file by first making a loader and then calling it.

{
  const loader = new THREE.TextureLoader();
  const texture = loader.load('resources/data/world/country-outlines-4k.png', render);
  const geometry = new THREE.SphereGeometry(1, 64, 32);
  const material = new THREE.MeshBasicMaterial({map: texture});
  scene.add(new THREE.Mesh(geometry, material));
}

async function loadJSON(url) {
  const req = await fetch(url);
  return req.json();
}

let countryInfos;
async function loadCountryData() {
  countryInfos = await loadJSON('resources/data/world/country-info.json');
     ...
  }
  requestRenderIfNotRequested();
}
loadCountryData();

Globe Example - Placing Labels

Now let's use that data to generate and place the labels.

In the article on optimizing lots of objects we had setup a small scene graph of helper objects to make it easy to compute latitude and longitude positions on our globe. See that article for an explanation of how they work.

We'll use that to compute a position for each label.

The code above looks very similar to the code we wrote for making cube labels making an element per label. When we're done we have an array, countryInfos, with one entry for each country to which we've added an elem property for the label element for that country and a position with its position on the globe.

Just like we did for the cubes we need to update the position of the labels and render time.

You can see the code above is substantially similar to the cube example before. The only major difference is we pre-computed the label positions at init time. We can do this because the globe never moves. Only our camera moves.

Lastly we need to call updateLabels in our render loop.

And this is what we get. That is way too many labels!

We have 2 problems:
- Labels facing away from us are showing up.
- There are too many labels.

const lonFudge = Math.PI * 1.5;
const latFudge = Math.PI;
// these helpers will make it easy to position the boxes
// We can rotate the lon helper on its Y axis to the longitude
const lonHelper = new THREE.Object3D();
// We rotate the latHelper on its X axis to the latitude
const latHelper = new THREE.Object3D();
lonHelper.add(latHelper);
// The position helper moves the object to the edge of the sphere
const positionHelper = new THREE.Object3D();
positionHelper.position.z = 1;
latHelper.add(positionHelper);

const labelParentElem = document.querySelector('#labels');
for (const countryInfo of countryInfos) {
  const {lat, lon, name} = countryInfo;

  // adjust the helpers to point to the latitude and longitude
  lonHelper.rotation.y = THREE.MathUtils.degToRad(lon) + lonFudge;
  latHelper.rotation.x = THREE.MathUtils.degToRad(lat) + latFudge;

  // get the position of the lat/lon
  positionHelper.updateWorldMatrix(true, false);
  const position = new THREE.Vector3();
  positionHelper.getWorldPosition(position);
  countryInfo.position = position;

  // add an element for each country
  const elem = document.createElement('div');
  elem.textContent = name;
  labelParentElem.appendChild(elem);
  countryInfo.elem = elem;

const tempV = new THREE.Vector3();

function updateLabels() {
  // exit if we have not yet loaded the JSON file
  if (!countryInfos) {
    return;
  }

  for (const countryInfo of countryInfos) {
    const {position, elem} = countryInfo;

    // get the normalized screen coordinate of that position
    // x and y will be in the -1 to +1 range with x = -1 being
    // on the left and y = -1 being on the bottom
    tempV.copy(position);
    tempV.project(camera);

    // convert the normalized position to CSS coordinates
    const x = (tempV.x *  .5 + .5) * canvas.clientWidth;
    const y = (tempV.y * -.5 + .5) * canvas.clientHeight;

    // move the elem to that position
    elem.style.transform = translate(-50%, -50%) translate(${x}px,${y}px);

    // set the zIndex for sorting
    elem.style.zIndex = (-tempV.z * .5 + .5) * 100000 | 0;
  }
}

function render() {
  renderRequested = false;

  if (resizeRendererToDisplaySize(renderer)) {
    const canvas = renderer.domElement;
    camera.aspect = canvas.clientWidth / canvas.clientHeight;
    camera.updateProjectionMatrix();
  }

  controls.update();

  updateLabels();

  renderer.render(scene, camera);
}

Hiding Back-Facing Labels

For issue #1 we can't really use the RayCaster like we did above as there is nothing to intersect except the sphere. Instead what we can do is check if that particular country is facing away from us or not. This works because the label positions are around a sphere. In fact we're using a unit sphere, a sphere with a radius of 1.0. That means the positions are already unit directions making the math relatively easy.

Above we use the positions as a direction and get that direction relative to the camera. Then we get the camera relative direction from the camera to that position on the globe and take the dot product. The dot product returns the cosine of the angle between the to vectors. This gives us a value from -1 to +1 where -1 means the label is facing the camera, 0 means the label is directly on the edge of the sphere relative to the camera, and anything greater than zero is behind. We then use that value to show or hide the element.

In the diagram we can see the dot product of the direction the label is facing to direction from the camera to that position. If you rotate the direction you'll see the dot product is -1.0 when the direction is directly facing the camera, it's 0.0 when exactly on the tangent of the sphere relative to the camera or to put it another way it's 0 when the 2 vectors are perpendicular to each other, 90 degrees It's greater than zero with the label is behind the sphere.

const tempV = new THREE.Vector3();
const cameraToPoint = new THREE.Vector3();
const cameraPosition = new THREE.Vector3();
const normalMatrix = new THREE.Matrix3();

function updateLabels() {
  // exit if we have not yet loaded the JSON file
  if (!countryInfos) {
    return;
  }

  const minVisibleDot = 0.2;
  // get a matrix that represents a relative orientation of the camera
  normalMatrix.getNormalMatrix(camera.matrixWorldInverse);
  // get the camera's position
  camera.getWorldPosition(cameraPosition);
  for (const countryInfo of countryInfos) {
    const {position, elem} = countryInfo;

    // Orient the position based on the camera's orientation.
    // Since the sphere is at the origin and the sphere is a unit sphere
    // this gives us a camera relative direction vector for the position.
    tempV.copy(position);
    tempV.applyMatrix3(normalMatrix);

    // compute the direction to this position from the camera
    cameraToPoint.copy(position);
    cameraToPoint.applyMatrix4(camera.matrixWorldInverse).normalize();

    // get the dot product of camera relative direction to this position
    // on the globe with the direction from the camera to that point.
    // 1 = facing directly towards the camera
    // 0 = exactly on tangent of the sphere from the camera
    // < 0 = facing away
    const dot = tempV.dot(cameraToPoint);

    // if the orientation is not facing us hide it.
    if (dot < minVisibleDot) {
      elem.style.display = 'none';
      continue;
    }

    // restore the element to its default display style
    elem.style.display = '';

    // get the normalized screen coordinate of that position
    // x and y will be in the -1 to +1 range with x = -1 being
    // on the left and y = -1 being on the bottom
    tempV.copy(position);
    tempV.project(camera);

    // convert the normalized position to CSS coordinates
    const x = (tempV.x *  .5 + .5) * canvas.clientWidth;
    const y = (tempV.y * -.5 + .5) * canvas.clientHeight;

    // move the elem to that position
    countryInfo.elem.style.transform = translate(-50%, -50%) translate(${x}px,${y}px);

    // set the zIndex for sorting
    elem.style.zIndex = (-tempV.z * .5 + .5) * 100000 | 0;
  }
}

Filtering Labels by Area and Adding GUI

For issue #2, too many labels we need some way to decide which labels to show. One way would be to only show labels for large countries. The data we're loading contains min and max values for the area a country covers. From that we can compute an area and then use that area to decide whether or not to display the country.

At init time let's compute the area.

Then at render time let's use the area to decide to display the label or not.

Finally, since I'm not sure what good values are for these settings lets add a GUI so we can play with them.

You can see as you rotate the earth labels that go behind disappear. Adjust the minVisibleDot to see the cutoff change. You can also adjust the minArea value to see larger or smaller countries appear.

The more I worked on this the more I realized just how much work is put into Google Maps. They have also have to decide which labels to show. I'm pretty sure they use all kinds of criteria. For example your current location, your default language setting, your account settings if you have an account, they probably use population or popularity, they might give priority to the countries in the center of the view, etc ... Lots to think about.

In any case I hope these examples gave you some idea of how to align HTML elements with your 3D.

const labelParentElem = document.querySelector('#labels');
for (const countryInfo of countryInfos) {
  const {lat, lon, min, max, name} = countryInfo;

  // adjust the helpers to point to the latitude and longitude
  lonHelper.rotation.y = THREE.MathUtils.degToRad(lon) + lonFudge;
  latHelper.rotation.x = THREE.MathUtils.degToRad(lat) + latFudge;

  // get the position of the lat/lon
  positionHelper.updateWorldMatrix(true, false);
  const position = new THREE.Vector3();
  positionHelper.getWorldPosition(position);
  countryInfo.position = position;

  // compute the area for each country
  const width = max[0] - min[0];
  const height = max[1] - min[1];
  const area = width * height;
  countryInfo.area = area;

  // add an element for each country
  const elem = document.createElement('div');
  elem.textContent = name;
  labelParentElem.appendChild(elem);
  countryInfo.elem = elem;
}

const large = 20 * 20;
const maxVisibleDot = 0.2;
// get a matrix that represents a relative orientation of the camera
normalMatrix.getNormalMatrix(camera.matrixWorldInverse);
// get the camera's position
camera.getWorldPosition(cameraPosition);
for (const countryInfo of countryInfos) {
  const {position, elem, area} = countryInfo;
  // large enough?
  if (area < large) {
    elem.style.display = 'none';
    continue;
  }

  ...

import * as THREE from 'three';
import {OrbitControls} from 'three/addons/controls/OrbitControls.js';
import {GUI} from 'three/addons/libs/lil-gui.module.min.js';

const settings = {
  minArea: 20,
  maxVisibleDot: -0.2,
};
const gui = new GUI({width: 300});
gui.add(settings, 'minArea', 0, 50).onChange(requestRenderIfNotRequested);
gui.add(settings, 'maxVisibleDot', -1, 1, 0.01).onChange(requestRenderIfNotRequested);

function updateLabels() {
  if (!countryInfos) {
    return;
  }

  const large = settings.minArea * settings.minArea;
  // get a matrix that represents a relative orientation of the camera
  normalMatrix.getNormalMatrix(camera.matrixWorldInverse);
  // get the camera's position
  camera.getWorldPosition(cameraPosition);
  for (const countryInfo of countryInfos) {

    ...

    // if the orientation is not facing us hide it.
    if (dot > settings.maxVisibleDot) {
      elem.style.display = 'none';
      continue;
    }


Indexed Textures for Picking and Color

Summary: A tutorial on using indexed textures in Three.js for GPU-based picking and per-country highlighting on a 3D globe, including shader modifications via onBeforeCompile to implement paletted graphics for highlighting selections.

Introduction

This article is a continuation of an article about aligning html elements to 3d. If you haven't read that yet you should start there before continuing here.

Sometimes using three.js requires coming up with creative solutions. I'm not sure this is a great solution but I thought I'd share it and you can see if it suggests any solutions for your needs.

In the previous article we displayed country names around a 3d globe. How would we go about letting the user select a country and show their selection?

The Problem: Picking Countries on a Globe

The first idea that comes to mind is to generate geometry for each country. We could use a picking solution like we covered before. We'd build 3D geometry for each country. If the user clicks on the mesh for that country we'd know what country was clicked.

So, just to check that solution I tried generating 3D meshes of all the countries using the same data I used to generate the outlines in the previous article. The result was a 15.5meg binary GLTF (.glb) file. Making the user download 15.5meg sounds like too much to me.

There are lots of ways to compress the data. The first would probably be to apply some algorithm to lower the resolution of the outlines. I didn't spend any time pursuing that solution. For borders of the USA that's probably a huge win. For a borders of Canada probably much less.

Another solution would be to use just actual data compression. For example gzipping the file brought it down to 11meg. That's 30% less but arguably not enough.

We could store all the data as 16bit ranged values instead of 32bit float values. Or we could use something like draco compression and maybe that would be enough. I didn't check and I would encourage you to check yourself and tell me how it goes as I'd love to know. 😅

The Solution: Index Texture for Picking

In my case I thought about the GPU picking solution we covered at the end of the article on picking. In that solution we drew every mesh with a unique color that represented that mesh's id. We then drew all the meshes and looked at the color that was clicked on.

Taking inspiration from that we could pre-generate a map of countries where each country's color is its index number in our array of countries. We could then use a similar GPU picking technique. We'd draw the globe off screen using this index texture. Looking at the color of the pixel the user clicks would tell us the country id.

So, I wrote some code to generate such a texture. Here it is.

Note: The data used to generate this texture comes from this website and is therefore licensed as CC-BY-SA.

It's only 217k, much better than the 14meg for the country meshes. In fact we could probably even lower the resolution but 217k seems good enough for now.

So let's try using it for picking countries.

Creating a Picking Scene

Grabbing code from the gpu picking example we need a scene for picking.

const pickingScene = new THREE.Scene();
pickingScene.background = new THREE.Color(0);

Adding the Globe with Index Texture to the Picking Scene

and we need to add the globe with the our index texture to the picking scene.

{
  const loader = new THREE.TextureLoader();
  const geometry = new THREE.SphereGeometry(1, 64, 32);

+  const indexTexture = loader.load('resources/data/world/country-index-texture.png', render);
+  indexTexture.minFilter = THREE.NearestFilter;
+  indexTexture.magFilter = THREE.NearestFilter;
+
+  const pickingMaterial = new THREE.MeshBasicMaterial({map: indexTexture});
+  pickingScene.add(new THREE.Mesh(geometry, pickingMaterial));

  const texture = loader.load('resources/data/world/country-outlines-4k.png', render);
  const material = new THREE.MeshBasicMaterial({map: texture});
  scene.add(new THREE.Mesh(geometry, material));
}

The GPUPickHelper Class

Then let's copy over the `GPUPickingHelper` class we used before with a few minor changes.

class GPUPickHelper {
  constructor() {
    // create a 1x1 pixel render target
    this.pickingTexture = new THREE.WebGLRenderTarget(1, 1);
    this.pixelBuffer = new Uint8Array(4);
-    this.pickedObject = null;
-    this.pickedObjectSavedColor = 0;
  }
  pick(cssPosition, scene, camera) {
    const {pickingTexture, pixelBuffer} = this;

    // set the view offset to represent just a single pixel under the mouse
    const pixelRatio = renderer.getPixelRatio();
    camera.setViewOffset(
        renderer.getContext().drawingBufferWidth,   // full width
        renderer.getContext().drawingBufferHeight,  // full top
        cssPosition.x * pixelRatio | 0,             // rect x
        cssPosition.y * pixelRatio | 0,             // rect y
        1,                                          // rect width
        1,                                          // rect height
    );
    // render the scene
    renderer.setRenderTarget(pickingTexture);
    renderer.render(scene, camera);
    renderer.setRenderTarget(null);
    // clear the view offset so rendering returns to normal
    camera.clearViewOffset();
    //read the pixel
    renderer.readRenderTargetPixels(
        pickingTexture,
        0,   // x
        0,   // y
        1,   // width
        1,   // height
        pixelBuffer);

+    const id =
+        (pixelBuffer[0] << 16) |
+        (pixelBuffer[1] <<  8) |
+        (pixelBuffer[2] <<  0);
+
+    return id;
-    const id =
-        (pixelBuffer[0] << 16) |
-        (pixelBuffer[1] <<  8) |
-        (pixelBuffer[2]      );
-    const intersectedObject = idToObject[id];
-    if (intersectedObject) {
-      // pick the first object. It's the closest one
-      this.pickedObject = intersectedObject;
-      // save its color
-      this.pickedObjectSavedColor = this.pickedObject.material.emissive.getHex();
-      // set its emissive color to flashing red/yellow
-      this.pickedObject.material.emissive.setHex((time * 8) % 2 > 1 ? 0xFFFF00 : 0xFF0000);
-    }
  }
}

Picking Countries

Now we can use that to pick countries.

The code above sets/unsets the `selected` property on the array of countries. If `shift` or `ctrl` or `cmd` is pressed then you can select more than one country.

const pickHelper = new GPUPickHelper();

function getCanvasRelativePosition(event) {
  const rect = canvas.getBoundingClientRect();
  return {
    x: (event.clientX - rect.left) * canvas.width  / rect.width,
    y: (event.clientY - rect.top ) * canvas.height / rect.height,
  };
}

function pickCountry(event) {
  // exit if we have not loaded the data yet
  if (!countryInfos) {
    return;
  }

  const position = getCanvasRelativePosition(event);
  const id = pickHelper.pick(position, pickingScene, camera);
  if (id > 0) {
    // we clicked a country. Toggle its 'selected' property
    const countryInfo = countryInfos[id - 1];
    const selected = !countryInfo.selected;
    // if we're selecting this country and modifiers are not
    // pressed unselect everything else.
    if (selected && !event.shiftKey && !event.ctrlKey && !event.metaKey) {
      unselectAllCountries();
    }
    numCountriesSelected += selected ? 1 : -1;
    countryInfo.selected = selected;
  } else if (numCountriesSelected) {
    // the ocean or sky was clicked
    unselectAllCountries();
  }
  requestRenderIfNotRequested();
}

function unselectAllCountries() {
  numCountriesSelected = 0;
  countryInfos.forEach((countryInfo) => {
    countryInfo.selected = false;
  });
}

canvas.addEventListener('pointerup', pickCountry);

Showing Selected Countries

All that's left is showing the selected countries. For now let's just update the labels.

function updateLabels() {
  // exit if we have not loaded the data yet
  if (!countryInfos) {
    return;
  }

  const large = settings.minArea * settings.minArea;
  // get a matrix that represents a relative orientation of the camera
  normalMatrix.getNormalMatrix(camera.matrixWorldInverse);
  // get the camera's position
  camera.getWorldPosition(cameraPosition);
  for (const countryInfo of countryInfos) {
-    const {position, elem, area} = countryInfo;
-    // large enough?
-    if (area < large) {
+    const {position, elem, area, selected} = countryInfo;
+    const largeEnough = area >= large;
+    const show = selected || (numCountriesSelected === 0 && largeEnough);
+    if (!show) {
      elem.style.display = 'none';
      continue;
    }

    ...

Paletted Graphics / Indexed Color

The code stills shows countries based on their area but if you click one just that one will have a label.

So that seems like a reasonable solution for picking countries but what about highlighting the selected countries?

For that we can take inspiration from paletted graphics.

Paletted graphics or Indexed Color is what older systems like the Atari 800, Amiga, NES, Super Nintendo, and even older IBM PCs used. Instead of storing bitmaps as RGBA colors 8bits per color, 32 bytes per pixel or more, they stored bitmaps as 8bit values or less. The value for each pixel was an index into a palette. So for example a value of 3 in the image means "display color 3". What color color#3 is is defined somewhere else called a "palette".

In JavaScript you can think of it like this

Applying Palettes to Country Textures

Where each pixel in the image data is an index into palette. If you interpreted the image data through the palette above you'd get this image

In our case we already have a texture above that has a different id per country. So, we could use that same texture through a palette texture to give each country its own color. By changing the palette texture we can color each individual country. For example by setting the entire palette texture to black and then for one country's entry in the palette a different color, we can highlight just that country.

To do paletted index graphics requires some custom shader code. Let's modify the default shaders in three.js. That way we can use lighting and other features if we want.

Like we covered in the article on animating lots of objects we can modify the default shaders by adding a function to a material's `onBeforeCompile` property.

Default Fragment Shader

The default fragment shader looks something like this before compiling.

#include <common>
#include <color_pars_fragment>
#include <uv_pars_fragment>
#include <map_pars_fragment>
#include <alphamap_pars_fragment>
#include <aomap_pars_fragment>
#include <lightmap_pars_fragment>
#include <envmap_pars_fragment>
#include <fog_pars_fragment>
#include <specularmap_pars_fragment>
#include <logdepthbuf_pars_fragment>
#include <clipping_planes_pars_fragment>
void main() {
    #include <clipping_planes_fragment>
    vec4 diffuseColor = vec4( diffuse, opacity );
    #include <logdepthbuf_fragment>
    #include <map_fragment>
    #include <color_fragment>
    #include <alphamap_fragment>
    #include <alphatest_fragment>
    #include <specularmap_fragment>
    ReflectedLight reflectedLight = ReflectedLight( vec3( 0.0 ), vec3( 0.0 ), vec3( 0.0 ), vec3( 0.0 ) );
    #ifdef USE_LIGHTMAP
        reflectedLight.indirectDiffuse += texture2D( lightMap, vLightMapUv ).xyz * lightMapIntensity;
    #else
        reflectedLight.indirectDiffuse += vec3( 1.0 );
    #endif
    #include <aomap_fragment>
    reflectedLight.indirectDiffuse *= diffuseColor.rgb;
    vec3 outgoingLight = reflectedLight.indirectDiffuse;
    #include <envmap_fragment>
    gl_FragColor = vec4( outgoingLight, diffuseColor.a );
    #include <premultiplied_alpha_fragment>
    #include <tonemapping_fragment>
    #include <colorspace_fragment>
    #include <fog_fragment>
}

Modifying the Shader with onBeforeCompile

Digging through all those snippets we find that three.js uses a variable called `diffuseColor` to manage the base material color. It sets this in the `<color_fragment>` snippet so we should be able to modify it after that point.

`diffuseColor` at that point in the shader should already be the color from our outline texture so we can look up the color from a palette texture and mix them for the final result.

Like we did before we'll make an array of search and replacement strings and apply them to the shader in `Material.onBeforeCompile`.

Above can see above we add 3 uniforms, `indexTexture`, `paletteTexture`, and `paletteTextureWidth`. We get a color from the `indexTexture` and convert it to an index. `vUv` is the texture coordinates provided by three.js. We then use that index to get a color out of the palette texture. We then mix the result with the current `diffuseColor`. The `diffuseColor` at this point is our black and white outline texture so if we add the 2 colors we'll get white outlines. If we subtract the current diffuse color we'll get black outlines.

{
  const loader = new THREE.TextureLoader();
  const geometry = new THREE.SphereGeometry(1, 64, 32);

  const indexTexture = loader.load('resources/data/world/country-index-texture.png', render);
  indexTexture.minFilter = THREE.NearestFilter;
  indexTexture.magFilter = THREE.NearestFilter;

  const pickingMaterial = new THREE.MeshBasicMaterial({map: indexTexture});
  pickingScene.add(new THREE.Mesh(geometry, pickingMaterial));

+  const fragmentShaderReplacements = [
+    {
+      from: '#include <common>',
+      to:
+        #include <common>
+        uniform sampler2D indexTexture;
+        uniform sampler2D paletteTexture;
+        uniform float paletteTextureWidth;
+      ,
+    },
+    {
+      from: '#include <color_fragment>',
+      to:
+        #include <color_fragment>
+        {
+          vec4 indexColor = texture2D(indexTexture, vUv);
+          float index = indexColor.r * 255.0 + indexColor.g * 255.0 * 256.0;
+          vec2 paletteUV = vec2((index + 0.5) / paletteTextureWidth, 0.5);
+          vec4 paletteColor = texture2D(paletteTexture, paletteUV);
+          // diffuseColor.rgb += paletteColor.rgb;   // white outlines
+          diffuseColor.rgb = paletteColor.rgb - diffuseColor.rgb;  // black outlines
+        }
+      ,
+    },
+  ];

  const texture = loader.load('resources/data/world/country-outlines-4k.png', render);
  const material = new THREE.MeshBasicMaterial({map: texture});
+  material.onBeforeCompile = function(shader) {
+    fragmentShaderReplacements.forEach((rep) => {
+      shader.fragmentShader = shader.fragmentShader.replace(rep.from, rep.to);
+    });
+  };
  scene.add(new THREE.Mesh(geometry, material));
}

Setting Up the Palette Texture

Before we can render we need to setup the palette texture and these 3 uniforms.

For the palette texture it just needs to be wide enough to hold one color per country + one for the ocean (id = 0). There are 240 something countries. We could wait until the list of countries loads to get an exact number or look it up. There's not much harm in just picking some larger number so let's choose 512.

Here's the code to create the palette texture

const maxNumCountries = 512;
const paletteTextureWidth = maxNumCountries;
const paletteTextureHeight = 1;
const palette = new Uint8Array(paletteTextureWidth * 4);
const paletteTexture = new THREE.DataTexture(
    palette, paletteTextureWidth, paletteTextureHeight);
paletteTexture.minFilter = THREE.NearestFilter;
paletteTexture.magFilter = THREE.NearestFilter;

DataTexture Description

A `DataTexture` let's us give a texture raw data. In this case we're giving it 512 RGBA colors, 4 bytes each where each byte is red, green, and blue respectively using values that go from 0 to 255.

Let's fill it with random colors just to see it work

for (let i = 1; i < palette.length; ++i) {
  palette[i] = Math.random() * 256;
}
// set the ocean color (index #0)
palette.set([100, 200, 255, 255], 0);
paletteTexture.needsUpdate = true;

Updating the Palette Texture

Anytime we want three.js to update the palette texture with the contents of the `palette` array we need to set `paletteTexture.needsUpdate` to `true`.

And then we still need to set the uniforms on the material.

const geometry = new THREE.SphereGeometry(1, 64, 32);
const material = new THREE.MeshBasicMaterial({map: texture});
material.onBeforeCompile = function(shader) {
  fragmentShaderReplacements.forEach((rep) => {
    shader.fragmentShader = shader.fragmentShader.replace(rep.from, rep.to);
  });
+  shader.uniforms.paletteTexture = {value: paletteTexture};
+  shader.uniforms.indexTexture = {value: indexTexture};
+  shader.uniforms.paletteTextureWidth = {value: paletteTextureWidth};
};
scene.add(new THREE.Mesh(geometry, material));

Color Helper Function

Now that we can see the index and palette textures are working let's manipulate the palette for highlighting.

First let's make function that will let us pass in a three.js style color and give us values we can put in the palette texture.

const tempColor = new THREE.Color();
function get255BasedColor(color) {
  tempColor.set(color);
  const base = tempColor.toArray().map(v => v * 255);
  base.push(255); // alpha
  return base;
}

Building the Palette

Calling it like this `color = get255BasedColor('red')` will return an array like `[255, 0, 0, 255]`.

Next let's use it to make a few colors and fill out the palette.

const selectedColor = get255BasedColor('red');
const unselectedColor = get255BasedColor('#444');
const oceanColor = get255BasedColor('rgb(100,200,255)');
resetPalette();

function setPaletteColor(index, color) {
  palette.set(color, index * 4);
}

function resetPalette() {
  // make all colors the unselected color
  for (let i = 1; i < maxNumCountries; ++i) {
    setPaletteColor(i, unselectedColor);
  }

  // set the ocean color (index #0)
  setPaletteColor(0, oceanColor);
  paletteTexture.needsUpdate = true;
}

Updating Palette on Country Selection

Now let's use those functions to update the palette when a country is selected

function getCanvasRelativePosition(event) {
  const rect = canvas.getBoundingClientRect();
  return {
    x: (event.clientX - rect.left) * canvas.width  / rect.width,
    y: (event.clientY - rect.top ) * canvas.height / rect.height,
  };
}

function pickCountry(event) {
  // exit if we have not loaded the data yet
  if (!countryInfos) {
    return;
  }

  const position = getCanvasRelativePosition(event);
  const id = pickHelper.pick(position, pickingScene, camera);
  if (id > 0) {
    const countryInfo = countryInfos[id - 1];
    const selected = !countryInfo.selected;
    if (selected && !event.shiftKey && !event.ctrlKey && !event.metaKey) {
      unselectAllCountries();
    }
    numCountriesSelected += selected ? 1 : -1;
    countryInfo.selected = selected;
+    setPaletteColor(id, selected ? selectedColor : unselectedColor);
+    paletteTexture.needsUpdate = true;
  } else if (numCountriesSelected) {
    unselectAllCountries();
  }
  requestRenderIfNotRequested();
}

function unselectAllCountries() {
  numCountriesSelected = 0;
  countryInfos.forEach((countryInfo) => {
    countryInfo.selected = false;
  });
+  resetPalette();
}

Handling Click vs Drag

One minor thing is we can't spin the globe without changing the selection state. If we select a country and then want to rotate the globe the selection will change.

Let's try to fix that. Off the top of my head we can check 2 things. How much time passed between clicking and letting go. Another is did the user actually move the mouse. If the time is short or if they didn't move the mouse then it was probably a click. Otherwise they were probably trying to drag the globe.

+const maxClickTimeMs = 200;
+const maxMoveDeltaSq = 5 * 5;
+const startPosition = {};
+let startTimeMs;
+
+function recordStartTimeAndPosition(event) {
+  startTimeMs = performance.now();
+  const pos = getCanvasRelativePosition(event);
+  startPosition.x = pos.x;
+  startPosition.y = pos.y;
+}

function getCanvasRelativePosition(event) {
  const rect = canvas.getBoundingClientRect();
  return {
    x: (event.clientX - rect.left) * canvas.width  / rect.width,
    y: (event.clientY - rect.top ) * canvas.height / rect.height,
  };
}

function pickCountry(event) {
  // exit if we have not loaded the data yet
  if (!countryInfos) {
    return;
  }

+  // if it's been a moment since the user started
+  // then assume it was a drag action, not a select action
+  const clickTimeMs = performance.now() - startTimeMs;
+  if (clickTimeMs > maxClickTimeMs) {
+    return;
+  }
+
+  // if they moved assume it was a drag action
+  const position = getCanvasRelativePosition(event);
+  const moveDeltaSq = (startPosition.x - position.x) ** 2 +
+                      (startPosition.y - position.y) ** 2;
+  if (moveDeltaSq > maxMoveDeltaSq) {
+    return;
+  }

-  const position = {x: event.clientX, y: event.clientY};
  const id = pickHelper.pick(position, pickingScene, camera);
  if (id > 0) {
    const countryInfo = countryInfos[id - 1];
    const selected = !countryInfo.selected;
    if (selected && !event.shiftKey && !event.ctrlKey && !event.metaKey) {
      unselectAllCountries();
    }
    numCountriesSelected += selected ? 1 : -1;
    countryInfo.selected = selected;
    setPaletteColor(id, selected ? selectedColor : unselectedColor);
    paletteTexture.needsUpdate = true;
  } else if (numCountriesSelected) {
    unselectAllCountries();
  }
  requestRenderIfNotRequested();
}

function unselectAllCountries() {
  numCountriesSelected = 0;
  countryInfos.forEach((countryInfo) => {
    countryInfo.selected = false;
  });
  resetPalette();
}

+canvas.addEventListener('pointerdown', recordStartTimeAndPosition);
canvas.addEventListener('pointerup', pickCountry);

Conclusion

I'm not a UX expert so I'd love to hear if there is a better solution.

I hope that gave you some idea of how indexed graphics can be useful and how you can modify the shaders three.js makes to add simple features. How to use GLSL, the language the shaders are written in, is too much for this article. There are a few links to some info in the article on post processing.


Canvas Textures

Summary: Extraction fallback content. [stop]

Canvas Textures

##### Canvas Textures

This article continues from [the article on textures](textures.html).
If you haven't read that yet you should probably start there.

In [the previous article on textures](textures.html) we mostly used
image files for textures. Sometimes though we want to generate a texture
at runtime. One way to do this is to use a [`CanvasTexture`](/docs/#api/en/textures/CanvasTexture).

A canvas texture takes a `<canvas>` as its input. If you don't know how to
draw with the 2D canvas API on a canvas [there's a good tutorial on MDN](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial).

Let's make a simple canvas program. Here's one that draws dots at random places in random colors.

it's pretty straight forward.

[click here to open in a separate window](/manual/examples/canvas-random-dots.html)

Now let's use it to texture something. We'll start with the example of texturing
a cube from [the previous article](textures.html).
We'll remove the code that loads an image and instead use
our canvas by creating a [`CanvasTexture`](/docs/#api/en/textures/CanvasTexture) and passing it the canvas we created.

And then call the code to draw a random dot in our render loop

The only extra thing we need to do is set the `needsUpdate` property
of the [`CanvasTexture`](/docs/#api/en/textures/CanvasTexture) to tell three.js to update the texture with
the latest contents of the canvas.

And with that we have a canvas textured cube

[click here to open in a separate window](/manual/examples/canvas-textured-cube.html)

Note that if you want to use three.js to draw into the canvas you're
better off using a `RenderTarget` which is covered in [this article](rendertargets.html).

A common use case for canvas textures is to provide text in a scene.
For example if you wanted to put a person's name on their character's
badge you might use a canvas texture to texture the badge.

Let's make a scene with 3 people and give each person a badge
or label.

Let's take the example above and remove all the cube related
stuff. Then let's set the background to white and add two [lights](lights.html).

Let's make some code to make a label using canvas 2D

Then we'll make simple people from a cylinder for the body, a sphere
for the head, and a plane for the label.

First let's make the shared geometry.

Then let's make a function to build a person from these
parts.

You can see above we put the body, head, and label on a root
[`Object3D`](/docs/#api/en/core/Object3D) and adjust their positions. This would let us move the
root object if we wanted to move the people. The body is 2 units
high. If 1 unit equals 1 meter then the code above tries to
make the label in centimeters so they will be size centimeters
tall and however wide is needed to fit the text.

We can then make people with labels

What's left is to add some [`OrbitControls`](/docs/#examples/controls/OrbitControls) so we can move
the camera.

and we get simple labels.

[click here to open in a separate window](/manual/examples/canvas-textured-labels.html)

Some things to notice.

- If you zoom in the labels get pretty low-res.

There is no easy solution. There are more complex font
rendering techniques but I know of no plugin solutions.
Plus they will require the user download font data which
would be slow.

One solution is to increase the resolution of the labels.
Try setting the size passed into to double what it is now
and setting `labelBaseScale` to half what it currently is.

- The labels get longer the longer the name.

If you wanted to fix this you'd instead choose a fixed sized
label and then squish the text.

This is pretty easy. Pass in a base width and scale the text to fit that
width like this

Then we can pass in a width for the labels

and we get labels where the text is centered and scaled to fit

[click here to open in a separate window](/manual/examples/canvas-textured-labels-scale-to-fit.html)

Above we used a new canvas for each texture. Whether or not to use a
canvas per texture is up to you. If you need to update them often then
having one canvas per texture is probably the best option. If they are
rarely or never updated then you can choose to use a single canvas
for multiple textures by forcing three.js to use the texture.
Let's change the code above to do just that.

[click here to open in a separate window](/manual/examples/canvas-textured-labels-one-canvas.html)

Another issue is that the labels don't always face the camera. If you're using
labels as badges that's probably a good thing. If you're using labels to put
names over players in a 3D game maybe you want the labels to always face the camera.
We'll cover how to do that in [an article on billboards](billboards.html).

For labels in particular, [another solution is to use HTML](align-html-elements-to-3d.html).
The labels in this article are  *inside the 3D world*  which is good if you want them
to be hidden by other objects where as [HTML labels](align-html-elements-to-3d.html) are always on top.

const cubes = [];  // just an array we can use to rotate the cubes
-const loader = new THREE.TextureLoader();
-
+const ctx = document.createElement('canvas').getContext('2d');
+ctx.canvas.width = 256;
+ctx.canvas.height = 256;
+ctx.fillStyle = '#FFF';
+ctx.fillRect(0, 0, ctx.canvas.width, ctx.canvas.height);
+const texture = new THREE.CanvasTexture(ctx.canvas);

const material = new THREE.MeshBasicMaterial({
-  map: loader.load('resources/images/wall.jpg'),
+  map: texture,
});
const cube = new THREE.Mesh(geometry, material);
scene.add(cube);
cubes.push(cube);  // add to our list of cubes to rotate

function render(time) {
  time *= 0.001;

  if (resizeRendererToDisplaySize(renderer)) {
    const canvas = renderer.domElement;
    camera.aspect = canvas.clientWidth / canvas.clientHeight;
    camera.updateProjectionMatrix();
  }

+  drawRandomDot();
+  texture.needsUpdate = true;

  cubes.forEach((cube, ndx) => {
    const speed = .2 + ndx * .1;
    const rot = time * speed;
    cube.rotation.x = rot;
    cube.rotation.y = rot;
  });

  renderer.render(scene, camera);

  requestAnimationFrame(render);
}

const scene = new THREE.Scene();
+scene.background = new THREE.Color('white');
+
+function addLight(position) {
+  const color = 0xFFFFFF;
+  const intensity = 1;
+  const light = new THREE.DirectionalLight(color, intensity);
+  light.position.set(...position);
+  scene.add(light);
+  scene.add(light.target);
+}
+addLight([-3, 1, 1]);
+addLight([ 2, 1, .5]);

+const bodyRadiusTop = .4;
+const bodyRadiusBottom = .2;
+const bodyHeight = 2;
+const bodyRadialSegments = 6;
+const bodyGeometry = new THREE.CylinderGeometry(
+    bodyRadiusTop, bodyRadiusBottom, bodyHeight, bodyRadialSegments);
+
+const headRadius = bodyRadiusTop * 0.8;
+const headLonSegments = 12;
+const headLatSegments = 5;
+const headGeometry = new THREE.SphereGeometry(
+    headRadius, headLonSegments, headLatSegments);
+
+const labelGeometry = new THREE.PlaneGeometry(1, 1);

+function makePerson(x, size, name, color) {
+  const canvas = makeLabelCanvas(size, name);
+  const texture = new THREE.CanvasTexture(canvas);
+  // because our canvas is likely not a power of 2
+  // in both dimensions set the filtering appropriately.
+  texture.minFilter = THREE.LinearFilter;
+  texture.wrapS = THREE.ClampToEdgeWrapping;
+  texture.wrapT = THREE.ClampToEdgeWrapping;
+
+  const labelMaterial = new THREE.MeshBasicMaterial({
+    map: texture,
+    side: THREE.DoubleSide,
+    transparent: true,
+  });
+  const bodyMaterial = new THREE.MeshPhongMaterial({
+    color,
+    flatShading: true,
+  });
+
+  const root = new THREE.Object3D();
+  root.position.x = x;
+
+  const body = new THREE.Mesh(bodyGeometry, bodyMaterial);
+  root.add(body);
+  body.position.y = bodyHeight / 2;
+
+  const head = new THREE.Mesh(headGeometry, bodyMaterial);
+  root.add(head);
+  head.position.y = bodyHeight + headRadius * 1.1;
+
+  const label = new THREE.Mesh(labelGeometry, labelMaterial);
+  root.add(label);
+  label.position.y = bodyHeight * 4 / 5;
+  label.position.z = bodyRadiusTop * 1.01;
+
+  // if units are meters then 0.01 here makes size
+  // of the label into centimeters.
+  const labelBaseScale = 0.01;
+  label.scale.x = canvas.width  * labelBaseScale;
+  label.scale.y = canvas.height * labelBaseScale;
+
+  scene.add(root);
+  return root;
+}

+makePerson(-3, 32, 'Purple People Eater', 'purple');
+makePerson(-0, 32, 'Green Machine', 'green');
+makePerson(+3, 32, 'Red Menace', 'red');

import * as THREE from 'three';
+import {OrbitControls} from 'three/addons/controls/OrbitControls.js';

const fov = 75;
const aspect = 2;  // the canvas default
const near = 0.1;
-const far = 5;
+const far = 50;
const camera = new THREE.PerspectiveCamera(fov, aspect, near, far);
-camera.position.z = 2;
+camera.position.set(0, 2, 5);

+const controls = new OrbitControls(camera, canvas);
+controls.target.set(0, 2, 0);
+controls.update();

-function makePerson(x, size, name, color) {
-  const canvas = makeLabelCanvas(size, name);
+function makePerson(x, labelWidth, size, name, color) {
+  const canvas = makeLabelCanvas(labelWidth, size, name);

...

}

-makePerson(-3, 32, 'Purple People Eater', 'purple');
-makePerson(-0, 32, 'Green Machine', 'green');
-makePerson(+3, 32, 'Red Menace', 'red');
+makePerson(-3, 150, 32, 'Purple People Eater', 'purple');
+makePerson(-0, 150, 32, 'Green Machine', 'green');
+makePerson(+3, 150, 32, 'Red Menace', 'red');


Billboards

Summary: A guide on using Three.js Sprite and SpriteMaterial to create billboards — 2D planes that always face the camera — for purposes like character labels and rendering facades of 3D objects to improve performance.

Billboards

In a previous article we used a CanvasTexture to make labels / badges on characters. Sometimes we'd like to make labels or other things that always face the camera. Three.js provides the Sprite and SpriteMaterial to make this happen.

Let's change the badge example from the article on canvas textures to use Sprite and SpriteMaterial

function makePerson(x, labelWidth, size, name, color) {
  const canvas = makeLabelCanvas(labelWidth, size, name);
  const texture = new THREE.CanvasTexture(canvas);
  // because our canvas is likely not a power of 2
  // in both dimensions set the filtering appropriately.
  texture.minFilter = THREE.LinearFilter;
  texture.wrapS = THREE.ClampToEdgeWrapping;
  texture.wrapT = THREE.ClampToEdgeWrapping;

-  const labelMaterial = new THREE.MeshBasicMaterial({
+  const labelMaterial = new THREE.SpriteMaterial({
    map: texture,
-    side: THREE.DoubleSide,
    transparent: true,
  });

  const root = new THREE.Object3D();
  root.position.x = x;

  const body = new THREE.Mesh(bodyGeometry, bodyMaterial);
  root.add(body);
  body.position.y = bodyHeight / 2;

  const head = new THREE.Mesh(headGeometry, bodyMaterial);
  root.add(head);
  head.position.y = bodyHeight + headRadius * 1.1;

-  const label = new THREE.Mesh(labelGeometry, labelMaterial);
+  const label = new THREE.Sprite(labelMaterial);
  root.add(label);
  label.position.y = bodyHeight * 4 / 5;
  label.position.z = bodyRadiusTop * 1.01;

Fixing label intersection

and the labels now always face the camera

[click here to open in a separate window]

One problem is from certain angles the labels now intersect the characters.

We can move the position of the labels to fix.

+// if units are meters then 0.01 here makes size
+// of the label into centimeters.
+const labelBaseScale = 0.01;
const label = new THREE.Sprite(labelMaterial);
root.add(label);
-label.position.y = bodyHeight * 4 / 5;
-label.position.z = bodyRadiusTop * 1.01;
+label.position.y = head.position.y + headRadius + size * labelBaseScale;

-// if units are meters then 0.01 here makes size
-// of the label into centimeters.
-const labelBaseScale = 0.01;
label.scale.x = canvas.width  * labelBaseScale;
label.scale.y = canvas.height * labelBaseScale;

Facades with billboards

Another thing we can do with billboards is draw facades.

Instead of drawing 3D objects we draw 2D planes with an image of 3D objects. This is often faster than drawing 3D objects.

For example let's make a scene with grid of trees. We'll make each tree from a cylinder for the base and a cone for the top.

First we make the cone and cylinder geometry and materials that all the trees will share

const trunkRadius = .2;
const trunkHeight = 1;
const trunkRadialSegments = 12;
const trunkGeometry = new THREE.CylinderGeometry(
    trunkRadius, trunkRadius, trunkHeight, trunkRadialSegments);

const topRadius = trunkRadius * 4;
const topHeight = trunkHeight * 2;
const topSegments = 12;
const topGeometry = new THREE.ConeGeometry(
    topRadius, topHeight, topSegments);

const trunkMaterial = new THREE.MeshPhongMaterial({color: 'brown'});
const topMaterial = new THREE.MeshPhongMaterial({color: 'green'});

Making tree meshes

Then we'll make a function that makes a Mesh each for the trunk and top of a tree and parents both to an Object3D.

function makeTree(x, z) {
  const root = new THREE.Object3D();
  const trunk = new THREE.Mesh(trunkGeometry, trunkMaterial);
  trunk.position.y = trunkHeight / 2;
  root.add(trunk);

  const top = new THREE.Mesh(topGeometry, topMaterial);
  top.position.y = trunkHeight + topHeight / 2;
  root.add(top);

  root.position.set(x, 0, z);
  scene.add(root);

  return root;
}

Placing trees in a grid

Then we'll make a loop to place a grid of trees.

Adding ground and background

Let's also add a ground plane while we're at it

// add ground
{
  const size = 400;
  const geometry = new THREE.PlaneGeometry(size, size);
  const material = new THREE.MeshPhongMaterial({color: 'gray'});
  const mesh = new THREE.Mesh(geometry, material);
  mesh.rotation.x = Math.PI * -0.5;
  scene.add(mesh);
}

Setting the scene background

and change the background to light blue

const scene = new THREE.Scene();
-scene.background = new THREE.Color('white');
+scene.background = new THREE.Color('lightblue');

Rendering an object to a texture

and we get a grid of trees

[click here to open in a separate window]

There are 11x11 or 121 trees. Each tree is made from a 12 polygon cone and a 48 polygon trunk so each tree is 60 polygons. 121 * 60 is 7260 polygons. That's not that many but of course a more detailed 3D tree might be 1000-3000 polygons. If they were 3000 polygons each then 121 trees would be 363000 polygons to draw.

Using facades we can bring that number down.

We could manually create a facade in some painting program but let's write some code to try to generate one.

Let's write some code to render an object to a texture using a RenderTarget. We covered rendering to a RenderTarget in the article on render targets.

function frameArea(sizeToFitOnScreen, boxSize, boxCenter, camera) {
  const halfSizeToFitOnScreen = sizeToFitOnScreen * 0.5;
  const halfFovY = THREE.MathUtils.degToRad(camera.fov * .5);
  const distance = halfSizeToFitOnScreen / Math.tan(halfFovY);

  camera.position.copy(boxCenter);
  camera.position.z += distance;

  // pick some near and far values for the frustum that
  // will contain the box.
  camera.near = boxSize / 100;
  camera.far = boxSize * 100;

  camera.updateProjectionMatrix();
}

function makeSpriteTexture(textureSize, obj) {
  const rt = new THREE.WebGLRenderTarget(textureSize, textureSize);

  const aspect = 1;  // because the render target is square
  const camera = new THREE.PerspectiveCamera(fov, aspect, near, far);

  scene.add(obj);

  // compute the box that contains obj
  const box = new THREE.Box3().setFromObject(obj);

  const boxSize = box.getSize(new THREE.Vector3());
  const boxCenter = box.getCenter(new THREE.Vector3());

  // set the camera to frame the box
  const fudge = 1.1;
  const size = Math.max(...boxSize.toArray()) * fudge;
  frameArea(size, size, boxCenter, camera);

  renderer.autoClear = false;
  renderer.setRenderTarget(rt);
  renderer.render(scene, camera);
  renderer.setRenderTarget(null);
  renderer.autoClear = true;

  scene.remove(obj);

  return {
    position: boxCenter.multiplyScalar(fudge),
    scale: size,
    texture: rt.texture,
  };
}

Notes on the facade rendering code

Some things to note about the code above:

We're using the field of view (fov) defined above this code.

We're computing a box that contains the tree the same way we did in the article on loading a .obj file with a few minor changes.

We call frameArea again adapted the article on loading a .obj file. In this case we compute how far the camera needs to be away from the object given its field of view to contain the object. We then position the camera -z that distance from the center of the box that contains the object.

We multiply the size we want to fit by 1.1 (fudge) to make sure the tree fits completely in the render target. The issue here is the size we're using to calculate if the object fits in the camera's view is not taking into account that the very edges of the object will end up dipping outside area we calculated. We could compute how to make 100% of the box fit but that would waste space as well so instead we just *fudge* it.

Then we render to the render target and remove the object from the scene.

It's important to note we need the lights in the scene but we need to make sure nothing else is in the scene.

We also need to not set a background color on the scene

const scene = new THREE.Scene();
-scene.background = new THREE.Color('lightblue');

Making facades for a grid of trees

Finally we've made the texture we return it and the position and scale we need to make the facade so that it will appear to be in the same place.

We then make a tree and call this code and pass it in

// make billboard texture
const tree = makeTree(0, 0);
const facadeSize = 64;
const treeSpriteInfo = makeSpriteTexture(facadeSize, tree);

Replacing tree models with sprites

We can then make a grid of facades instead of a grid of tree models

+function makeSprite(spriteInfo, x, z) {
+  const {texture, offset, scale} = spriteInfo;
+  const mat = new THREE.SpriteMaterial({
+    map: texture,
+    transparent: true,
+  });
+  const sprite = new THREE.Sprite(mat);
+  scene.add(sprite);
+  sprite.position.set(
+      offset.x + x,
+      offset.y,
+      offset.z + z);
+  sprite.scale.set(scale, scale, scale);
+}

for (let z = -50; z <= 50; z += 10) {
  for (let x = -50; x <= 50; x += 10) {
-    makeTree(x, z);
+    makeSprite(treeSpriteInfo, x, z);
  }
}

Restoring the background

In the code above we apply the offset and scale needed to position the facade so it appears the same place the original tree would have appeared.

Now that we're done making the tree facade texture we can set the background again

Conclusion

and now we get a scene of tree facades

[click here to open in a separate window]

Compare to the trees models above and you can see it looks fairly similar. We used a low-res texture, just 64x64 pixels so the facades are blocky. You could increase the resolution. Often facades are used only in the far distance when they are fairly small so a low-res texture is enough and it saves on drawing detailed trees that are only a few pixels big when far away.

Another issue is we are only viewing the tree from one side. This is often solved by rendering more facades, say from 8 directions around the object and then setting which facade to show based on which direction the camera is looking at the facade.

Whether or not you use facades is up to you but hopefully this article gave you some ideas and suggested some solutions if you decide to use them.