Creating a custom outline shader in React Three Fiber

Introduction

In this blog, we’ll explore how to implement a simple outline shader in React Three Fiber (R3F). Outline shaders can help highlight specific objects, enhance interactivity, or add a unique artistic touch to your scene. Whether you’re building a 3D product configurator, a game, or just experimenting with creative designs, this tutorial will walk you through crafting a basic outline shader with minimal complexity. Let’s dive in!

Our goal!​

We want to create a highlight or outline effect around certain objects in our 3D scene. For example, when we hover the mouse over a mesh, that mesh should get a glowing outline around it, making it stand out. We will achieve this by creating a post-processing shader.

A brief overview​

The idea is pretty simple – Create a separate mask to render the scene again, only this time, we draw the selected (or hovered upon) objects in white and everything else is black. This can be further interpreted as creating silhouettes of the selected objects. The below gif shows how the silhouette mask works. 

After the mask is created, a post processing shader is run that looks at the mask and checks each pixel (in the mask). If there is a colour change in the neighbouring pixels, we have come across an edge and we draw an outline. 

The code!

1. Selected object

Firstly, we will set the selected object when we hover over an object in the scene. For this, we create a local state using useState.

const [selectedObject, setSelectedObject] = useState(null);

Then, using pointer events (over and leave), we set the selectedObject state to the given hovered object.

const handlePointerOver = (e) => {
    e.stopPropagation();
    setSelectedObject(e.object);
  };

const handlePointerLeave = (e) => {
    e.stopPropagation();
    setSelectedObject(null);
};

return (
  <mesh
    onPointerOver={handlePointerOver}
    onPointerLeave={handlePointerLeave}
    position={[4, 0, 0]}
  >
    <sphereGeometry args={[5]} />
    <meshStandardMaterial color={"lightgreen"} />
  </mesh>
);

2. The Silhouette Mask

The next step is to implement the silhouette mask. Again, the goal of this mask is to render the entire scene with the selected mesh displayed in white, while all other elements’ visibility is set to false (as can be viewed in the GIF).

We traverse the scene graph and check whether the traversed object is the same as the selected object. If so, we assign it a white color (using MeshBasicMaterial). Additionally, we will also use a map to store the objects in the scene with their original materials and visibility.

scene.traverse((obj) => {
    if (obj.isMesh) {
      originalMeshes.current.set(obj, {
        visible: obj.visible,
        material: obj.material,
      });
      if (selectedObject == obj) {
        obj.visible = true;
        obj.material = whiteMaterial;
      } else {
        obj.visible = false;
      }
    }
 });

Once we have created the mask, we assign the original material and visibility to all the meshes in the scene. Overall, the code would look something like this,

import { useFrame, useThree } from "@react-three/fiber";
import { useRef } from "react";
import * as THREE from "three";

/**
 * Mask Pass to render silhouettes of the meshes
 * Firstly, we traverse the scene and check if
 * mesh is the selected mesh. If so, give it white color
 * else, make the mesh invisible. Render it onto the FBO.
 * 
 * Then, restore the original meshes with the original 
 * visibility and materials
 * 
 */
const SilhouetteMaskPass = ({ selectedObject, fbo }) => {
  const { scene, camera, gl } = useThree();

  const whiteMaterial = new THREE.MeshBasicMaterial({color: "white"});
  const originalMeshes = useRef(new Map());

  useFrame(() => {
    // set selected mesh to white color
    // and rest all as invisible
    scene.traverse((obj) => {
      if (obj.isMesh) {
        originalMeshes.current.set(obj, {
          visible: obj.visible,
          material: obj.material,
        });
        if (selectedObject == obj) {
          obj.visible = true;
          obj.material = whiteMaterial;
        } else {
          obj.visible = false;
        }
      }
    });

    // render into frame buffer
    gl.setRenderTarget(fbo);
    gl.clear(1, 1, 1);
    gl.render(scene, camera);

    // reset render target
    gl.setRenderTarget(null);

    // set original material and visibility to meshes in the scene
    scene.traverse((obj) => {
      if(obj.isMesh && originalMeshes.current.has(obj)) {
        const {visible, material} = originalMeshes.current.get(obj);
        obj.visible = visible;
        obj.material = material;
      }
    })
  });

  return null;
};

export default SilhouetteMaskPass;

The silhouette mask pass also takes in a Frame Buffer Object (FBO) as a prop. This FBO is used as a render target on which our mask is rendered.

3. The Outline Fragment Shader

The task of the fragment shader is quite simple – to look for deviations in the red channel across neighbouring pixels. In the silhouette mask, the red channel will switch from 0 to 1 when a neighbouring pixel is within the white silhouette.

uniform sampler2D tSilhouette;
uniform vec2 uResolution;
uniform float uOutlineThickness; 
uniform vec3 uOutlineColor;
uniform sampler2D tDiffuse;

void mainImage(const in vec4 inputColor, const in vec2 uv, out vec4 outputColor) {
    float texelSizeX = uOutlineThickness * (1.0 / uResolution.x);
    float texelSizeY = uOutlineThickness * (1.0 / uResolution.y);
    
    float maskColor = texture2D(tSilhouette, uv).r;
    float left = texture2D(tSilhouette, uv + vec2(-texelSizeX, 0.0)).r;
    float right = texture2D(tSilhouette, uv + vec2(texelSizeX, 0.0)).r;
    float up = texture2D(tSilhouette, uv + vec2(0.0, texelSizeY)).r;
    float down = texture2D(tSilhouette, uv + vec2(0.0, -texelSizeY)).r;

    // calculate slope
    float dx = (left - right);
    float dy = (down - up);
    float edgeVal = sqrt(dx * dx + dy * dy);

    vec3 baseColor = inputColor.rgb;
    vec3 finalColor = mix(baseColor, uOutlineColor, clamp(edgeVal, 0.0, 1.0));

    outputColor = vec4(finalColor, 1.0);
}

A few key points to note:

1. The outline thickness is multiplied by the texel size, which defines how far the shader spans to sample neighboring pixels.
2. The final color is calculated as an interpolation between the base color and the outline color, which is passed as a uniform.
3. The interpolation factor is determined by the deviation value, clamped between 0 and 1.

4. The Outline Effect

Lastly, we create a post processing effect with the outline shader we just created. We pass the silhouette mask texture, resolution (to calculate texel’s dimensions), outline colour and outline thickness as uniforms.

/**
 * Outline effect that takes in, 
 * maskTexture - Silhouette FB texture
 * 
 * resolution - resolution of the canvas
 * 
 * outlineThickness - Thickness of the outline
 * 
 * outlineColor - Color of the outline
 */
class OutlineEffectImpl extends Effect {
  constructor({
    maskTexture,
    resolution = new Vector2(1024, 768),
    outlineThickness = 1.0,
    outlineColor = new Color("red"),
  }) {
    super("OutlineEffect", outlineFragmentShader, {
      uniforms: new Map([
        ["tSilhouette", new Uniform(maskTexture)],
        ["uResolution", new Uniform(resolution)],
        ["uOutlineThickness", new Uniform(outlineThickness)],
        ["uOutlineColor", new Uniform(outlineColor)],
      ]),
    });
  }
}

const OutlineEffect = forwardRef(
  (
    {
      maskTexture,
      resolution = new Vector2(1024, 768),
      outlineThickness = 1.0,
      outlineColor = new Color("red"),
    },
    ref
  ) => {
    const effect = useMemo(
      () =>
        new OutlineEffectImpl({
          maskTexture,
          resolution,
          outlineThickness,
          outlineColor,
        }),
      [maskTexture, resolution, outlineThickness, outlineColor]
    );
    return <primitive ref={ref} object={effect} dispose={null} />;
  }
);

export default OutlineEffect;

Wrapping it up!

To use the outline effect, we make use of the EffectComposer provided by @react-three/postprocessing. We add the components that we created above.

const OutlinePostProcessing = ({ selectedObject, showFBTexture = false, outlineColor, outlineThickness }) => {
  const fbo = useFBO({
    width: window.innerWidth,
    height: window.innerHeight,
    stencilBuffer: false,
    depthBuffer: true,
    format: THREE.RGBAFormat,
  });
  const { size } = useThree();
  useMemo(() => {
    fbo.setSize(size.width, size.height);
  }, [size, fbo]);

  return (
    <>
      <SilhouetteMaskPass selectedObject={selectedObject} fbo={fbo} />
      <EffectComposer>
        <OutlineEffect
          maskTexture={fbo.texture}
          resolution={[fbo.width, fbo.width]}
          outlineThickness={outlineThickness}
          outlineColor={new THREE.Color(outlineColor)}
        />
      </EffectComposer>
    </>
  );
};

export default OutlinePostProcessing;

This component can be injected within the canvas to create an outline effect!

Conclusions

This approach provides a simple yet effective way to render outlines on meshes. By utilizing a mask to create the outlines, issues like z-fighting can be avoided. However, this method has a limitation when outlining a group of selected overlapping meshes, as it generates an outline around the entire boundary of the group. There are more advanced techniques available to address this limitation, which I may explore in future blog posts!

Thanks for reading! ✨

---