GLSL NPR Toon Shaders
UC Berkeley - Summer 2020

Overview

For the final project of the CS184: Computer Graphics class at UC Berkeley, I chose to explore non-photorealistic rendering (NPR) in real time 3D applications through OpenGL Shading Language (GLSL) "Toon" shaders. I implemented a vertex shader and three fragment shaders: a Toon/Cel shader, a Noir shader with grain textures, and a "Hope" shader with halftone shading (inspired by Barack Obama's 2008 campaign poster). All three shaders use Lambert's Cosine Law to determine how to color a fragment, usually a pixel, based on the intensity of light at that fragment on the surface of a 3D model. I used GLSL, Three.js, JavaScript, and HTML/CSS to render 3D models of a sphere and Grogu from the Star Wars series The Mandalorian with my shaders (Thingiverse).

Technical Approach

Vertex Shader
In order to draw to the screen of a display, shaders run on the graphics processing unit (GPU) to compute positions in a 3D coordinate space and color in the pixels that will become an image. When working with 3D models, a vertex shader runs once per vertex to transform a model's 3D coordinates in model space, to world space, and then screen space (Learn OpenGL). This is done by applying a series of transformation matrices to rotate, move, and clip a model in a 3D scene into the viewpoint of a "camera" or "eye" that will be displayed to a screen as a 2D image. My vertex shader takes in the position of a given vertex, applies the model, view, and projection transformation matrices to the vertex, and outputs the screen space position of the vertex to give to the fragment shader. This output is placed into the gl_Position variable.

In addition to calculating the position, my vertex shader takes in the model space surface normal of a vertex and transforms it into world space. A surface normal is the perpendicular vector that points out from the tangent surface that a vertex lies on. This transformation uses a normal matrix, which is the inverse transpose of a model view matrix, to transform the surface normal to world space (WebGLProgram). Just using the model view matrix on a surface normal would, in some cases, incorrectly scale the normal if the scale was not uniform (since the model would stretch unevenly and change the direction of the normal). The world space surface normal is passed onto the fragment shader to be used for calculations of light intensity.

Lambert's Cosine Law
Lambert's Cosine Law states that ‘irradiance at [a] surface is proportional to [the] cosine of [the] angle between light direction and [the] surface normal" (Ren Ng). Irradiance is "the power per unit area incident at a surface point," so the intensity of light at a given surface is proportional to the cosine of the angle between the surface normal of a point and the 3D vector that holds the direction of where a light source is placed in a 3D scene (Ren Ng). When the direction of the light and the surface normal are normalized, the cosine of their angle can be computed using a dot product. When the dot product is 1, the light direction vector and surface normal are pointing in the same direction, so the light is shining directly onto the fragment. When the cosine of the angle is 0, the light direction vector and the surface normal are perpendicular to each other, so there is little to no light shining on the fragment from the light source.

Toon/Cel Fragment Shader
The Toon/Cel fragment shader computes the color for each fragment, usually a pixel, in a model. For a toon/cel shader, an image is shaded using a small handful of colors so that the image appears "flat" or drawn. Thus, shadows and lights are represented with only a few colors instead of a gradient of colors (Cel Shading). I followed an example from Lighthouse3D to build my toon/cel fragment shader that uses a "light intensity" at each fragment to assign colors (Lighthouse3D). The "light intensity" is Lambert's Cosine Law, which uses the world space surface normal from the vertex shader and the vector direction of the 3D scene's light source to compute how intense the light is at a given fragment on an object's surface.

I picked four shades of green ranging from light to dark and set up a conditional statement that chooses the color for a fragment based on its intensity value like in the Lighthouse3D toon shader example (Yoda Color Scheme). I tweaked the intensity intervals so that the range (0.8, 1.0] would be the brightest color of green because these fragments have high intensity and are directly illuminated by the light source. I then chose a lighter green color for intensities in the range (0.55, 0.8], a forest green for (0.32, 0.55], and dark green for the shadows in the [0.0, 0.32] interval. The fragment shader will assign the color for the fragment in the gl_FragColor variable, which will be used when rendering the 3D model to the screen.

Noir Fragment Shader and PRNG Function
My second fragment shader is a Noir shader that has a "dark and mysterious" color theme. I based my color scheme on the Komikaze Toon Shader Pack for Blender, which has a grayscale color scheme except for one coral color that is a rim for the very lowest light intensities (Komikaze). The rim gives a backlight effect that adds a pop of color to the model. An interesting part of this shader is that it has a grain texture for the non-black colors. To recreate this, I used a pseudo-random number generator (PRNG) function from The Book of Shaders by Patricio Gonzalez Vivo and Jen Lowe (Vivo and Lowe).

The "2D Random" PRNG from The Book of Shaders goes from a 2D coordinate to a floating point number between 0 and 1. The function reduces the coordinate from 2D to 1D with a dot product, and then passes the dot product to a sine function. To break up the evaluated sine wave into smaller pieces, the sine wave is multiplied by a large number. With a sufficiently high number, the fractional component of the sine wave loses its wave shape and turns into a pattern that looks like "TV noise," shown above (Vivo and Lowe). I adapted this function to a 3D case by using a 3D vector in the dot product instead of 2D. The numbers in the 3D vector are somewhat arbitrarily chosen and change the pattern of the noise (kind of like a seed for a random function), but I went with the vector values found in The Book of Shaders and the article "From random number to texture - GLSL noise functions" by Thorsten Renk (Thorsten Renk). This PRNG allowed me to use the 3D coordinates of the fragment to get a random number than I could use to determine whether a fragment should be assigned the background color or the noise color (a lighter color) for a given interval of light intensity.

The Noir fragment shader does the same light intensity calculation with Lambert's Cosine Law like the toon/cel shader, but also incorporates the PRNG into the final decision of whether a fragment should be a noise color or a background color. Instead of using a 50/50 decider for whether a fragment should be a noise color, I chose 0.6, 0.2, and 0.3 as the probability of being a noise color for different intervals to make the noise more concentrated or sparse.

"Hope" Fragment Shader and Halftone Shading
The "Hope" poster from Barack Obama's 2008 presidential campaign has a red, blue, and tan color scheme that can be represented well using a normal toon/cel shader ("Hope" Color Scheme). However, the original poster has one interval of color that has a tan and light blue stripes design. I wanted to recreate this effect, but I chose to use halftone shading instead of stripes (because stripes proved a bit too tricky). Halftone shading "simulates continuous-tone imagery through the use of dots, varying either in size or in spacing, thus generating a gradient-like effect" (Wikipedia: Halftone). For example, instead of having a normal gradient shadow, a shadow in a halftone image would be represented by a grid of black dots.

For my halftone shader, I followed a halftone shader tutorial by Stefan Gustavson (Gustavson). I decided to leave out the parts of his shader than change the diameter of the dots depending on the reflectance of the underlying texture because the original "Hope" poster has uniform stripes and I wanted my dots to be uniform as well. The fragment shader computes the distance from a fragment's UV coordinates (in texture space) to the nearest point in a grid of dots across a unit square. The grid contains a frequency * frequency number of dots, and I chose 120 dots as my frequency.

The distance is then passed into a linearly interpolated step function called smoothstep, a built-in GLSL function, to determine whether the current fragment's distance to the center of the nearest dot is inside the dot's radius, outside the dot, or somewhere on the boundary of the dot. If the fragment is inside the dot radius, smoothstep returns 0; if it's outside the radius, smoothstep returns 1. If the distance is on the boundary of the radius, meaning within the radius plus and minus the fwidth of the distance to the nearest dot, smoothstep uses Hermite interpolation to get a value between 0 and 1. The interpolation is done to antialias the edges of the dots so that the edges have a gradual fade and not an abrupt cutoff that looks jagged. The fwidth returns the "sum of the absolute value of derivatives in x and y" for the passed in expression (Khronos). This means that for a given expression, the fwidth calculates the total rate of change in the pixel one over and the pixel one above the current pixel (StackExchange). For halftone shading, the fwidth is used to check how the distance changes in neighboring fragments so the fragments on the border of a dot can be assigned intermediate color values and soften the edges of the dots.

To assign the fragment color, the output of the smoothstep function is passed into a built-in mix function that linearly interpolates between two RGB colors. Using the halftone shader, the second to the highest intensity interval of the "Hope" Shader has dots that add dimension. The rest of the intervals all work similarly to the toon/cel shader using Lambert's Cosine Law.

Three.js Real Time Rendering
After implementing my shaders, I used Three.js to render my models. I followed an example from Three.js Fundamentals to setup up a canvas in an html file that would hold the rendered scene with a camera, directional and ambient light, and of course my shaded 3D models (Three.js Fundamentals and OBJLoader Setup). Three.js allowed me to create a ShaderMaterial that would render the surface of a given 3D model, called a geometry in Three.js, using my shaders in real-time (Vic Sidious).

Problems Encountered

Aliasing
I encountered a problem in all three shaders where there was aliasing, which "is an effect that causes different signals to become indistinguishable (or aliases of one another) when sampled," between the changes in colors in the final rendering (Wikipedia: Aliasing). At high frequencies in the rendering, meaning places where there are abrupt changes in the colors of the 3D model, the transition between colors had jaggies. Jaggies are staircase-like artifacts that make the transition between shapes and colors look pixelated and not continuous. The photos below show the aliasing in the "Hope" shader and the Noir shader, and the jaggies are especially visible on Baby Yoda's eyes.

Baby Yoda model with hope shader and aliasing

Baby Yoda model with noir shader and aliasing

To antialias my renders, I used the built in GLSL "mix" function to linearly interpolate between two colors at high frequencies in the rendering, so that the transitions between changes in color would have a gradual change over a gradient and reduce the jaggies. The mix function takes in two colors a and b, the "edge" colors, and a value x to use for where "in-between" the two colors to interpolate. True to its name, mix then produces a new color that is a mix of the edge colors using the equation (1 - x) * color a + x * color b. I used this mix function when the light intensity of a fragment was close to the cutoff boundary for a given color interval. This fix led to somewhat successful antialiasing, but the high frequency transitions are not as smooth as I would have liked.

I also followed an example from Three.js Fundamentals on responsive design in order to dynamically resize the rendering canvas to fit my screen (Three.js Responsive Canvas). This maintains the aspect ratio of the rendered scene and also increases the resolution of the canvas for bigger screen sizes. This helped to antialias my images because higher resolutions lead to less jaggies.

Screen Space UV Coordinates
I found out the hard way that .OBJ files do not have UV texture coordinates. This was a problem because my halftone shading in the "Hope" shader relies on UV coordinates to calculate the distance to the nearest point in a dot grid. In order to apply the "Hope" Shader to the Baby Yoda .OBJ model, I had to calculate UV coordinates using 3D screen space coordinates. I modified my vertex shader to include a varying variable, varying meaning a variable created in the vertex shader that's used in the fragment shader, that holds the screen space coordinates of a given vertex, which is just the vertex's model space coordinates transformed using the model view matrix and projection matrix. In the fragment shader, I divided the x and y components of the screen space position of a fragment by the w component, which is the homogeneous coordinate. The division by w is to ensure that the perspective of the model is not skewed when applying 2D texture coordinates, or in the case of a halftone shader, the grid of dots, in 3D.