The final step of our cel-shading effect is to draw bold outlines around our object. There are several ways to do that, and the method weâre about to explore involves drawing the object a second time in a second pass, slightly larger than the first time, and excluding the pixels that were drawn in the first pass.
Outline Effect
Weâre almost at the end of our journey with Ethan - as of the end of the fresnel lighting tutorial, he looks like this:
Itâs a fine-looking effect for sure and wouldnât look out of place in a cartoonish game. However, we sometimes want an extra bit of contract to make some characters or objects stand out; for this, weâll capture the comic book aesthetic a little with a thick, bold outline.
The methodology is simple. Weâll draw Ethan as usual in a first pass, then in a second pass we will use a vertex shader to extrude each vertex position slightly along its normal, so that the object is rendered slightly larger than normal. The fragment shader will draw the entire second pass in a single colour, which could be hard-coded into the shader or passed in Properties like a well-written shader should do. Finally, weâll have to find a way to avoid drawing over the first pass - for this, weâll use a stencil to mask out the pixels drawn in the first pass.
Our template, found in Shaders/OutlineCelShaded.shader, uses Shaders/Complete/RimCelShaded.shader as a base - the first pass is already defined and a skeleton second pass is included. Our first step will be to populate the second pass with a vertex and fragment shader.
Pass
{
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#include "UnityCG.cginc"
...
ENDCG
}
Remember the basics of writing a vertex-fragment program - the rest of our code is going to be written in place of the .... Letâs add things step by step. Weâre going to want to set the size and colour of the outline as properties. and include them in this shader after the #include.
// In Properties.
_OutlineSize("Outline Size", Float) = 0.01
_OutlineColor("Outline Color", Color) = (0, 0, 0, 1)
// In second pass.
float _OutlineSize;
float4 _OutlineColor;
We wonât need any other global variables inside our shader - not even _MainTex, as weâre setting a block colour and wonât be doing any texture sampling in the fragment shader. Now letâs consider the pipeline of variables weâll pass between shader stages. We will need to pass the vertex positions and vertex normals into the vertex shader - both will be used to calculate the final vertex positions that will be passed through the rasterisation step.
struct appdata
{
float4 vertex : POSITION;
float3 normal : NORMAL;
};
Our fragment shader is just going to draw block colours, so the only thing that needs to be output by the vertex shader are the vertex positions - the normals wonât be needed past this stage.
struct v2f
{
float4 vertex : SV_POSITION;
};
The vertex shader will read the vertex positions passed in appdata and will add the normalised normal vector multiplied by the _OutlineSize we defined in Properties to that position.
v2f vert(appdata v)
{
v2f o;
float3 normal = normalize(v.normal) * _OutlineSize;
float3 pos = v.vertex + normal;
o.vertex = UnityObjectToClipPos(pos);
return o;
}
Itâs important the normals and positions are added before the call to UnityObjectToClipPos(), because both the input normals and input positions exist in Unity object space. Next up, weâll deal with the fragment shader. All it needs to do is set the output colour to the _OutlineColor we defined in Properties.
float4 frag(v2f i) : SV_Target
{
return _OutlineColor;
}
If we run the shader now, after setting the _OutlineSize to something like 0.01 and the _OutlineColour to black, the result is something like this:
Somethingâs not right. If you look very closely, youâll notice the entirety of Ethan is shaded with the _OutlineColour, which might not be what you wanted. Letâs fix it by introducing a powerful new tool: stencils.
Stencil Buffer
A stencil buffer is a way of remembering a section of pixels so that we can modify only a specific section of an image or discard a subset of pixels at a later stage. In our example, weâre interested in the latter case; we want to draw Ethan as usual, remember which pixels he was drawn on, then draw Ethanâs outline on top, but discard the pixels where Ethan was already drawn. We can do this by writing a stencil when drawing Ethan the first time, then reading that stencil during the outline pass.
The syntax for stencils is straightforward and ShaderLab holds our hand for the most part. A stencil runs over the whole image alongside the rest of the Pass, so each of the following stages happen on a per-pixel basis. At the top of the first Pass, between our Tags and CGPROGRAM block, we define a Stencil.
Stencil
{
...
}
To link two stencils together, we use a reference value, Ref. Think of a stencil like a bit mask - the stencil buffer is another screen-sized 2D array where each pixel has its own reference value. When writing to the stencil buffer, weâre setting the reference value. Note that defining a reference value doesnât mean it will get written to the stencil buffer - we must specify this using another keyword.
// Inside the Stencil{} block.
Ref 1
Next up, we need a comparison function - a stencil test, Comp. This is used when weâre reading the stencil to decide whether we want to render a pixel. In the first pass, we always want to render the pixel no matter what the value in the stencil buffer is - so we use the always comparison function.
Comp always
Then, we define what stencil behaviour should occur if the stencil test and depth test both succeed, using Pass. We want to replace the previous contents of the stencil buffer with the reference value. The default value is keep, which wonât write to the stencil buffer, so instead weâll use replace.
Pass replace
Finally, weâll define what happens to the stencil buffer value if the stencil test fails, using the Fail keyword. The default behaviour is to keep the reference value inside the stencil buffer. The same is true of ZFail when the stencil test passed but the depth test fails; the default behaviour is to keep. Weâll keep both defaults.
Fail keep
ZFail keep
For a bit more information on the parameters available for a stencil buffer, check the official Unity documentation. Of course, this wonât have any effect just yet, since weâre only writing to the stencil buffer in the first pass. We must now read from it in the second pass.
Before writing the next stencil, I added a couple âsanity checksâ to make sure Z-testing works as intended. I donât want the second pass to write to the depth buffer, but I do want it to read from the buffer and perform the depth test.
Pass
{
ZWrite off
ZTest on
}
Now weâre going to define a second stencil, at the top of the second pass, with a reference of 1 - the same reference value as the first stencil.
Pass
{
...
Stencil
{
Ref 1
}
}
Now our two stencils are linked by reference. Weâll only want to write to the framebuffer when the stencil reference value is not equal to 1, so weâll use the notequal comparison function.
Comp notequal
We can modify Pass or Fail parameters if we wish, but the defaults are also fine. Now, when we run our shader, we should have the outline effect weâve been aiming for!
Conclusion
Weâve completed our cel-shading effect! Now weâve added an outline to the effect, with configurable width and colour, learning how to use the stencil buffer on the way. In the final part of this series on Cel Shading, weâll put the finishing touches on the effect and recap the shader features weâve seen throughout the series.