Reverse engineering the rendering of The Witcher 3, part 6 - sharpen

This post is a part of the series "Reverse engineering the rendering of The Witcher 3".


Hi,

Today we will take a closer look at another postprocess from The Witcher 3 - sharpen.
Sharpening makes an output image a bit crisper. The effect is known from Photoshop and other image editors.

In The Witcher 3 sharpening has two presets: low and high. I will discuss differences between them later, let's take a look at some screenshots now:

"Low" setting - before

"Low" setting - after

"High" setting - before

"High" setting - after

If you want to see more (interactive) comparisons, see section in Nvidia's performance guide for The Witcher 3. As you can see, the effect is particularly visible on grass and foliage.

In this post we will investigate frame from the very beginning of the game: I selected this one purposefully, because here we see terrain (long draw distance) and skydome.

In terms of input, sharpening requires color buffer t0 (LDR, after tonemapping and lens flares) and depth buffer t1.

Let's see the pixel shader, assembly:

 ps_5_0  
    dcl_globalFlags refactoringAllowed  
    dcl_constantbuffer cb3[3], immediateIndexed  
    dcl_constantbuffer cb12[23], immediateIndexed  
    dcl_sampler s0, mode_default  
    dcl_resource_texture2d (float,float,float,float) t0  
    dcl_resource_texture2d (float,float,float,float) t1  
    dcl_input_ps_siv v0.xy, position  
    dcl_output o0.xyzw  
    dcl_temps 7  
   0: ftoi r0.xy, v0.xyxx  
   1: mov r0.zw, l(0, 0, 0, 0)  
   2: ld_indexable(texture2d)(float,float,float,float) r0.x, r0.xyzw, t1.xyzw  
   3: mad r0.x, r0.x, cb12[22].x, cb12[22].y  
   4: mad r0.y, r0.x, cb12[21].x, cb12[21].y  
   5: max r0.y, r0.y, l(0.000100)  
   6: div r0.y, l(1.000000, 1.000000, 1.000000, 1.000000), r0.y  
   7: mad_sat r0.y, r0.y, cb3[1].z, cb3[1].w  
   8: add r0.z, -cb3[1].x, cb3[1].y  
   9: mad r0.y, r0.y, r0.z, cb3[1].x  
  10: add r0.y, r0.y, l(1.000000)  
  11: ge r0.x, r0.x, l(1.000000)  
  12: movc r0.x, r0.x, l(0), l(1.000000)  
  13: mul r0.z, r0.x, r0.y  
  14: round_z r1.xy, v0.xyxx  
  15: add r1.xy, r1.xyxx, l(0.500000, 0.500000, 0.000000, 0.000000)  
  16: div r1.xy, r1.xyxx, cb3[0].zwzz  
  17: sample_l(texture2d)(float,float,float,float) r2.xyz, r1.xyxx, t0.xyzw, s0, l(0)  
  18: lt r0.z, l(0), r0.z  
  19: if_nz r0.z  
  20:  div r3.xy, l(0.500000, 0.500000, 0.000000, 0.000000), cb3[0].zwzz  
  21:  add r0.zw, r1.xxxy, -r3.xxxy  
  22:  sample_l(texture2d)(float,float,float,float) r4.xyz, r0.zwzz, t0.xyzw, s0, l(0)  
  23:  mov r3.zw, -r3.xxxy  
  24:  add r5.xyzw, r1.xyxy, r3.zyxw  
  25:  sample_l(texture2d)(float,float,float,float) r6.xyz, r5.xyxx, t0.xyzw, s0, l(0)  
  26:  add r4.xyz, r4.xyzx, r6.xyzx  
  27:  sample_l(texture2d)(float,float,float,float) r5.xyz, r5.zwzz, t0.xyzw, s0, l(0)  
  28:  add r4.xyz, r4.xyzx, r5.xyzx  
  29:  add r0.zw, r1.xxxy, r3.xxxy  
  30:  sample_l(texture2d)(float,float,float,float) r1.xyz, r0.zwzz, t0.xyzw, s0, l(0)  
  31:  add r1.xyz, r1.xyzx, r4.xyzx  
  32:  mul r3.xyz, r1.xyzx, l(0.250000, 0.250000, 0.250000, 0.000000)  
  33:  mad r1.xyz, -r1.xyzx, l(0.250000, 0.250000, 0.250000, 0.000000), r2.xyzx  
  34:  max r0.z, abs(r1.z), abs(r1.y)  
  35:  max r0.z, r0.z, abs(r1.x)  
  36:  mad_sat r0.z, r0.z, cb3[2].x, cb3[2].y  
  37:  mad r0.x, r0.y, r0.x, l(-1.000000)  
  38:  mad r0.x, r0.z, r0.x, l(1.000000)  
  39:  dp3 r0.y, l(0.212600, 0.715200, 0.072200, 0.000000), r2.xyzx  
  40:  dp3 r0.z, l(0.212600, 0.715200, 0.072200, 0.000000), r3.xyzx  
  41:  max r0.w, r0.y, l(0.000100)  
  42:  div r1.xyz, r2.xyzx, r0.wwww  
  43:  add r0.y, -r0.z, r0.y  
  44:  mad r0.x, r0.x, r0.y, r0.z  
  45:  max r0.x, r0.x, l(0)  
  46:  mul r2.xyz, r0.xxxx, r1.xyzx  
  47: endif  
  48: mov o0.xyz, r2.xyzx  
  49: mov o0.w, l(1.000000)  
  50: ret  

50 lines of assembly seems like pretty doable task. Let's start it then.

Sharpen amount generation

The first step is to Load depth buffer (line 1). Note that The Witcher 3 uses revesed depth (1.0 - near, 0.0 - far). As you may know, hardware depth is mapped in non-linear way (see this article for details).

Lines 3-6 perform very interesting way of mapping this hardware depth [1.0 - 0.0] to [near-far] values (you set them during MatrixPerspectiveFov). See values from constant buffer:

Having near value of 0.2 and far of 5000 I believe you can calculate values of cb12_v21.xy this way:

cb12_v21.y = 1.0 / near
cb12_v21.x = - (1.0 / near) + (1.0 / near) * (near / far)

This piece of code appears quite often in shaders from TW3, so I believe it's just a function.

When we already have "frustum depth", line 7 uses scale/bias to create a interpolation coefficient (we use saturate here to make sure it's clamped to [0-1] range).

cb3_v1.xy are intensities of sharpening at near and far distances, respectively. Let's call them "sharpenNear" and "sharpenFar". And this is the only difference between "Low" and "High" presets of this effect in The Witcher 3.

Now it's time to use the obtained coefficient. Lines 8-9 are just lerp(sharpenNear, sharpenFar, interpolationCoeff). What is this for? Thanks to that we can have different intensity near Geralt and far away from him). See:

It may be barely visible, but here we interpolated the intensity of sharpen near the player (2.177151) with intensity of the effect far away (1.91303) based on distance. Once we have calculated it we add 1.0 (line 10) to intensity. What is this for? Let's assume that lerp from above gave us 0.0. When we add 1.0 we will have 1.0 of course and this is value which will not affect the pixel during sharpening. More on this later.

During sharpening process we don't want to affect sky. We can achieve this using simple conditional test:

   // Do not perform sharpen on sky  
   float fSkyboxTest = (fDepth >= 1.0) ? 0 : 1;  

In The Witcher 3 depth value for sky pixels is 1.0, so we use it to get some sort of "binary filter" (fun fact: step does not work properly in this case)
Now we can multiply interpolated intenstiy with "sky filter":

This multiplication takes place in line 13.
Example shader code:

   // Calculate final sharpen amount  
   float fSharpenAmount = fSharpenIntensity * fSkyboxTest;  

Sampling center of the pixel

There is an aspect of SV_Position which will be important here: half-pixel offset. It turns out that pixel at top left corner (0, 0) is not (0, 0) in terms of SV_Position.xy, but (0.5, 0.5). Wow!

Here we want to sample in center of the pixel, so take a look at lines 14-16. We can write it in HLSL:

   // Sample the center of the pixel.   
   // Get rid of "half-pixel" offset from SV_Position.xy.  
   float2 uvCenter = trunc( Input.Position.xy );  

   // Add half-pixel to make sure we will sample the center of the pixel  
   uvCenter += float2(0.5, 0.5);  
   uvCenter /= g_Viewport.xy  

And later we sample input color texture from "uvCenter" texcoords. Don't worry, the effect of the sampling will be the same as using "typical" (SV_Position.xy / ViewportSize.xy).

To sharpen or not to sharpen

The decision whether to sharpen or not is based on fSharpenAmount.

   // Get the value of current pixel  
   float3 colorCenter = TexColorBuffer.SampleLevel( samplerLinearClamp, uvCenter, 0 ).rgb;  
     
   // Final result  
   float3 finalColor = colorCenter;  
   
   if ( fSharpenAmount > 0 )  
   {  
     // do the sharpening here...  
   }  
   
  return float4( finalColor, 1 );  
   

Sharpen

It's time to look at the heart of the algorithm.

Basically:
- sample the input color texture four times at the corners of the pixel,
- add the samples and calculate average value,

- calcuate the difference between "center" and "cornerAverage",

- find maximum absolute component of the difference,

- adjust max. abs. component using scale+biasvalues,

- determine amount of the effect using max. abs. component,

- calculate luma of "centerColor" and "averageColor",
- divide the colorCenter by its luma,

- caclulate the new, interpolated luma using amount of the effect,

- multiply the colorCenter by the new luma

Seems like lots of things and it was a challenge for me to understand it, since I've never played with sharpening filters. 

Let's start with sampling pattern. As you can see in the assembly, there are four texture fetches.
It will be best to show it using this image of pixel (Paint level expert):

All fetches in the shader use bilinear sampling (D3D11_FILTER_MIN_MAG_LINEAR_MIP_POINT).

The offset from center to any corner is (±0.5, ±0.5), depending on corner.

See how this can be done in HLSL? Let's see:

    float2 uvCorner;  
    float2 uvOffset = float2( 0.5, 0.5 ) / g_Viewport.xy;  // remember about division!
    
    float3 colorCorners = 0;  
             
    // Top left corner  
    // -0,5, -0.5  
    uvCorner = uvCenter - uvOffset;  
    colorCorners += TexColorBuffer.SampleLevel( samplerLinearClamp, uvCorner, 0 ).rgb;  
   
    // Top right corner  
    // +0.5, -0.5  
    uvCorner = uvCenter + float2(uvOffset.x, -uvOffset.y);  
    colorCorners += TexColorBuffer.SampleLevel( samplerLinearClamp, uvCorner, 0 ).rgb;  
   
    // Bottom left corner  
    // -0.5, +0.5  
    uvCorner = uvCenter + float2(-uvOffset.x, uvOffset.y);  
    colorCorners += TexColorBuffer.SampleLevel( samplerLinearClamp, uvCorner, 0 ).rgb;  
   
    // Bottom right corner  
    // +0.5, +0.5  
    uvCorner = uvCenter + uvOffset;  
    colorCorners += TexColorBuffer.SampleLevel( samplerLinearClamp, uvCorner, 0 ).rgb;  

So now we have all four samples summed in "colorCorners" variable. Let's perform the next steps:

   // Calculate the average of four corners  
   float3 averageColorCorners = colorCorners / 4.0;  
   
   // Calculate the color difference  
   float3 diffColor = colorCenter - averageColorCorners;  
   
   // Find max absolute RGB component of the difference  
   float fDiffColorMaxComponent = max( abs(diffColor.x), max( abs(diffColor.y), abs(diffColor.z) ) );  
   
   // Adjust this factor  
   float fDiffColorMaxComponentScaled = saturate( fDiffColorMaxComponent * sharpenLumScale + sharpenLumBias );  
   
   // Calculate how much pixel will be sharpened.  
   // Note the "1.0" here - this is why we added "1.0" before to fSharpenIntensity.  
   float fPixelSharpenAmount = lerp(1.0, fSharpenAmount, fDiffColorMaxComponentScaled);  
    
   // Calculate luminance of "center" of the pixel and luminance of average value.  
   float lumaCenter = dot( LUMINANCE_RGB, finalColor );  
   float lumaCornersAverage = dot( LUMINANCE_RGB, averageColorCorners );  
       
   // divide "centerColor" by its luma  
   float3 fColorBalanced = colorCenter / max( lumaCenter, 1e-4 );  
     
   // Calc the new luma  
   float fPixelLuminance = lerp(lumaCornersAverage, lumaCenter, fPixelSharpenAmount);  
       
   // Calc the output color  
   finalColor = fColorBalanced * max(fPixelLuminance, 0.0);  
}

return float4(finalColor, 1.0);

The edge detection is done by calculating max. abs. component of the difference. Smart! See its visualization:

Visualization of maximum absolute component of the difference.

Phew. The final HLSL shader is available here. Sorry for quite poor formatting. Feel free to use my HLSLexplorer and play with the code.

I am happy to say that the code above gives exactly the same assembly as in the game! :)

To sum up, The Witcher 3's sharpening shader is very well written (notice that fPixelSharpenAmount is larger than 1.0! that is interesting...). Also, the primary way to modify intensity of the effect are near/far intensities. In the game, they are not constant throughout the gamplay; I collected some example values:

Skellige:

	sharpenNear	sharpenFar	sharpenDistanceScale	sharpenDistanceBias	sharpenLumScale	sharpenLumBias
low	0.40	0.20	0.025	-0.25	-13.33333	1.33333
high	2.0	1.8	0.025	-0.25	-13.33333	1.33333

Kaer Morhen:

	sharpenNear	sharpenFar	sharpenDistanceScale	sharpenDistanceBias	sharpenLumScale	sharpenLumBias
low	0.57751	0.31303	0.06665	-0.33256	-1.0	2.0
high	2.17751	1.91303	0.06665	-0.33256	-1.0	2.0

That's it for today. I hope you enjoyed it :)
Thanks for reading!

M.

Few words about HLSLexplorer

Welcome,

HLSLexplorer started as a hobby project (which still is!) and was never meant to be "Godbolt for shaders" nor any "competition" for already available solutions (see Shader Playground by Tim Jones for example). I just realised during its development that someone might find it useful - that's all.

Today I am happy to say that new version of HLSLexplorer is done and now it's open source!
Let's go through the most important changes.

HLSLexplorer 1.0 in action

Support for modern DirectX compiler and AMD GCN ISA
There are three tabs on the right window. DXBC (d3dcompiler_47.dll for Shader Model 4.0-5.1) and new ones: DXIL (for modern DirectX shader compiler) and AMD GCN ISA

Real-time Pixel Shader Preview
I thought it would be nice to have an option to visualize the result of pixel shader in real time, so I made a tool to do it.

If you click Insert -> Insert simple PS, a dummy pixel shader appear on the left.
Press F7 to start pixel shader preview window:

The left window is rendering output. The right panel is a place to load textures. You can load dds textures, also png/jpg/bmp are supported. Load any texture to texture0 channel... Huh, no result. To see it, switch back to main window, tap F5 to compile shader, and you should see the output in preview window:

Let's go back to dummy pixel shader. There are some differences comparing to previous version of the application:

 cbuffer cbData : register (b12)  
 {  
     float elapsedTime;  
     uint  numFrames;  
     float2 pad;  
   
     float2 viewportSize;  
     float2 viewportInvSize;  
 }  
   
 SamplerState samplerPointClamp  : register (s0);  
 SamplerState samplerPointWrap   : register (s1);  
 SamplerState samplerLinearClamp : register (s2);  
 SamplerState samplerLinearWrap  : register (s3);  
 SamplerState samplerAnisoClamp  : register (s4);  
 SamplerState samplerAnisoWrap   : register (s5);  

Now you have access to some values which are usually available in typical scenarios, like elapsed time (in seconds), viewport / invViewport size and various samplers, you can use them in your pixel shaders and see the effect without closing HLSLexplorer. Pressing F5 updates shader in preview (assuming there is no errors of course). Please note that this feature currently works only with Shader Model 4.0 - 5.0.

HLSLexplorer goes open source
Source code is available on GitHub. If you look just for binary release, click here (Google Drive).

If you would like to build it by yourself:
HLSLexplorer is linked statically against wxWidgets 3.1.1. The compilation of wxWidgets for debug/release configurations should be easy, but please note that I use environmental variable (WXWIN) in HLSLexplorer's solution configuration (additional library directories).

Other improvements worth mentioning
- The program asks if user wants to save HLSL source from the left window when user wants to close the application,
- User can load external hlsl files into the program
- User can save disassembled shaders (DXBC, DXIL, AMD GCN ISA) to hard drive. Just switch to proper window and tap Ctrl+S. The title of saving window tells what you attempt to save.
- Slight UI tweaks
- Improved "About" window ;)

I hope you like it. Go now and disassemble some shaders! :)

Reverse engineering the rendering of The Witcher 3: Index

Welcome,

This is the index page for my "Reverse engineering the rendering of The Witcher 3" small series where I select some rendering techniques from "The Witcher 3", analyze them using RenderDoc, then grab DirectX assembly for shaders and finally try to turn these instructions back to readable HLSL shader.
I focus especially on understanding and explaining presented techniques; it's all for learning purposes after all.

The genesis of the series is simply "I want to know how it's done". Having RenderDoc and other tools which allow us to see every stage of the pipeline is great way to learn how games do rendering stuff.

I have not worked on "The Witcher 3: Wild Hunt" nor its expansion packs in any way so all of code and conclusions here are results of my analysis and tinkering.

Disclaimer: All opinions and views expressed on my blog and on this series are my own and are in no way representative of Rockstar Games.

Here is the current index:

Part 1 - tonemapping

Part 2 - eye adaptation

Part 3 - chromatic aberration

Part 4 - vignette

Part 5 - drunk effect

Part 6 - sharpen

Part 7a - average luminance, part 1 (histogram of luminance)

Part 7b - average luminance, part 2 (calculation)

Part 8 - the Moon and lunar phases

Part 9 - GBuffer

Part 10 - distant rain shafts

Part 11 - lightnings

Part 12 - stupid sky tricks

Part 13a - witcher senses, part 1 (objects & intensity map)

Part 13b - witcher senses, part 2 (outline map)

Part 13c - witcher senses, part 3 (fisheye effect & final combining)

Part 14 - cirrus clouds

Part 15 - fog

Part 16 - shooting stars

Part 17 - the Milky Way

Part 18 - color grading

Part 19 - portals

Part 20 - light shafts

Part 21 - the painted world

Extra:

Someone has done awesome job and prepared Russian version of my posts!
Translation into Russian (parts 1-5)
Translation into Russian (parts 6-10)
Translation into Russian (parts 11-13)

I'm the author of HLSLexplorer which greatly helped me in the process of learning HLSL assembly and was invaluable in writing this series. See the post for details, source code and binaries.


Enjoy! :)