Calculating vertex normals after displacement in the vertex shader

Hey community,
I tought I’d share the solution to calculating smooth shaded normals in the shader since it took some asking around to find the solution. I did not see any three.js implementation of this.

So, if you’re displacing the vertices in the vertex shader (with some noise in my case), you will need to recalculate also the normals in the shader if you want to use lighting.

You can use the derivatives dFdx and dFdy to calculate flat shading in the fragment shader.

However if you want to use smooth shading, it is a bit harder.

The solution came from this old article and a bit from this example and this other example from spite.

Basically, you can calculate the normal by the cross product of the tangent and bitangent.

And you can calculate the tangent and bitangent by calling the displacement function with the neightbour position.

This is what it boils down to:

vec3 tangent = neighbour1 - position;
vec3 bitangent = neighbour2 - position;

vec3 normal = normalize(cross(tangent, bitangent));

If instead you’re using a texture to do the displacement, you can sample the neighbour texel. This is shown here.

I did a minimal codepen example of this:

11 Likes

its-so-shiny_fb_1017995

Are the scrolling seams a result of a scrolling noise texture or an artefact from the calculations :thinking: ?

Not sure if they are always there, but can see some at everything set to 0.5:
Screen Shot 2020-07-11 at 00.20.03

@mjurczyk they’re artifacts caused by the simplex noise function. They disappear if you use sin() as displacement for example.

I don’t know why that happens ¯\_(ツ)_/¯

2 Likes

I’m trying to make this work by default when using material.displacementMap.

Unfortunately the solution presented here only works with a plane.

How to make it so it works with any geometry?

This is what I have so far:

vec2 texelSize = vec2( 1.0 / 512.0, 1.0 / 512.0 ); // temporarily hardcoding texture resolution

float dx = texture2D( displacementMap, vUv + vec2( - texelSize.x, 0 ) ).x - texture2D( displacementMap, vUv + vec2( texelSize.x, 0 ) ).x;
float dy = texture2D( displacementMap, vUv + vec2( 0, - texelSize.y ) ).x - texture2D( displacementMap, vUv + vec2( 0, texelSize.y ) ).x;

vec3 tangent = cross( objectNormal, vec3( 0, dy, 1.0 ) );
vec3 bitangent = cross( objectNormal, vec3( dx, 0, 1.0 ) );

transformedNormal = cross( tangent, bitangent );

Almost…! :sweat_smile:

So seems like this is not possible in the vertex shader without also providing tangents… :thinking:

@mrdoob check out the full code in the codepen I posted, there is a function that uses the following approach to compute a possible tangent :nerd_face:

http://lolengine.net/blog/2013/09/21/picking-orthogonal-vector-combing-coconuts

I’ll paste the relevant code here for reference

// the function which defines the displacement
float displace(vec3 point) {
  return noise(vec3(point.x * frequency, point.z * frequency, time * speed)) * amplitude;
}

// http://lolengine.net/blog/2013/09/21/picking-orthogonal-vector-combing-coconuts
vec3 orthogonal(vec3 v) {
  return normalize(abs(v.x) > abs(v.z) ? vec3(-v.y, v.x, 0.0)
  : vec3(0.0, -v.z, v.y));
}

// ...

vec3 displacedPosition = position + normal * displace(position);

float offset = ${SIZE / RESOLUTION};
vec3 tangent = orthogonal(normal);
vec3 bitangent = normalize(cross(normal, tangent));
vec3 neighbour1 = position + tangent * offset;
vec3 neighbour2 = position + bitangent * offset;
vec3 displacedNeighbour1 = neighbour1 + normal * displace(neighbour1);
vec3 displacedNeighbour2 = neighbour2 + normal * displace(neighbour2);

// https://i.ya-webdesign.com/images/vector-normals-tangent-16.png
vec3 displacedTangent = displacedNeighbour1 - displacedPosition;
vec3 displacedBitangent = displacedNeighbour2 - displacedPosition;

// https://upload.wikimedia.org/wikipedia/commons/d/d2/Right_hand_rule_cross_product.svg
vec3 displacedNormal = normalize(cross(displacedTangent, displacedBitangent));

But I think that code doesn’t work with texture maps?

@mrdoob in your example, have you tried just computing the normal from the slope? (I guess it’s like flat shading in the fragment shader using derivatives)

transformedNormal = normalize( cross( dx, dy ) );

But I think that code doesn’t work with texture maps?

yeah, unfortunately it applies to world space.

In UV space it’s different, because you don’t know the relation between the UV space and the world space. For example, you could try fiddling with the the world space offset and texelSize, but it won’t work for every mesh out there

// the function which defines the displacement
float displace(vec2 vUv) {
  return texture2D(displacementMap, vUv);
}

// http://lolengine.net/blog/2013/09/21/picking-orthogonal-vector-combing-coconuts
vec3 orthogonal(vec3 v) {
  return normalize(abs(v.x) > abs(v.z) ? vec3(-v.y, v.x, 0.0)
  : vec3(0.0, -v.z, v.y));
}

// ...
vec3 displacedPosition = position + normal * displace(vUv);

float texelSize = 1.0 / 512.0; // temporarily hardcoding texture resolution
float offset = 0.01;

vec3 tangent = orthogonal(normal);
vec3 bitangent = normalize(cross(normal, tangent));
vec3 neighbour1 = position + tangent * offset;
vec3 neighbour2 = position + bitangent * offset;
// demo for now, the direction should be the same of the tangent and bitangent
vec2 neighbour1uv = vUv + vec2(-texelSize, 0);
vec2 neighbour2uv = vUv  + vec2(0, -texelSize);
vec3 displacedNeighbour1 = neighbour1 + normal * displace(neighbour1uv);
vec3 displacedNeighbour2 = neighbour2 + normal * displace(neighbour2uv);

// https://i.ya-webdesign.com/images/vector-normals-tangent-16.png
vec3 displacedTangent = displacedNeighbour1 - displacedPosition;
vec3 displacedBitangent = displacedNeighbour2 - displacedPosition;

// https://upload.wikimedia.org/wikipedia/commons/d/d2/Right_hand_rule_cross_product.svg
vec3 displacedNormal = normalize(cross(displacedTangent, displacedBitangent));
1 Like