Get custom displacementMap (RGB encoded) to work with shadows in r132

Questions
,
1

I need to make a follow-up related to the previous posts regarding a custom displacement map.

My displacement map is an RGB encoded elevation texture and I’m offsetting a plane in a custom vertex shader (based on THREE.ShaderLib.phong). The approach was pretty straight forward, I replaced the default texture2D call from:

transformed += normalize(objectNormal) * (texture2D(displacementMap, vUv).x * displacementScale + displacementBias);

to a custom texture2D lookup that considers each RGB encoded value:
… dot(texture2D(displacementMap, vUv).rgb, unpackFactors) …
where unpackFactors = new Vector3(256.0 * 255.0, 255.0, 255.0 / 256.0);

To also have this vertex displacement affecting self-shadowing, I have a customDepthMaterial active where in its vertex shader, I’m doing the very same: I.e., change the displacementMap lookup to the one above.

While displacing the mesh works like a charm, the customDepthMaterial doesn’t (since r132). One would think that applying the same logic to the depth material should bring correct shadows but that’s not the case.

The customDepthMaterial seems to have no effect anymore, there is no self-shadowing on the displaced mesh. I posted here already (see my intro), then hoped to get this to work with r132’s capability to do correct self-shadowing on displacementMap meshes on its own. I got this to work with regular R-channel encoded displacement maps (thanks @Mugen87!).

However, as explained, I need to adjust the texture2D lookup in the vertex shader to decode the elevation – but this kills the self-shadowing: As soon as I do just a little more magic than merely using the R channel of the texture, shadows are not working anymore. It seems that there are some optimizations that I don’t understand. I’ve spent hours already digging through the repository but the core logic seems to be beyond me.

The documentation of Object3D says that as soon as vertices are adjusted in the vertex shader, one needs a customDepthMaterial.

I’m also open to new strategies and suggestions!

tl;dr: I have an RGB encoded displacementMap. Since elevation is not only in the R channel, I can’t just use the available displacementMap attribute of MeshPhongMaterial (which works perfect including self-shadowing!), but need a custom shader which decodes the RGB channels and displaces the vertices. While the displacing of the vertices works, self shadowing (using a customDepthMaterial) does not. It used to work until r131.

Thank you for help, I’m almost in despair :upside_down_face: