Use the value of each pixel of a gray texture to turn another colorful texture with same size into a 3D mesh.

Questions
, ,
1

Currently, I have two textures of the same size loaded onto two canvases, one is an RGB image (rgbTexture), and the other is a depth image (grayscale) (depthTexture). In the three.js scene, I want to create a 3D mesh where the color comes from the RGB texture, and the depth (or the z displacement of each vertex) comes from the depth texture. Below are some of my current ideas, but it’s not working. Could anyone provide any suggestions? Thank you!

geometry = new THREE.BufferGeometry();

const uniforms = {
iChannel0: {
type: “t”,
value: rgbTexture,
},
iChannel1: {
type: “t”,
value: depthTexture,
},
uTextureProjectionMatrix: {
type: “m4”,
value: camera.projectionMatrix,
},
};

const shaderMaterial = new THREE.ShaderMaterial({
uniforms: uniforms,
overdraw: true,
vertexShader: vertShader,
fragmentShader: fragShader,
transparent: false,
});

const mesh = new THREE.Mesh(geometry, shaderMaterial);

vertShader = `
uniform sampler2D iChannel1;
precision highp float;
varying vec2 vUv;
float displacement;

void main() {
vUv = uv;
displacement = texture2D(iChannel1, uv).r.
vec3 newPosition = position;
newPosition.z += displacement;
gl_Position = projectionMatrix * mvPosition * modelViewMatrix * vec4(newPosition, 1.0);
}`;

fragShader = `
precision highp float;

uniform sampler2D iChannel0;
uniform mat4 uTextureProjectionMatrix;

varying vec2 vUv;

out vec4 fragColor;

void main()
{
vec3 col = texture(iChannel0, vUv).rgb;
fragColor = vec4(col, 1.0);
}`

2

I hope that someone with experience with shaders may provide more successful suggestion.

My suggestion is first to try without shaders. And only if the built-in functionality in Three.js is not sufficient, then to go down to shader stuff.

For your case, have you tried to use the color texture as map of the material, and the depth texture as displacementMap? If the 3D-ness is not so big, you might use bumpMap. Note that the use of displacementMap requires sufficiently dense net of vertices, while bumpMap is happy with just a few of them.