Use GPUComputationRenderer texture for dynamic InstancedMesh objects positions

Hi!

I have a dynamic InstancedMesh that represents a grid of objects, 128x128 objects lets say.

I have a proxy bidimensional array to keep track of the objects positions, 128 rows by 128 columns, so I can modify the positions of any object in the main render loop, apply noise, apply tweens, etc.

I also wanted good wave interaction so I took the water simulation using GPUComputationRenderer from the examples, three.js examples, and I have a 128x128 texture computing very nice waves simulation, I draw the texture into a plane so I can see it in realtime, and it works perfect with flawless performance.

The question is: How can I get the pixel color values of every pixel in that GPU texture as it was a bitmap/bytearray so I can use that information in the render loop to use it to modify the ‘z’ position (height) of each of the objects in the grid?

Thanks a lot!!!

I’m attaching a couple images of my grid (already with a bit of Simplex noise applied) with a plane on top that shows the GPUComputationRenderer texture to illustrate my question:

This? WebGLRenderingContext.readPixels() - Web APIs | MDN

Is there a specific reason you want to access the value by javascript?

You can avoid that by passing the GPUComputationRenderer texture as a uniform to your InstancedMesh’s material, then modify that material to apply the relevant Z offset to each instance’s coordinates.

3 Likes

Thanks a lot for your help @Harold and @Arthur.

As @Arthur pointed out the best approach seems to be to use directly the texture as a height map uniform in the vertex shader, my tests with ‘.readPixels()’ sadly seems to kill the performance so I discarded that approach.

I’ve been all day trying a similiar method as the one in the water example. In that example they copy the Phong vertexshader and modify it a little to apply one of the .rgb components of the height map texture to the .z position of the vertices and normal calculation.

The problem is that in the example the vertexshader is applied to each vertex, as it’s a PlaneGeometry, but in my case I have an InstancedMesh of several instances of a somewhat complex geometry, so I can’t apply it to each vertex as the instances geometry break appart.

Reading through the Phong vertexshader I’ve seen that ‘instanceMatrix’ is used to calculate the vertex positions and normals, so intuition tells me that I should do some kind of operation to apply the height map value to the ‘instanceMatrix’ at the begining of the vertexshader, but I have no clue how.

Anyone knows how I could apply the height map, or any offset, to the ‘instanceMatrix’ in the vertexshader so I can modify the vertices in a ‘per instance’ basis?

Thanks a lot in advance :slight_smile:

Then, you are doing something wrong.
WebGL.readPixels() for 128x128 on my machine is 6 milliseconds to return ABV.

For 1280x720 takes about 130 milliseconds to do ArrayBufferView.

I can help on how to conform geometry base on height map on old THREE.Geometry pre 125.

Read the texture with heights in vertex shader and add that value to Y-component of translate values of instanceMatrix.

    shader.vertexShader = `
      uniform sampler2D heightMap;
      attribute vec2 instUV;
      varying float vHeight;
      ${shader.vertexShader}
    `.replace(
      `#include <project_vertex>`,
      `
      vec4 mvPosition = vec4( transformed, 1.0 );
      #ifdef USE_INSTANCING
        mat4 im = instanceMatrix;
        float h = texture2D(heightMap, instUV).r; // read
        vHeight = h;
        im[3].y += h * 5.; // add
        mvPosition = im * mvPosition;
      #endif
      mvPosition = modelViewMatrix * mvPosition;
      gl_Position = projectionMatrix * mvPosition;
      `
    );

Demo: https://codepen.io/prisoner849/full/zYRGBmK

PS It’s just an option, not the ultimate solution.

4 Likes

@prisoner849 your advice worked like a charm, I was trying a very similar method but I was not passing the instUVs, but thanks to your example now I have it working.

A couple new problems arise:

  • Shadows are not taking into account the modified instanceMatrix
  • How to make it work in a non square grid, how to take a ‘crop’ of the texture2D(heightMap) so it’s not stretched.

Here is an example of the shadows not working as expected:

And here is an example of how the shadows should look when they work fine when I apply a bit of noise in .z position of each instance:

What I’m doing is I’m taking a MeshStandardMaterial, and like in @prisoner849 codepen demo I modify the instanceMatrix in all the shader chunks (only 3 ocurrences of instanceMatrix in all the vertexShader chunks). I apply the heightMap to the instanceMatrix to make a z translation and use this modifed matrix in the 3 shader chunks where instanceMatrix appears.
I don’t modify the fragmentShader.

If using the CPU I do:

dummy.position.set( x , y, 0);
dummy.updateMatrix();
this.hexMesh.setMatrixAt(ii, dummy.matrix);

then the shadows are working well. I thought the modifications to the instanceMatrix at the vertexshader would result in the same outcome but I’m missing something :frowning:

I paste the material with the vertexShader modifications for reference:

this.hexMaterial = new THREE.MeshStandardMaterial({
            color: 0xffffff,
            //transparent : true,
            //opacity: 0.5
            roughness: 1,
            metalness: 0.5
            //wireframe: true
        });
        this.hexMaterial.onBeforeCompile =  function (shader) {
            shader.uniforms.heightMap = _.hexMaterialUniforms.heightMap;
            shader.vertexShader = `
                #define USE_HEIGHTMAP

                uniform sampler2D heightMap;
                attribute vec2 instUV;
                varying float vHeight;
                ${shader.vertexShader}
            `.replace(
                `#include <defaultnormal_vertex>`,
                `
                vec3 transformedNormal = objectNormal;
                #ifdef USE_INSTANCING
                    // this is in lieu of a per-instance normal-matrix
                    // shear transforms in the instance matrix are not supported
                    mat4 im = instanceMatrix;
                    #if defined( USE_HEIGHTMAP )
                        float h = texture2D(heightMap, instUV).r;
                        vHeight = h;
                        im[3].z += h * 20.;
                    #endif
                    mat3 m = mat3( im );
                    transformedNormal /= vec3( dot( m[ 0 ], m[ 0 ] ), dot( m[ 1 ], m[ 1 ] ), dot( m[ 2 ], m[ 2 ] ) );
                    transformedNormal = m * transformedNormal;
                #endif
                transformedNormal = normalMatrix * transformedNormal;
                #ifdef FLIP_SIDED
                    transformedNormal = - transformedNormal;
                #endif
                #ifdef USE_TANGENT
                    vec3 transformedTangent = ( modelViewMatrix * vec4( objectTangent, 0.0 ) ).xyz;
                    #ifdef FLIP_SIDED
                        transformedTangent = - transformedTangent;
                    #endif
                #endif
                `
            ).replace(
                `#include <project_vertex>`,
                `
                vec4 mvPosition = vec4( transformed, 1.0 );
                #ifdef USE_INSTANCING
                mvPosition = im * mvPosition;
                #endif
                mvPosition = modelViewMatrix * mvPosition;
                gl_Position = projectionMatrix * mvPosition;
                `
            ).replace(
                `#include <worldpos_vertex>`,
                `
                #if defined( USE_ENVMAP ) || defined( DISTANCE ) || defined ( USE_SHADOWMAP ) || defined ( USE_TRANSMISSION )
                    vec4 worldPosition = vec4( transformed, 1.0 );
                    #ifdef USE_INSTANCING
                        worldPosition = im * worldPosition;
                    #endif
                    worldPosition = modelMatrix * worldPosition;
                #endif
                `
            );
            shader.fragmentShader = shader.fragmentShader;
            _.hexMaterial.userData.shader = shader;
        };

I still can’t believe it but I got the shadows working!

In order to solve it I had to assign a .customDepthMaterial to the instancedMesh, this depthMaterial is a copy of the default one, I pass it the same heightMap and instUV uniforms of the main modified material and modify the vertex chunks in almost the same way (depthMaterial doesn’t use “defaultnormal_vertex” chunk so I apply the heightMap transformations after the “begin_vertex” chunk).

You can see the result here with the ‘GPU heightMap’ + ‘CPU simplex noise’ (shadows are applied in the center area only because I set it on purpose in the shadow camera properties of the DirectionalLight that casts the shadow):

Here is the working code for the main material for the Mesh:

this.hexMaterial = new THREE.MeshStandardMaterial({
            //map: new THREE.TextureLoader().load('https://cdn.rawgit.com/egor-sorokin/threejs-examples/8ca7a514/assets/img/triangles.jpg'),
            color: 0xffffff,
            //transparent : true,
            //opacity: 0.5
            roughness: 1,
            metalness: 0.5
            //wireframe: true
        });
        this.hexMaterial.onBeforeCompile =  function (shader) {
            shader.uniforms.heightMap = _.hexMaterialUniforms.heightMap;
            shader.vertexShader = `
                #define USE_HEIGHTMAP

                uniform sampler2D heightMap;
                attribute vec2 instUV;
                varying float vHeight;
                ${shader.vertexShader}
            `.replace(
                `#include <defaultnormal_vertex>`,
                `
                vec3 transformedNormal = objectNormal;
                #ifdef USE_INSTANCING
                    // this is in lieu of a per-instance normal-matrix
                    // shear transforms in the instance matrix are not supported
                    mat4 im = instanceMatrix;
                    #if defined( USE_HEIGHTMAP )
                        float h = texture2D(heightMap, instUV).r;
                        vHeight = h;
                        im[3].z += h * 20.;
                    #endif
                    mat3 m = mat3( im );
                    transformedNormal /= vec3( dot( m[ 0 ], m[ 0 ] ), dot( m[ 1 ], m[ 1 ] ), dot( m[ 2 ], m[ 2 ] ) );
                    transformedNormal = m * transformedNormal;
                #endif
                transformedNormal = normalMatrix * transformedNormal;
                #ifdef FLIP_SIDED
                    transformedNormal = - transformedNormal;
                #endif
                #ifdef USE_TANGENT
                    vec3 transformedTangent = ( modelViewMatrix * vec4( objectTangent, 0.0 ) ).xyz;
                    #ifdef FLIP_SIDED
                        transformedTangent = - transformedTangent;
                    #endif
                #endif
                `
            ).replace(
                `#include <project_vertex>`,
                `
                vec4 mvPosition = vec4( transformed, 1.0 );
                #ifdef USE_INSTANCING
                mvPosition = im * mvPosition;
                #endif
                mvPosition = modelViewMatrix * mvPosition;
                gl_Position = projectionMatrix * mvPosition;
                `
            ).replace(
                `#include <worldpos_vertex>`,
                `
                #if defined( USE_ENVMAP ) || defined( DISTANCE ) || defined ( USE_SHADOWMAP ) || defined ( USE_TRANSMISSION )
                    vec4 worldPosition = vec4( transformed, 1.0 );
                    #ifdef USE_INSTANCING
                        worldPosition = im * worldPosition;
                    #endif
                    worldPosition = modelMatrix * worldPosition;
                #endif
                `
            );
            shader.fragmentShader = shader.fragmentShader;
            _.hexMaterial.userData.shader = shader;
        };

And here is the working code for the custom DepthMaterial of the mesh:

this.hexCustomDepthMaterial = new THREE.MeshDepthMaterial({
            depthPacking: THREE.RGBADepthPacking,
            alphaTest: 0.5,
            onBeforeCompile: shader => {
                shader.uniforms.heightMap = _.hexMaterialUniforms.heightMap;
                shader.vertexShader = `
                    #define USE_HEIGHTMAP
                    uniform sampler2D heightMap;
                    attribute vec2 instUV;
                    varying float vHeight;
                    ${shader.vertexShader}
              `.replace(
                    `#include <begin_vertex>`,
                    `#include <begin_vertex>
                    
                    mat4 im = instanceMatrix;
                    #if defined( USE_HEIGHTMAP )
                        float h = texture2D(heightMap, instUV).r;
                        vHeight = h;
                        im[3].z += h * 20.;
                    #endif
                    mat3 m = mat3( im );

                    transformed = m * transformed;
                    `
              ).replace(
                `#include <project_vertex>`,
                `
                vec4 mvPosition = vec4( transformed, 1.0 );
                #ifdef USE_INSTANCING
                mvPosition = im * mvPosition;
                #endif
                mvPosition = modelViewMatrix * mvPosition;
                gl_Position = projectionMatrix * mvPosition;
                `
            ).replace(
                `#include <worldpos_vertex>`,
                `
                #if defined( USE_ENVMAP ) || defined( DISTANCE ) || defined ( USE_SHADOWMAP ) || defined ( USE_TRANSMISSION )
                    vec4 worldPosition = vec4( transformed, 1.0 );
                    #ifdef USE_INSTANCING
                        worldPosition = im * worldPosition;
                    #endif
                    worldPosition = modelMatrix * worldPosition;
                #endif
                `
            );

            }
        });

I’m still struggling to apply the heightMap in case the grid of instances is not a 1:1 array (for example if the grid is 30 columns by 50 rows, or 50 columns by 30 rows), because I create the grid depending on the visible 3DWorld in the viewport.

I have the intuition it should not be very hard but I’m very noob at shader computation, so if anyone can give a hand I would very much appreciate it, surely it has to do with this line of the vertexshader…

float h = texture2D(heightMap, instUV).r;

Anyway thanks a lot to everybody for your help until now, I couldn’t have reach this point without your expertise, and thanks in advance for any insight to resolve this last issue.

I think I managed to solve the problem to apply the heightMap to non square grids.

If you notice something strange please let me know!

I must credit again @prisoner849 , this post by him, Video Texture - how to keep video aspect in new material - #4 by prisoner849, gave me the insight to solve it.

I created a new uniform called ‘aspectRatio’, so if the grid is for example 30 columns by 50 rows then aspectRatio = 30/50;
Also, prior to this I had in the vertex shaders of the mesh material and the customdepthMaterial this:

float h = texture2D(heightMap, instUV).r;

And I changed it to this:

vec2 tInstUV = instUV;
if(aspectRatio <= 1.0){
    tInstUV.x *= aspectRatio;
    tInstUV.x -= (0.5 - (1. / aspectRatio) * 0.5) * aspectRatio;
}else{
    tInstUV.y /= aspectRatio;
    tInstUV.y -= (0.5 - (1. * aspectRatio) * 0.5) / aspectRatio;
}
//float h = texture2D(heightMap, tInstUV).r;

That seems to work in the tests I made so far (until I discover an ugly bug…), so I flag this problem SOLVED :slight_smile:

Thanks a lot to everybody, I will share a link to a live demo when I’m finished thinkering with the interaction, raycasting, viewport responsiveness, etc.

I also share some images for reference of this last solved issue:

That is what happened before, notice that the height map seems stretched:


Even worse in horizontal viewport in my ultrawide monitor:

With the modifications I mentioned before now we get this:


And this:

The changes make the effect to ‘crop’ the height map texture, somewhat like a CSS background-size: cover…

In very extreme aspect ratios you can notice the circle displacementMap I use to debug being cropped a lot but think I will use the water GPUComputation shader texture I mentioned previously, so I need that the texture maps as best as possible with the instances, even if the texture is ‘cropped’ in x or y axis.

2 Likes