skinnedMesh (gltf) with shaderMaterial doesn't respond to blender animations

Questions
,
1

Hey everyone. Like the title says I am hoping to apply a simple shader (change an image texture to grayscale) to a material on part of a gltf dynamically. With a mesh standard material (which is in use when the grayscale shader material is not being applied), the animations work as expected, but when using ShaderMaterial / RawShaderMaterial the part of the model with the shader material applied does not respond to gltf animation. I am definitely missing something in my shader, but I am not sure what. I know this issue has been covered before (with solutions like onBeforeCompile: Custom shader works for SkinnedMesh - #4 by Fedor_van_Eldijk) or here (Meshes created with ShaderMaterial do not play skeletal animation? · Issue #6567 · mrdoob/three.js · GitHub) and here (three.js - Synching two meshes, one with ShaderMaterial - Stack Overflow). It seems like there are some examples where it can be done quite simply with shader chunks which is what I am going for in the code below… but I am finding it quite tricky to deal with. Anyway, I was hoping someone with some real shader know how could point me in the right direction. Here’s a little code snippet (using glslify and react):

import React, {useEffect, useRef} from 'react';
import {Texture, NormalBlending, ShaderMaterial} from 'three';
import * as THREE from 'three';

const vertexShader = `
  precision highp float;
  precision highp int;
  uniform mat4 modelViewMatrix;
  uniform mat4 projectionMatrix;
  attribute vec3 position;
  attribute vec3 normal;
  attribute vec2 uv;
  varying vec2 vUv;
  #include <common>
  #include <skinning_pars_vertex>
  
  void main() {
    #include <skinnormal_vertex>
    #include <skinbase_vertex>
    #include <begin_vertex>
    #include <skinning_vertex>
    #include <project_vertex>
    
    vUv = uv;

    gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
  }
`;

const fragmentShader = `
  precision highp float;
  precision highp int;
  varying vec2 vUv;
  uniform sampler2D uTexture;
  uniform sampler2D tDiffuse;
  uniform vec3 uColor;
  uniform float rPower;
  uniform float gPower;
  uniform float bPower;
  uniform float offsetX;
  uniform float offsetY;
  uniform float repeatValue;

  void main() {
    vec4 texel = texture2D( uTexture, vec2((vUv.x - offsetX) * repeatValue, (vUv.y - offsetY) * repeatValue));
    
    float grey = texel.r * rPower + texel.g * gPower + texel.b * bPower;
    gl_FragColor = vec4( vec3(grey), texel.w );
  }
`;

interface GreyscaleMaterialProps {
  diffuse: number;
  rPower: number;
  gPower: number;
  bPower: number;
  uTexture: Texture;
  offsetX: number;
  offsetY: number;
  repeatValue: number;
  skinning: boolean;
}

export default function GreyscaleShaderMaterial({
  diffuse,
  rPower,
  gPower,
  bPower,
  uTexture,
  offsetX,
  offsetY,
  repeatValue,
  skinning
}: GreyscaleMaterialProps) {
  const material = useRef<typeof ShaderMaterial>(null!);

  const uniforms = {
    tDiffuse: {value: diffuse},
    rPower: {value: rPower},
    gPower: {value: gPower},
    bPower: {value: bPower},
    uTexture: {value: uTexture},
    offsetX: {value: offsetX},
    offsetY: {value: offsetY},
    repeatValue: {value: repeatValue},
    skinning: {value: skinning}
  };

  useEffect(() => {
    uTexture.needsUpdate = true;
    uTexture.center.set(0.5, 0.5);
    uTexture.repeat.set(3, 3);
    uTexture.offset.x = 0;
    uTexture.offset.y = 0;
    uTexture.wrapS = THREE.RepeatWrapping;
    uTexture.wrapT = THREE.RepeatWrapping;
    uTexture.minFilter = THREE.NearestFilter;
    uTexture.magFilter = THREE.NearestFilter;
    uTexture.flipY = false;
  });

  return (
    <rawShaderMaterial
      ref={material}
      blending={NormalBlending}
      attach='material'
      args={[
        {
          uniforms,
          vertexShader: vertexShader,
          fragmentShader: fragmentShader,
        },
      ]}
    />
  );
}
1 Like
2

Sorry to bump an old post but did you happen to figure this one out eventually? I spent a while narrowing down the problem I’m having with animations to my custom ShaderMaterial but I’m not sure what my shader could be missing. If you didn’t or just don’t remember anymore that’s alright, I appreciate your time either way

3

I never did. I had to abandon the shader approach due to time constraints so used a canvas texture for my needs. that being said it is definitely possible so don’t give up if you absolutely need it!

1 Like
4

Okay I figured it out. All I actually needed was the skinning_vertex chunk (I placed it before the project_vertex chunk), and the params for it with the skinning_pars_vertex chunk (placed at the end of the rest of the pars since I don’t think the order matters there.)

And then I had to make sure that I was using mvPosition for my final gl_Position because otherwise the calculations that the skinning chunk did would be ignored. (I was stuck on this part for way too long, had to figure out how to log the whole shader (with the shader chunks) in order to figure out what was actually happening.)

1 Like
5

well done!! Could you share an example by any chance? I have revisited and am still struggling with this.

I feel like I am pretty close:

const vertexShader = `
  precision highp float;
  precision highp int;
  uniform mat4 modelViewMatrix;
  uniform mat4 projectionMatrix;
  uniform float uTime;
  attribute vec3 position;
  attribute vec3 normal;
  attribute vec2 uv;
  varying vec2 vUv;
  #include <common>
  #include <skinning_pars_vertex>

	#ifdef USE_SKINNING
		uniform mat4 bindMatrix;
		uniform mat4 bindMatrixInverse;
		#ifdef BONE_TEXTURE
			uniform sampler2D boneTexture;
			uniform int boneTextureSize;
			mat4 getBoneMatrix( const in float i ) {
				float j = i * 4.0;
				float x = mod( j, float( boneTextureSize ) );
				float y = floor( j / float( boneTextureSize ) );
				float dx = 1.0 / float( boneTextureSize );
				float dy = 1.0 / float( boneTextureSize );
				y = dy * ( y + 0.5 );
				vec4 v1 = texture2D( boneTexture, vec2( dx * ( x + 0.5 ), y ) );
				vec4 v2 = texture2D( boneTexture, vec2( dx * ( x + 1.5 ), y ) );
				vec4 v3 = texture2D( boneTexture, vec2( dx * ( x + 2.5 ), y ) );
				vec4 v4 = texture2D( boneTexture, vec2( dx * ( x + 3.5 ), y ) );
				mat4 bone = mat4( v1, v2, v3, v4 );
				return bone;
			}
		#else
			uniform mat4 boneMatrices[ 4 ];
			mat4 getBoneMatrix( const in float i ) {
				mat4 bone = boneMatrices[ int(i) ];
				return bone;
			}
		#endif
	#endif
	void main() {
		// #include <skinnormal_vertex>
    #include <skinbase_vertex>
    #include <begin_vertex>
    #include <skinning_vertex>
    // #include <project_vertex>
    
    vUv = uv;

		#ifdef USE_SKINNING
			mat4 boneMatX = getBoneMatrix( skinIndex.x );
			mat4 boneMatY = getBoneMatrix( skinIndex.y );
			mat4 boneMatZ = getBoneMatrix( skinIndex.z );
			mat4 boneMatW = getBoneMatrix( skinIndex.w );
			mat4 skinMatrix = mat4( 0.0 );
			skinMatrix += skinWeight.x * boneMatX;
			skinMatrix += skinWeight.y * boneMatY;
			skinMatrix += skinWeight.z * boneMatZ;
			skinMatrix += skinWeight.w * boneMatW;
			skinMatrix  = bindMatrixInverse * skinMatrix * bindMatrix;
			vec4 skinVertex = bindMatrix * vec4( position, 1.0 );
			vec4 skinned = vec4( 0.0 );
			skinned += boneMatX * skinVertex * skinWeight.x;
			skinned += boneMatY * skinVertex * skinWeight.y;
			skinned += boneMatZ * skinVertex * skinWeight.z;
			skinned += boneMatW * skinVertex * skinWeight.w;
			skinned  = bindMatrixInverse * skinned;
			vec4 mvPosition = modelViewMatrix * skinned;
		#else
			vec4 mvPosition = modelViewMatrix * vec4( position, 1.0 );
		#endif
		gl_Position = projectionMatrix * mvPosition;
	}
`;

Though when I try to use project vertex as you suggested, the material disappears…

1 Like
6

Not sure about what you sent me, not enough time to debug atm. Here’s my vertex shader though if it’s useful (note that I did remove an irrelevant part):

#include <common>
#include <fog_pars_vertex>
#include <shadowmap_pars_vertex>
#include <skinning_pars_vertex>

varying vec3 vNormal;
varying vec3 fogPosition;
varying vec3 vWorldPosition;

void main() {
	
	#include <skinbase_vertex>
	#include <begin_vertex>
	#include <beginnormal_vertex>
	#include <defaultnormal_vertex>
	vec4 worldPosition = modelMatrix * vec4( position, 1.0 );
	#include <skinning_vertex>
	#include <project_vertex>
	#include <shadowmap_vertex>
	#include <fog_vertex>
	
	vNormal = normal;
	fogPosition = mvPosition.xyz;
	vWorldPosition = worldPosition.xyz;
	gl_Position = projectionMatrix * mvPosition;
}
3 Likes
7

Sorry for the late reply! I appreciate you sharing. There are definitely some chunks in here I haven’t tried applying yet, along with using the worldPosition. Will let you know how it goes! :slight_smile:

1 Like
8

@ChiriVulpes Thanks again for your insight. I was able to figure it out thanks to your help infused with some other info out there like so:

  const vertexShader = `
    varying vec3 Normal;
    varying vec3 Position;
    #include <common>
    #include <skinning_pars_vertex>

    void main() {
      #include <skinbase_vertex>
      #include <begin_vertex>
      #include <beginnormal_vertex>
      #include <defaultnormal_vertex>
      #include <skinning_vertex>
      #include <project_vertex>

      Normal = normalize(normalMatrix * normal);
      Position = vec3(modelViewMatrix * vec4(position, 1.0));
      gl_Position = projectionMatrix * mvPosition;
    }
  `;

Also, at first I was using a rawShaderMaterial, and this code block doesn’t work with rawShaderMaterial, only shaderMaterial :slight_smile: The mvPosition was indeed the key however!

1 Like