skinnedMesh (gltf) with shaderMaterial doesn't respond to blender animations

Hey everyone. Like the title says I am hoping to apply a simple shader (change an image texture to grayscale) to a material on part of a gltf dynamically. With a mesh standard material (which is in use when the grayscale shader material is not being applied), the animations work as expected, but when using ShaderMaterial / RawShaderMaterial the part of the model with the shader material applied does not respond to gltf animation. I am definitely missing something in my shader, but I am not sure what. I know this issue has been covered before (with solutions like onBeforeCompile: Custom shader works for SkinnedMesh - #4 by Fedor_van_Eldijk) or here (Meshes created with ShaderMaterial do not play skeletal animation? · Issue #6567 · mrdoob/three.js · GitHub) and here (three.js - Synching two meshes, one with ShaderMaterial - Stack Overflow). It seems like there are some examples where it can be done quite simply with shader chunks which is what I am going for in the code below… but I am finding it quite tricky to deal with. Anyway, I was hoping someone with some real shader know how could point me in the right direction. Here’s a little code snippet (using glslify and react):

import React, {useEffect, useRef} from 'react';
import {Texture, NormalBlending, ShaderMaterial} from 'three';
import * as THREE from 'three';

const vertexShader = `
  precision highp float;
  precision highp int;
  uniform mat4 modelViewMatrix;
  uniform mat4 projectionMatrix;
  attribute vec3 position;
  attribute vec3 normal;
  attribute vec2 uv;
  varying vec2 vUv;
  #include <common>
  #include <skinning_pars_vertex>
  
  void main() {
    #include <skinnormal_vertex>
    #include <skinbase_vertex>
    #include <begin_vertex>
    #include <skinning_vertex>
    #include <project_vertex>
    
    vUv = uv;

    gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
  }
`;

const fragmentShader = `
  precision highp float;
  precision highp int;
  varying vec2 vUv;
  uniform sampler2D uTexture;
  uniform sampler2D tDiffuse;
  uniform vec3 uColor;
  uniform float rPower;
  uniform float gPower;
  uniform float bPower;
  uniform float offsetX;
  uniform float offsetY;
  uniform float repeatValue;

  void main() {
    vec4 texel = texture2D( uTexture, vec2((vUv.x - offsetX) * repeatValue, (vUv.y - offsetY) * repeatValue));
    
    float grey = texel.r * rPower + texel.g * gPower + texel.b * bPower;
    gl_FragColor = vec4( vec3(grey), texel.w );
  }
`;

interface GreyscaleMaterialProps {
  diffuse: number;
  rPower: number;
  gPower: number;
  bPower: number;
  uTexture: Texture;
  offsetX: number;
  offsetY: number;
  repeatValue: number;
  skinning: boolean;
}

export default function GreyscaleShaderMaterial({
  diffuse,
  rPower,
  gPower,
  bPower,
  uTexture,
  offsetX,
  offsetY,
  repeatValue,
  skinning
}: GreyscaleMaterialProps) {
  const material = useRef<typeof ShaderMaterial>(null!);

  const uniforms = {
    tDiffuse: {value: diffuse},
    rPower: {value: rPower},
    gPower: {value: gPower},
    bPower: {value: bPower},
    uTexture: {value: uTexture},
    offsetX: {value: offsetX},
    offsetY: {value: offsetY},
    repeatValue: {value: repeatValue},
    skinning: {value: skinning}
  };

  useEffect(() => {
    uTexture.needsUpdate = true;
    uTexture.center.set(0.5, 0.5);
    uTexture.repeat.set(3, 3);
    uTexture.offset.x = 0;
    uTexture.offset.y = 0;
    uTexture.wrapS = THREE.RepeatWrapping;
    uTexture.wrapT = THREE.RepeatWrapping;
    uTexture.minFilter = THREE.NearestFilter;
    uTexture.magFilter = THREE.NearestFilter;
    uTexture.flipY = false;
  });

  return (
    <rawShaderMaterial
      ref={material}
      blending={NormalBlending}
      attach='material'
      args={[
        {
          uniforms,
          vertexShader: vertexShader,
          fragmentShader: fragmentShader,
        },
      ]}
    />
  );
}

1 Like

Sorry to bump an old post but did you happen to figure this one out eventually? I spent a while narrowing down the problem I’m having with animations to my custom ShaderMaterial but I’m not sure what my shader could be missing. If you didn’t or just don’t remember anymore that’s alright, I appreciate your time either way

I never did. I had to abandon the shader approach due to time constraints so used a canvas texture for my needs. that being said it is definitely possible so don’t give up if you absolutely need it!

1 Like

Okay I figured it out. All I actually needed was the skinning_vertex chunk (I placed it before the project_vertex chunk), and the params for it with the skinning_pars_vertex chunk (placed at the end of the rest of the pars since I don’t think the order matters there.)

And then I had to make sure that I was using mvPosition for my final gl_Position because otherwise the calculations that the skinning chunk did would be ignored. (I was stuck on this part for way too long, had to figure out how to log the whole shader (with the shader chunks) in order to figure out what was actually happening.)

1 Like