Extending material for probes volume

Hi,

I’m working on some threejs material extension in order to have a probe system implemented for standard,physical and phong material.

Project repo is here https://github.com/gillesboisson/threejs-probes-test
Demo is here : https://three-probes.dotify.eu/

It uses a blender plugin i’m working on https://github.com/gillesboisson/blender-probes-export

I have two issues on the material extension side :

  • uniform sync get messy when a material is used by two meshes.
  • shader has cache issue as it is build for each material.

I tried to create some class factory based on materiel extension librairies I saw on github.

The uniforms life cycle looks this on my implementation

  • it first initialiazed uniforms on constructor
  • copy it to shader.uniforms on beforeCompile and reassigned it to this.uniforms
  • and update this.uniforms in beforeRender (which seems to be same for each shader program)

for cache key issues I use a custom method but it seems the getPrograms method of the WebGLRenderer does get programs

// ... WebGLRenderer implementation
function getProgram( material, scene, object ) {

	if ( scene.isScene !== true ) scene = _emptyScene; // scene could be a Mesh, Line, Points, ...

	const materialProperties = properties.get( material );

	const lights = currentRenderState.state.lights;
	const shadowsArray = currentRenderState.state.shadowsArray;

	const lightsStateVersion = lights.state.version;

	const parameters = programCache.getParameters( material, lights.state, shadowsArray, scene, object );
	const programCacheKey = programCache.getProgramCacheKey( parameters );


        // !!!! this return undefined for each material
	let programs = materialProperties.programs;


 // ... WebGLRenderer implementation

Here is my factory implementation and extend example

import {
  BufferGeometry,
  Camera,
  IUniform,
  Material,
  MaterialParameters,
  Object3D,
  Scene,
  Shader,
  WebGLRenderer,
  UniformsUtils,
} from 'three'
import { replaceShaderSourceIncludes } from './utils'
import {
  probesMaterialFragmentChunksOverride,
  probesMaterialVertexChunksOverride,
} from './shaderShunk'
import {
  defines,
  irradianceMapNames,
  maxIrradianceMaps,
  maxReflectionMaps,
  // materialUniforms,
  ratioVar,
  reflectionLodVar,
  reflectionMapNames,
} from './shaderConstants'
import { ProbeVolumeHandler } from '../ProbeVolumeHandler'
import { ProbeRatio, ProbeRatioLod } from '../type'
import {
  IrradianceProbeVolume,
  ProbeVolumeRatio,
  ReflectionProbeVolume,
} from '../volume'

const irradianceRatioVarname = ratioVar('irradiance')
const reflectionRatioVarname = ratioVar('reflection')
const reflectionLodVarname = reflectionLodVar()

export function extendProbesMaterial<
  MaterialT extends Material = Material,
  MaterialParamsT extends MaterialParameters = MaterialParameters
>(
  SuperMaterial: typeof Material,
  defaultParams: Partial<MaterialParamsT> = {},
  shaderDefinition?: {
    vertexShader?: (shader: string) => string
    fragmentShader?: (shader: string) => string
    uniforms?: Record<string, IUniform>
    defines?: Record<string, any>
  }
): {
  new (
    probeVolumeHander: ProbeVolumeHandler,
    params?: Partial<MaterialParamsT>
  ): MaterialT
} {
  return class ExtendedProbeMaterial extends SuperMaterial {
    protected uniforms: Record<string, IUniform>

    private _irradianceProbeRatio: ProbeRatio[] = []
    private _reflectionProbeRatio: ProbeRatioLod[] = []

    private _irradianceGlobalProbeRatio: ProbeVolumeRatio<IrradianceProbeVolume>[] =
      []
    private _reflectionGlobalProbeRatio: ProbeVolumeRatio<ReflectionProbeVolume>[] =
      []

    protected _probesIntensity: number = 1

    get probesIntensity(): number {
      return this._probesIntensity
    }

    set probesIntensity(value: number) {
      this._probesIntensity = value
    }

    constructor(
      readonly probeVolumeHander: ProbeVolumeHandler,
      params: Partial<MaterialParamsT> = {}
    ) {
      super()

      if (shaderDefinition?.defines) {
        this.defines = {
          ...defines,
          ...shaderDefinition.defines,
        }
      }

      const uniforms: Record<string, IUniform> = shaderDefinition?.uniforms
        ? UniformsUtils.clone(shaderDefinition.uniforms)
        : {}

      irradianceMapNames.map((name) => {
        const uniform: IUniform = { value: null }
        uniforms[name] = uniform
      })

      reflectionMapNames.map((name) => {
        const uniform: IUniform = { value: null }
        uniforms[name] = uniform
      })

      uniforms[irradianceRatioVarname] = {
        value: new Float32Array(maxIrradianceMaps),
      }

      uniforms[reflectionLodVarname] = {
        value: new Float32Array(maxReflectionMaps),
      }

      uniforms[reflectionRatioVarname] = {
        value: new Float32Array(maxReflectionMaps),
      }

      uniforms.probesIntensity = { value: this._probesIntensity }

      this.uniforms = UniformsUtils.clone(uniforms)
      // debugger
      this.setValues({
        ...defaultParams,
        ...params,
        // uniforms,
      })
    }

    onBeforeRender(
      renderer: WebGLRenderer,
      scene: Scene,
      camera: Camera,
      geometry: BufferGeometry,
      object: Object3D,
      group: Object3D
    ) {
      // console.log('object',object);
      const uniforms = this.uniforms

      const irradianceProbeRatio = this._irradianceProbeRatio
      const reflectionProbeRatio = this._reflectionProbeRatio

      const irradianceRatioBufferData = uniforms[irradianceRatioVarname].value as Float32Array
      const reflectionRatioBufferData = uniforms[reflectionRatioVarname].value as Float32Array
      const reflectionLodBufferData = uniforms[reflectionLodVarname].value as Float32Array
      
      this.probeVolumeHander.irradianceVolumes.getSuroundingProbes(
        object.position,
        irradianceProbeRatio,
        this._irradianceGlobalProbeRatio
      )
      this.probeVolumeHander.reflectionVolumes.getSuroundingProbes(
        object.position,
        reflectionProbeRatio,
        this._reflectionGlobalProbeRatio,
        (this as any).roughness
      )

      for (let i = 0; i < irradianceRatioBufferData.length; i++) {
        if (i < irradianceProbeRatio.length) {
          irradianceRatioBufferData[i] = irradianceProbeRatio[i][1]
          uniforms[irradianceMapNames[i]].value =
            irradianceProbeRatio[i][0].texture
        } else {
          irradianceRatioBufferData[i] = 0
          uniforms[irradianceMapNames[i]].value = null
        }
      }

      for (let i = 0; i < reflectionRatioBufferData.length; i++) {
        if (i < reflectionProbeRatio.length) {
          reflectionRatioBufferData[i] = reflectionProbeRatio[i][1]
          reflectionLodBufferData[i] = reflectionProbeRatio[i][2]
          uniforms[reflectionMapNames[i]].value =
            reflectionProbeRatio[i][0].texture
          // ;(reflectionTextureUniforms[i] as any).needsUpdate = true
        } else {
          reflectionRatioBufferData[i] = 0
          reflectionLodBufferData[i] = 0
          uniforms[reflectionMapNames[i]].value = null
        }
      }

      this.uniforms.probesIntensity.value = this._probesIntensity
      this.needsUpdate = true
    }

    customProgramCacheKey(): string {
      return this.name + super.customProgramCacheKey()
    }

    onBeforeCompile(shader: Shader, renderer: WebGLRenderer): void {
      super.onBeforeCompile(shader, renderer)
      
      for (let key in this.uniforms) {
        shader.uniforms[key] = this.uniforms[key]
      }

      this.uniforms = shader.uniforms

      shader.vertexShader = shaderDefinition?.vertexShader
        ? shaderDefinition.vertexShader(shader.vertexShader)
        : replaceShaderSourceIncludes(
            shader.vertexShader,
            probesMaterialVertexChunksOverride
          )

      shader.fragmentShader = shaderDefinition?.fragmentShader
        ? shaderDefinition.fragmentShader(shader.fragmentShader)
        : replaceShaderSourceIncludes(
            shader.fragmentShader,
            probesMaterialFragmentChunksOverride
          )
      ;(shader as any).defines = {
        ...(shader as any).defines,
        ...defines,
      }
    }
  } as any
}

// IMPLEMENTATION

import { MeshPhysicalMaterial } from 'three';

export class MeshProbePhysicalMaterial extends extendProbesMaterial<MeshPhysicalMaterial>(
  MeshPhysicalMaterial
) {
  name = 'MeshProbePhysicalMaterial';
}

If you have some doc or example on how cache and uniforms works on custom material (without using ShaderMaterial) it would be great

Cheers.

I kind of resolved the uniforms sync issue. the issues was only for per object uniform properties (same material : many objects) changed on material onBeforeRender hook.

The only trick i found was to emulate ShaderMaterial behaviour by

  • setting isShaderMaterial to true
  • setting uniformsNeedUpdate to true
  • add uniformsGroups empty array
  return class ExtendedProbeMaterial extends SuperMaterial {
    // ...
    protected uniforms: Record<string, IUniform>
    protected uniformsGroups: UniformsGroup[] = []
    readonly isShaderMaterial = true
    protected uniformsNeedUpdate = false

    // ...
    onBeforeRender(
      renderer: WebGLRenderer,
      scene: Scene,
      camera: Camera,
      geometry: BufferGeometry,
      object: Object3D,
      group: Object3D
    ) {
    // ...
    let objectUniformsNeedsUpdate = false
    // compare and update uniforms


    this.uniformsNeedUpdate = objectUniformsNeedsUpdate
    }
}

I still have the shader cache issue …

Are you using material.customProgramCacheKey ? : three.js docs ?

Hi,

Yes I tried with and without it. I did a test by checking when onBeforeCompile is called and I saw that it is called once per material for more than one render call. I thought is was an issue but it seems to be expected behaviour when I looked at WebGLRenderer getProgram method.

On this part of the code

const parameters = programCache.getParameters( material, lights.state, shadowsArray, scene, object );
const programCacheKey = programCache.getProgramCacheKey( parameters );
let programs = materialProperties.programs;

materialProperties.programs is a different for each material. which trigger onBeforeCompile at least once per material.