Minimizing cross-talk between main and embedded rendering?

My game uses a separate rendering pass to render portraits of the characters in an overlay HUD. These portraits are not animated, so they only render occasionally - basically when a character is selected with the mouse. Each portrait uses a separate scene with its own camera and lights.

To minimize flickering, I render the overlay portraits first, and then transfer the pixels from the DOM element to a canvas. These canvases are then layered on top of the main view.

Unfortunately, there is still some flicker which happens whenever a portrait is rendered - it appears that something funny is going on with either the lighting, shadows or the environment map. The main view renders lighter for one frame after the portrait is rendered.

I don’t have these problems with the portals because portals use an offscreen render target. However, my understanding is that render targets are not the right solution for generating images which are not part of the scene but are instead separate DOM elements.

It would be nice if there was a way to ensure that there was no bleed-over of state between the various renderings.

Yikes, that sounds like a tricky bug. :confused:

Are you able to narrow it down further? Some ideas to try would be …

  • render without env map
  • render with renderer.outputEncoding = THREE.LinearEncoding
  • render without lights

… and see if the flicker is still visible.


I wonder if keeping a separate renderer with a smaller size (e.g. size of 1 portrait?) might be easier and more memory-efficient too. But either way it’d be nice to track down the bug you describe…

So, both the main scene and the portrait scene have two lights: a directional light and a hemisphere light. Interestingly, if I delete the hemisphere light from the portrait scene, I get the following exception when the main scene attempts to render the next frame:

Uncaught TypeError: Cannot read properties of undefined (reading 'direction')
    at StructuredUniform.setValue (three.module.js:17814:20)
    at StructuredUniform.setValue (three.module.js:17814:6)
    at WebGLUniforms.upload (three.module.js:17943:7)
    at setProgram (three.module.js:28257:18)
    at WebGLRenderer.renderBufferDirect (three.module.js:27177:19)
    at renderObject (three.module.js:27807:10)
    at renderObjects (three.module.js:27776:5)
    at renderScene (three.module.js:27698:35)
    at WebGLRenderer.render (three.module.js:27518:4)
    at Engine.render (Engine.ts:418:19)

This happens when uploading the uniforms for a custom shader that is used to render the terrain. The terrain is only present in the main scene, and is not used in the portraits.

So my current thinking is that there is some internal state that is created as a side effect of rendering the lights, and this state is not updated when switching scenes.

It gets weirder. Using a separate renderer makes the problem worse, not better. As soon as I render a portrait, the terrain in the main scene becomes unlit (fully bright, no shadows - like a basic material) and stays that way. This persists until (you’re going to love this) any portal comes into view.

Rendering a portal calls a number of state settings functions:

renderer.setRenderTarget(this.renderTarget);
renderer.setViewport(this.portalViewport);
renderer.clippingPlanes[0].copy(this.clippingPlane);
renderer.clear(true, true, true);
renderer.render(this.destinationScene, this.portalCamera);

I’m guessing that one or more of these functions causes the internal state of the renderer to be reset.

Note that when a portal is not on-screen, these methods are also called for the main view, but with the same value each time, so nothing changes.

@donmccurdy So, I still have not figured out a solution or a workaround for this. I’ve tried the various things you suggested with no luck.

From what I can tell, it appears to be a problem with the lighting and my custom terrain shader. Here are the details:

  • The terrain shader is based on the standard material shader, except that the color is determined by an algorithm rather than by a texture or constant color. The shader does lighting calculations by including the lights_physical_fragment, lights_fragment_begin and lights_fragment_end shader chunks. It doesn’t do anything special with lighting.
  • Only the terrain shader appears to be affected by the bug. The standard material shaders appear to work OK.
  • I’ve been using this shader for over 2 years and never noticed this problem until I started trying to do off-screen renders of the character portrait.
  • The exact symptoms are as follows: if I try and re-use the same WebGLRenderer instance for both scenes, then immediately after I render a character portrait, then the very next call to render the main scene uses the lighting setup from the portrait scene. A subsequent call to render() the main scene fixes the problem.
  • The above is only true if both scenes have the same number of lights. If the number of lights is different, then the renderer simply throws an exception when attempting to render the main scene.
  • When I attempt to isolate the problem by giving the portraits their own dedicated instance of WebGLRenderer, the problem gets worse: rendering the portraits affects the main scene as before, but the main scene no longer fixes itself on subsequent renders. Instead, the problem with the lights persists.

My guess is that somehow my custom shader is not initializing the light state, that somehow I’m getting a stale lighting state from the previous render. However, I have no idea what would cause this, normally when writing custom shaders you don’t do anything special to get lighting as long as you inherit from ShaderMaterial which sets all that stuff up automatically.

@donmccurdy So after a great deal of trial and error, I came up with a workaround that prevents the flashing problem, but it is an ugly, ugly hack:

After I render a character portrait, I set a flag called fixLightsHack. When this flag is set, I force the main render loop to do an extra call to render the main scene, but with shadows turned off. I then re-enable shadows and render the scene normally.

    // Do an extra render pass, with shadows disabled - this forces
    // a recalculation of all the lighting state.
    if (this.fixLightsHack) {
      this.fixLightsHack = false;
      r.directionalLight.castShadow = false;
      this.renderer.render(this.scene, this.camera);
      r.directionalLight.castShadow = true;
    }

    this.renderer.render(this.scene, this.camera);

This means a frame rate drop when I render a portrait, but I don’t render portraits very often so I can live with it.

While investigating this, I discovered a few other things. First, let me explain the setup:

  • Two WebGLRenderers - one for the main scene, and one for rendering portraits to a canvas element. Each renderer has its own separate scene and camera.
  • The main scene uses a conventional animation / render loop via RAF.
  • The portrait renderer is created lazily when needed and disposed immediately after each use.
  • The main scene has a custom shader which does physical lighting using the standard three.js lighting chunks. The portrait renderer does not use this shader.

What I found:

  1. If both the main renderer scene and the portrait renderer scene have the same number of lights then rendering with the portrait renderer breaks the lighting in the main renderer. In this case “breaks” means that the custom shader renders everything as unlit.
  2. If the portrait renderer has fewer lights than the main scene, then rendering a portrait causes the main renderer to crash.

The crash is in the three.js material upload() function. The reason is because the uniform variable direcrionalShadowMatrix is undefined, and the three.js code is expecting it to be an array.

This only happens with my custom shader, not with other materials. However, I don’t think its the fault of the custom shader since it’s not doing anything unusual with lights.

By doing an extra render pass, and flipping the state of the castShadow property, it forces three.js to recalculate all of the light-related uniforms, which fixes the bug.

I tried your suggestions of playing with encoding and environment maps, but they did not solve the problem.

Sounds like three does some nasty caching on the global scope rather than the renderer instance, if you ask me.

Do you experience the same issues when you move your portrait renderer to a web worker?