R3F/ThreeJS, Memory Leak when Canvas is Scrolled out of View

Good Evening/Morning Everyone,

Hoping that someone can help.

I’m relatively new to the threeJS, react-fiber and 3d modelling world. I managed to get a 3D model loaded on a webpage I’m building. I got the model animations playing smoothly and got some interactivity with the model moving its head to follow the cursor - all great so far. However, I encountered a memory leak that for the life of me I can’t work out.

The model has no issue whatsoever when the website loads initially, however, if I scroll further down the page when the Canvas begins to leave the visible window, all of a sudden the memory usage begins to spike. I’ve done quite a bit of research but I can’t find a solution so I’m hoping the resident geniuses here will be able to help.

I attempted to create a codesandbox but had some difficulties with dependencies - if necessary I can retry however. What I have included instead is the relevant code and a recording showing the memory-heap spike:

The Canvas

import { Canvas } from "@react-three/fiber"
import { ReactNode, Suspense, useEffect, useRef} from "react"
import {EnvironmentProps, Stage } from "@react-three/drei";
import useMediaQuery from "@/utils/hooks/useMediaQuery";

interface ModelViewerProps {
    environment?: EnvironmentProps,
    children: ReactNode,
}

const ModelViewer = ({environment, children}:ModelViewerProps) => {
    const isMobile = useMediaQuery('mobile');
    
    return (
        <Canvas
            className="modelViewer"
            shadows 
            camera={{ 
                position: [.2, 0, 1],
            }}
            resize={{scroll:false}}
        >
            <Suspense fallback={null}>
                <Stage
                    intensity={0.5}
                    preset="rembrandt"
                    shadows={{ 
                        type: 'contact', 
                        color: '#408FCE', 
                        offset: -0.38,
                        scale: 4,
                        blur: 4,
                    }}
                    environment="city"
                    adjustCamera={isMobile ? 1 : 0}
                    center={{disableX: !isMobile}}
                >
                    <mesh>
                        {children}
                    </mesh>
                </Stage>
            </Suspense>
        </Canvas>
  )
}

The Model

import { getMouseDegrees } from "@/utils/functions/getMouseDegrees";
import useMediaQuery from "@/utils/hooks/useMediaQuery";
import { useAnimations, useGLTF } from "@react-three/drei";
import { dispose, useFrame } from "@react-three/fiber";
import { nanoid } from "nanoid";
import { useEffect, useRef, useState } from "react";
import { MathUtils, SkinnedMesh } from "three";

export interface MouseType {
  x: number,
  y: number
}

type CustomBoneType = THREE.Bone & {
  rotation: {
    xD: number,
    yD: number
  }
}

const moveHead = (mouse:MouseType, bone: CustomBoneType, degreeLimit = 40) => {
  const degrees = getMouseDegrees(mouse.x,mouse.y,degreeLimit);
  bone.rotation.xD = MathUtils.lerp(bone.rotation.xD || 0, degrees.y + 18, 0.1)
  bone.rotation.yD = MathUtils.lerp(bone.rotation.yD || 0, degrees.x, 0.1)
  bone.rotation.x = MathUtils.degToRad(bone.rotation.xD)
  bone.rotation.y = MathUtils.degToRad(bone.rotation.yD)
}

const Azura3DModelNew = ({...props}) => {

    const group = useRef(null);
    const { nodes, materials, animations } = useGLTF('/3dModels/azura3dModel-meshopt.glb');
    const { actions, names } = useAnimations(animations, group);
    const isDesktop = useMediaQuery('desktop');
    const mouse = useRef({x:0,y:0});

    const bonesToModify = [
      [nodes['spine009'],5],
      [nodes['spine010'],10],
      [nodes['spine011'],30]  ,
    ]
    
    useFrame((state, delta) => {
      if (isDesktop) {
        mouse.current = {x: state.size.width / 2 + (state.mouse.x * state.size.width) / 2, y: state.size.height / 2 + (-state.mouse.y * state.size.height) / 2}
        bonesToModify.forEach((bone) => moveHead(mouse.current,bone[0] as CustomBoneType,bone[1] as number));
      }
    })

    useEffect(() => {
        actions['foxSitIdle']?.reset().fadeIn(0.5).play();

// I assume the below isn't needed but left here to highlight that I had tried this also.
//        return () => {
//          dispose(materials);
//          dispose(nodes);
//          useGLTF.clear('/3dModels/azura3dModel-meshopt.glb');
        }
    }, [actions])


  return (
      <group 
        ref={group} 
        dispose={null} 
        position={[.3,0,0]} 
        scale={[1,1,1]}
        rotation={[0,-.3,0]}
        key={nanoid()}
        >
        <group name="metarig" scale={0.7} dispose={null}>
          <primitive object={nodes.master}/>
          <primitive object={nodes.floorConst} />
          <group name="Azura" dispose={null}>
            <skinnedMesh
              name="Plane"
              geometry={(nodes.Plane as SkinnedMesh).geometry}
              material={materials["Material.005"]}
              skeleton={(nodes.Plane as SkinnedMesh).skeleton}
              // receiveShadow castShadow
              dispose={null}
              />
            <skinnedMesh
              name="Plane_1"
              geometry={(nodes.Plane_1 as SkinnedMesh).geometry}
              material={materials["Material.002"]}
              skeleton={(nodes.Plane_1 as SkinnedMesh).skeleton}
              // receiveShadow castShadow
              dispose={null}
              />
            <skinnedMesh
              name="Plane_2"
              geometry={(nodes.Plane_2 as SkinnedMesh).geometry}
              material={materials["Material.003"]}
              skeleton={(nodes.Plane_2 as SkinnedMesh).skeleton}
              // receiveShadow castShadow
              dispose={null}
              />
          </group>
        </group>
      </group>
  )
}

export default Azura3DModelNew

I tried to use the React Profiler and Chrome Dev tools in an attempt to diagnose the issue but without success.

In the video below you will see that after loading, I simply scroll to the bottom of the currently empty page (I removed all other elements for simplicity). Very soon after the memory begins to spike. I can only assume that there is some kind of issue with disposing the model as needed.

Video Recording Showing Memory Spike:
2023-02-26 01-37-19.mkv (2.9 MB)

Thank you very much in advance to anyone that may be able to help.

Kind Regards,

David

i don’t think there’s anything wrong with the code. but best upload it into a small codesandbox so that it can be profiled. it looks like a leak in THREE.AnimationMixer, there shouldn’t be any difference running an animation whether the canvas is in view or not.

ps, you can try to pinpoint what it is by removing things one by one until the leak doesn’t happen no more and then you add it back to make sure it’s the code that’s causing it. from what i see there react just produces a static view and then keeps out of it, you’re changing a few bone rotations but none of this should be adding memory.

i would also add a console into the render function to see if it gets triggered in a loop, just to make sure.

Hey @drcmda!

Thanks for the swift response. I’ve managed to throw it into a working codesandbox which can be found here.

Thanks for your tips regarding how to pinpoint the issue, these are things that I had tried (I should have certainly mentioned this in my initial post). When commenting out the various bits of code, the only thing that seems to make a difference is commenting out the entire R3F/ThreeJS components. I attempted commenting out:

  • The useFrame
  • The useEffect
  • The useMediaQuery hook
  • The primitives
  • The skinnedMesh
  • The ‘Azura3dModelNew’ component
  • The ‘ModelViewer’ component

What is particularly odd, is that if I comment out the ‘Azura3dModelNew’ component, it still causes a memory leak (i.e. when there is no model). If I comment out the ‘ModelViewer’ and wrap the model in a simple ‘Canvas’ element instead there is still a memory leak. Which suggests that perhaps there is a leak in both ‘ModelViewer’ and in ‘Azura3dModelNew’.

In terms of consoling into the various components I had done this previously but I have added this into the codesandbox as an example. The ‘Azura3DModelNew.tsx’ does seem to render twice on initial load and the ‘useMediaQuery’ hook seems to render many times (4+) but none of them seem to trigger additional times when scrolling down when the leak occurs.

I really appreciate your time in helping with this, I’m truly at a loss.

Thanks again,

David

doesn’t seem like the page has any issues (mac M1)

the first three heap reads were done with the fox on the screen, the rest when scrolled down, each a half a minute apart. heap doesn’t grow so there’s no leak. the heap should represent the memory the page is actually using, all the open objects that are in memory. if that doesn’t grow i wouldn’t worry.

what the browser does outside of that, delegating more or less memory to a thread, it may be unrelated. maybe it’s just anticipating the tab to be hungrier than the others?

either way, definitively try the devtools memory tab on your system to make sure.

Hey,

Thanks again for getting back to me so quickly and apologies for the belated response - it’s been busy at work.

Thanks for sharing your profiling results when doing Heap Snapshots - I had a similar records. I’m not too familiar with this side of the dev tools so scouring through the snapshots doesn’t help me much honestly - I can see a number of locations where the ‘Retained Size’ is larger than the ‘Shallow Size’ which I’ve read is potentially indicative of a memory leak but looking into it I don’t feel that’s the case.

I’m at a bit of a loss because if the internal dev tools show no memory leak but Task manager shows that memory allocation is constantly increasing and then the browser eventually freezes and crashes - this is obviously a problem and I can’t use this model client-facing which is a real shame.

What’s particularly weird is that if I have the Chrome internal task manager open, the memory allocation increase doesn’t occur… which makes no sense to me, it’s like it’s doing some kind of alternate memory management. I’ve attached a couple videos to show what I’m talking about:

You’ll see in the first video, The memory increases aggressively after scroll (as experienced previously) but then in the second video when I do the same thing with the Chrome internal Task Manager open it doesn’t occur - furthermore when I close the internal task manager window, it starts increasing again…

I also then tried this with Edge, Firefox and Chrome. Edge and Chrome both had the same issue, Firefox interestingly didn’t have the issue and this seems to correlate to it actively using GPU instead…?

Just curious what your thoughts on this are?

It’s worth noting that all of these examples were tested on my Windows. When attempting to duplicate on my Mac I don’t have any of theses issues (only tested in Chrome). I don’t feel the specs of my Alienware should be problematic however.
Mac vs Windows:
Mac

Windows
image

As a potential fix all I can really think of is having the Canvas conditionally render on Scroll. i.e. If the user scrolls below a certain point, unmount the canvas. It shouldn’t be necessary but I can’t really think of another fix. I might also try using a different model format but I don’t feel that should have any bearing whatsoever.

Would really value any insights that you may have.

Warm Regards,

David

Profiling in dev tools in chrome I originally got the same result as @drcmda, this potentially is a local environment behaviour but

This, under the circumstances of such a low poly, low texture size environment sounds a bit too far fetched, are you suggesting that you remount and reload the entire canvas if a user scrolls back to the top of page?

A better check would be to see, if the canvas is in view, render the scene, if not don’t, but your environment shouldn’t be behaving in this way nonetheless

Hey @Lawrence3DPK ,

Thanks for the reply!

Yep, agreed - there is the plausibility, in fact likelihood, that this is a local issue (which is frustrating). I spent some time testing the same functionality on my mobile (Samsung s23 Ultra) and also on another windows computer (Lenovo) I have and could not replicate the issue. In all instances, it seems that whenever the device decides to use the GPU that the issue does not occur - for some reason my Alienware laptop doesn’t want to use GPU for Chrome or Edge when running this site. My main concern is that if it is a localized issue on my computer, there is the potential that it can be a localized issue for someone else (although your and @drcmda’s contributions give me hope that this might not be the case).

In terms of my reference to unmount/mount I was thinking something akin to:

{isCanvasInView && (
  <ModelViewer>
    <Azura3DModel/>
  </ModelViewer>
)

Can you recommend something performant that might still alleviate the issue in question?

Many thanks again.

Warm Regards,

David

but oculd use use/post a heap readout? because the screenshots you posted are just chrome tab memory, it may point to something or not, these aren’t useful for memory profiling. but heap will tell you exactly. make 10 snapshots, leave some time between each. first one at the start, scroll, snap, wait, snap, wait, …

scrolling and some low poly mesh isn’t any relevant stress on neither cpu nor gpu. i don’t even think this is a local issue. there may be some bug yes, im guessing there could be an infinite loop running or something of that nature. but first we need to isolate the issue.

1 Like

Hey @drcmda,

Thanks for the follow-up. Of course, happy to do so - the difficulty that I was having on my windows computer is that the memory for chrome would spike so high so quickly that I wouldn’t actually be able to take a snapshot - it would freeze, chrome would realize somethings wrong and do some kind of cleanup, then take the snapshot, somewhat defeating the purpose. I managed to have one rogue run where Chrome decided to use GPU (appreciate this shouldn’t be necessary btw) and as a result it slowed down but didn’t freeze (hopefully it still represents the default experience), in the example below I took snapshots every 20 seconds. The top four are at the top of the screen, the remaining are scrolled down to the bottom.

Overall the heap did grow but only from 46.87 to 47.4 - it then seemed to remain static so I don’t feel the overall heap is necessarily indicative.

In the following screenshot I have placed the Objects allocated between Snapshot 4 and 5 (or between the top and scrolled to the bottom) in focus. I’m curious about a the Retained Size vs Shallow size for a number of Events here.

Here is the same view but objects allocated between 8 and 7:


I find it interesting that anything should be allocated given that no actions are occurring on screen (did not scroll or interact aside from the snapshot itself).

This is a comparison between 5 and 4:


This is a comparison between 8 and 7:

I’ve saved a copy of these heap snapshots here:

Thanks again for your on-going help, I appreciate it.

Warm Regards,

David

your heap is fine, there’s no javascript caused leak, meaning objects that accumulate and the GC doesn’t flush them.

so the problem must be something else. at this point i’d inspect browser versions (see if your chrome is old), try another browser and look if the same happens there, check your drivers, check if the gpu is even enabled, which gpu it uses (integrated/dedicated) and the gpu status/error page in chrome.

you said that you don’t even need a model to repro the problem. that means that all that’s running is a THREE.WebGLRenderer + gl.render(emptyScene, camera). i would also cross check with other webgl apps, like three examples.

Hey @drcmda ,

Thanks for getting back to me.

Chrome is fully up to date, I might try uninstalling and reinstalling later although I have a feeling that won’t be fruitful. As mentioned above, I did try this in multiple browsers, the issue also occurs in Edge but not in Firefox which suggests it may be Chromium-related?

Anything in particular I should be checking with drivers? Everything appears to be up-to-date and functioning.

Here are the current settings for GPU in Chrome (nothing edited from default to my knowledge):
GPUStatus.docx (44.8 KB)

GPU-wise everything seems to be enabled appropriately however, I did notice that Chrome is choosing to use my integrated GPU rather than my dedicated GPU. This is perfectly fine until I scroll out of view from the Canvas and GPU drops down to 0% (It’s almost as if it decides if the 3d model isn’t in view it wont turn on despite the fact it still needs to render). I updated my local settings for Chrome to force it into high-performance mode (previously it was “let windows decide”) and use my dedicated NVIDIA graphics card instead. When enabled I didn’t receive the error and everything worked as expected… it seems bizarre to me that such a small scene would cause this to be necessary however, particularly in the precise scenario where the canvas is scrolled out of view. Also, given that the issue isn’t replicable on my Mac or on my other windows computer which doesn’t have a dedicated GPU it seems odd.

I also added a random 3d model to test if it would occur, I copied the Soldier example from three.js and it’s the same story - fine when on-screen after scrolling to the bottom the memory for Chrome begins to creep up.

It’s great that this is working locally now with high-performance mode enabled for Chrome - what is your opinion that this issue might occur for other users? Simply as I want confidence if this is appropriate to use client-facing.

I hope it goes without saying how much I appreciate your help - I’m confident you have numerous better things to do that respond to my many essays.

All the best,

David