Hey!
I am trying to render the view faced by my model using the canvas element. I am facing lags when there is movement in the model. It isn’t smooth and misses a lot of frames in between. Is there a better way to capture the frames and stream? Initially, when I do orbital controls it gets stuck and then moves smoothly.
Here is the code I used to render the first-person view.
Thanks in advance. Would appreciate solutions to make it a fast and smooth view.
function Render({ pCamera }) {
const { setCameraView } = useStore();
const aTarget = useFBO(640, 480, {
type: THREE.UnsignedByteType
})
const guiCamera = useRef()
useThree()
const debugBG = new THREE.Color('#fff')
useFrame(({ gl, camera, scene }) => {
gl.autoClear = false
scene.background = debugBG
/** Render scene from camera A to a render target */
if (pCamera && pCamera.current) {
gl.setRenderTarget(aTarget)
gl.render(scene, pCamera.current)
const width = aTarget.width
const height = aTarget.height
// Create a temporary canvas to draw the texture
const canvas = document.createElement('canvas')
canvas.width = width
canvas.height = height
const context = canvas.getContext('2d')
// Read pixels from the render target
const pixels = new Uint8Array(4 * width * height)
gl.readRenderTargetPixels(aTarget, 0, 0, width, height, pixels)
// Create ImageData with the correct dimensions
const imageData = context.createImageData(width, height)
// Copy the pixel data to the ImageData, handling potential padding
for (let i = 0; i < imageData.data.length; i += 4) {
imageData.data[i] = pixels[i]
imageData.data[i + 1] = pixels[i + 1]
imageData.data[i + 2] = pixels[i + 2]
imageData.data[i + 3] = pixels[i + 3]
}
// Put the image data on the canvas
context.putImageData(imageData, 0, 0)
// Flip the image vertically
context.scale(1, -1)
context.translate(0, -height)
context.drawImage(canvas, 0, 0)
// Get the data URL
const dataURL = canvas.toDataURL()
setCameraView(dataURL);
}
scene.overrideMaterial = null
gl.setRenderTarget(null)
gl.render(scene, camera)
}, 1)
return <OrthographicCamera ref={guiCamera} near={0.0001} far={1} />
}
const canvas = document.createElement('canvas') is a very expensive thing to do (and will eventually crash the tab, since you’re creating new canvas contexts 60 times per seconds.)for (let i = 0; i < imageData.data.length; i += 4) { is an even more expensive thing to do.const dataURL = canvas.toDataURL() super expensive thing to do.
You’ve put a lot of strain on the CPU, which is single threaded in the browser. All lines between gl.render(scene, pCamera.current) and setCameraView(dataURL); should be happening on the GPU. Why do you need dataURL outside of GPU (the only case I could imagine that is using the frame in a separate <img> element or exporting it to server / local machine, but either shouldn’t be happening at 60fps) ? Do you need it continuously updated, or you just want to use it as a texture in WebGL (in which case you can just use aTarget.texture and pass it to further materials.)
I want it to be continuously updated (can be 30fps) and streamed to the server.
Can you recommend a better way to do it? how do I access the camera view and stream it?
Before diving too deep into streaming, I’d try the following and see if performance-wise it’s acceptable:
// 1. Create the canvas only once
// Pass { willReadFrequently: true } to canvas context
// to make it do the I/O a bit faster
const width = 640;
const height = 480;
const canvas = document.createElement("canvas");
canvas.width = width;
canvas.height = height;
const context = canvas.getContext("2d", {
willReadFrequently: true,
});
// 2. This can also be reused, no need to recreate on each framr
const pixels = new Uint8Array(4 * width * height);
// 3. Create the component
function Render({ pCamera }) {
const { setCameraView } = useStore();
const aTarget = useFBO(width, height, { type: THREE.UnsignedByteType });
const guiCamera = useRef();
const debugBG = new THREE.Color("#fff");
// 4. Replace useFrame with useEffect, useFrame runs at 60fps, useEffect runs at whatever you tell it to
// If you see flickering due to interval race-conditioning with useFrame, use requestAnimationFrame
// instead of setInterval
const { gl, camera, scene } = useThree();
useEffect(() => {
const render = () => {
if (!pCamera.current) {
return;
}
gl.autoClear = false;
scene.background = debugBG;
gl.setRenderTarget(aTarget);
gl.render(scene, pCamera.current);
gl.readRenderTargetPixels(aTarget, 0, 0, width, height, pixels);
// 5. Pass pixels directly to ImageData (https://developer.mozilla.org/en-US/docs/Web/API/ImageData/ImageData)
// to avoid the loop
const imageData = new ImageData(pixels, width, height);
context.putImageData(imageData, 0, 0);
context.scale(1, -1);
context.translate(0, -height);
context.drawImage(canvas, 0, 0);
const dataURL = canvas.toDataURL();
setCameraView(dataURL);
scene.overrideMaterial = null;
gl.setRenderTarget(null);
// 6. Note - if you're using R3F, no need to call .render manually
// gl.render(scene, camera);
};
const interval = setInterval(() => render(), 1000 / 30); // 30fps
return () => clearInterval(interval);
}, [ gl, camera, scene ]);
return <OrthographicCamera ref={guiCamera} near={0.0001} far={1} />;
}
At that low resolution, this could be enough to make the app run smoothly.
Thank you so much for the help!!
I tried the changes you mentioned. setInterval with useEffect was updating the camera feed once my WASD controls stopped.
I replaced it with requestAnimationFrame in useEffect and it seems to work better.
the CPU time was reduced by 50% but the FPS of the entire scene remained at 11-12 even after the changes. Before I add this camera viewer the FPS was 20-22.
How can I increase overall fps of the scene?
This indicates the app itself has some significant issues with optimisation, since the default FPS would generally be 60 (on low end devices) or 120 (on high end devices.) If it’s 20 - I’d remove the streaming / FBO entirely and first optimise the rest of the code. When it gets back to 60-120 fps, only then work with the FBOs.