Struggling for days: Poor device camera performance with Three.js when fed to a shader

Questions
1

Symptoms: dropped frames, video stuttering, no-smooth playback for more than a while at best, even for 30fps video (let alone 60fps).

Conditions:
My PC is old but I see this with a 15% CPU load and almost zero VGA load.
OS: Win7 x64.
Browsers: Chrome, FFox, Opera.

I’ve tried almost everything:

-different methods,
-different resolutions (even …stamp-size ones)
-different amount of lighting,
-reduced render loop to 30fps,
-standard loop, or WebXR-style loop,
-HTML variations,
-JS variations,
-fed as videoTexture or grabbing frames one-by-one,

ALL work poorly, that is, I never get smooth playback for more than a few seconds, when I feed the video with whatever method to a texture in order to post-process it, even if I don’t do anything more than simple 1x sampling before I render it to the screen.

At first, I thought it was an issue with my phone (which is not a weak one BTW), but it also happens on the PC , and it happens with playback of local video too, at whatever resolution.

Since it’s neither the CPU or GPU, I’m guessing that there is something really wrong with the getUserMedia() based API or other WEBGL API, or hopefully there is some critical bug in Three.js implementation.

So the question is: Why is that happening? Any ideas?
Have you ever seen perfectly smooth playback of webcam/mobile cam/local video when that video is processed via a shader (not just applied to a scene texture) with Three.js?

Note: in order to be able to notice the dropped frames, the video should be one with a constant motion, eg: POV Driving Scania R450 Xerta to Gandesa 🇪🇸 ,cockpit view 4K - YouTube look at the sides of the video.
Or, move your webcam/mobile camera with steady speed and see if it’s always smooth, with any example code.

2

I not sure if I understand good (I not speak english). But this example works smooth to me three.js examples
it uses texture video and post processing bloom. maybe you can get a idea.

Source code: three.js/webgl_materials_video.html at d4aa9e00ea29808534a3e082f602c544e5f2419c · mrdoob/three.js · GitHub

3

Thanks, but that’s hard to even see the video, let alone see the dropped frames…

…so I simplified it a bit, to use just a single steady plane, I uploaded it to jsfiddle, and replaced that video with a continuous-motion one, (one that I’ve made several years ago for testing) which I uploaded it to a free host to be used by jsfiddle.

Now you can see how smooth the play back of that example actually is, via WEBGL/Three.js when it is post-processed: https://jsfiddle.net/84j9t7px/1/

You can download the video to your PC if you like, play it back with a player (eg pod player) and compare the smoothness, but I bet that you can see that it’s anything but smooth (watch the “60” number, or even “30”).

*that’s a 60fps 1-minute video, and along with Three.js’ 60fps render loop and a 60fps monitor, it should have 60fps perfectly smooth motion…

4

Since this is 60 megabytes (vs about 3 of the example’s), I added a progress indication, to know when the video has fully loaded before judging its smoothness: https://jsfiddle.net/84j9t7px/1/

5

Im not sure if I really understand the problem. I test your code and the only wrong thing I found was here

<link rel="stylesheet" crossorigin="anonymous"</head>

<link rel="stylesheet" crossorigin="anonymous"></head>

I test your code and works good, I got a smoth 60 fps playing a heavy video on local from start to end.
So, what about your online video. In my case I have problems even playing directly from the host. and also when I was downloading it i noticed it was downloading slow, I have a 120mb internet and it took me 5 minutes for complete the download it. (thats too mucho for a 60 mb file). So its not a surprise that the video stop for buffering when you attempt to play it online.

Here a example using your code

I also test your video and some others, and and they all worked correctly at 60fps.

my system:
*windows 10
*i5 6500
*gtx960 2gb