VREffect Video Artifacts

Hi All

I am currently working on a VR project which uses a video texture on a plane as a virtual “TV”. I am testing using a Galaxy Gear VR headset (with S7 Edge).

Unfortunately, the “quality” is below what I would consider acceptable for the purposes of this project. Specifically, the rendered video feels low res, and has serious artifacts with head movement. I believe it needs some sort of antialiasing, but I have been unable to find a way to correct it.

So far I have tried:

  • Enabling AA on the renderer.
  • Different resolutions of video, including specifically encoding at power-of-2 dimensions.
  • Copying to a canvas beforerender at various resolutions, including 2-step (upscale followed by downscale and vice versa).
  • Copying to a canvas using a fast JS blur alogorithm (too slow, drops framerate unacceptably)

I believe some postprocessing may be useful. The various AA algorithms seem to do what I need, but I have been unable to figure out how to use them with VREffect (effect composer doesn’t seem to be compatible with VREffect).

Can anybody offer any suggestions? Put simply, there is no point continuing with the project if I cannot improve the video quality to somewhere in the region of the built-in video players on the Gear VR.

Thanks in advance
Mouse

OK, I’ve done a little more work.

With a bit of “hacking” of the code I’ve managed to add the EffectComposer into VREffect (not the best way, but it works as a proof of concept).

However, adding SMAA does little if anything to improve the artifacts on the video texture, as well as dropping the framerate to a barely acceptable level.

I’m starting to think that my application (a VR theatre for a specific media server) is not going to be possible with any kind of quality through three.js/WebVR. This would be disappointing as the whole reason I went down this path (rather than a native app) was to keep it cross-platform.

Does anyone have any suggestions before I abandon the WebVR approach?

Kind regards
Karl

Does this example render low quality too?
https://threejs.org/examples/#webvr_video

It is slightly pixelated, but not in a noticable way. However, the difference in detail in this video, as well as the fact that it is 360, probably make a significant difference.

Maybe I should describe my scene. Basically, I create a plane in front of the camera, sized apropriately for the video (by THX specifications, diagonal being distance from the viewer multiplied by 0.84). I apply a VideoTexture to this plane, which is the video I wish to display.

I have tested with a couple of my own videos as well as Big Buck Bunny, in various resolutions. What happens is that details such as the trees in the background produce noticable artifacts. I’m sure we all remember these from earlier computer games. As the detail in the trees is higher resolution than the display, you get a fairly jarring pixelation as they move, each pixel seeming to suddenly change and distort. I can’t think of a better way to describe it. This is compounded in VR as it happens even when the scene is not moving but the user’s head does.

I have tested the same videos in the native video player on the Gear VR. While it’s not an exact comparison as the display surface is slightly larger, there are no appreciable artifacts. This is what I was hoping for.

If you wish, I will throw together an example to demonstrate. I can’t really post my current code in it’s present state, (it’s rather complicated and uses some libraries I have built for myself, so would probably confuse at first glance and not make for an easy troubleshooting excercise).

That’d be very helpful!

OK, I’ve thrown together an example here. This uses a 30s loop from Big Buck Bunny, put onto a virtual screen of approximately the dimensions I wish to use. Note it is just a hacked copy of the WebVR Video example.

The areas to consider in particular are the trees in the background, and also the grass. See the “artifacts”, i.e. how the pixelation of the grass/leaves create a distracting distortion (not sure how to explain it). If you fire this up in the standard VR Video player, these are almost absent (just go to recent videos in the Samsung VR Internet Browser to see it in the default player).

BTW, since I last tried this, there has been an update to the Gear VR stuff which seems to have slightly improved things, but it’s still bad enough to be distracting.

Thank you in advance for any help you may be able to offer.

hey @drmouse I was wondering if you could post that hack you made I was hoping to do the same and some help is always appreciated!