VideoTexture Chromakey - WebXR AR & Three.js

Questions
,
#1

Using WebXR and Three.js for Augmented Reality, I’m trying to display a videotexture (with chromakey) on a box which is display where the user has touch the ground surface.

I arrive to place the model on the ground and it seems the videotexture is attached to the box. But it never play and update (Keep black screen).

I use this three.js extension for video Chromakey. https://github.com/hawksley/Threex.chromakey

In my code structure, I don’t know where to put video.startVideo() and video.update(). I tried to place the video.update() in onXRFrame(time, frame) but it didn’t worked.

Have you a solution for this, where material.update() should be place? Here is my code :

   async onSessionStarted(session) {
 ...

 this.renderer = new THREE.WebGLRenderer({
 alpha: true,
preserveDrawingBuffer: true,
});
this.renderer.autoClear = false;

this.gl = this.renderer.getContext();
this.scene = new THREE.Scene();

var canPlayMp4  = document.createElement('video').canPlayType('video/mp4') !== '' ?   true : false
if( canPlayMp4 ){
var url = 'imaes/video.mp4'
}else{  alert('cant play mp4 or ogv')}

var material = new THREEx.ChromaKeyMaterial(url, 0xd432); 
var geometry = new THREE.PlaneBufferGeometry( 3, 2);
this.model = new THREE.Mesh( geometry, material );
...

this.reticle = new Reticle(this.session, this.camera);
this.scene.add(this.reticle);
this.frameOfRef = await this.session.requestFrameOfReference('eye-level');
this.session.requestAnimationFrame(this.onXRFrame);
window.addEventListener('click', this.onClick);
}

async onClick(e) {
...
this.scene.add(this.model);
this.material.startVideo();
}

onXRFrame(time, frame) {
let session = frame.session;
let pose = frame.getDevicePose(this.frameOfRef);

this.reticle.update(this.frameOfRef);

if (this.reticle.visible && !this.stabilized) {
  this.stabilized = true;
  document.body.classList.add('stabilized');
}

session.requestAnimationFrame(this.onXRFrame);
this.gl.bindFramebuffer(this.gl.FRAMEBUFFER, this.session.baseLayer.framebuffer);

if (pose) {

  for (let view of frame.views) {
    const viewport = session.baseLayer.getViewport(view);
    this.renderer.setSize(viewport.width, viewport.height);

    this.camera.projectionMatrix.fromArray(view.projectionMatrix);
    const viewMatrix = new THREE.Matrix4().fromArray(pose.getViewMatrix(view));
    this.camera.matrix.getInverse(viewMatrix);
    this.camera.updateMatrixWorld(true);

    // Render our scene with our THREE.WebGLRenderer
    this.renderer.render(this.scene, this.camera);

  }
  }
  }
#2

/cc

#3

Hello,

Thanks you, yes its my post on StackOverflow.
I think it is more appropriate to ask directly on the three.js forum.

#4

Does it work if you just use a “normal” video texture without using the third-party plugin Threex.chromakey? Please try it with the code form this example:

https://jsfiddle.net/75amk4rv/

#5

I integrated your code example in my code, it works well I arrive to watch the video on a surface in AR. I didn’t needed to put animate() because the frame loop is done by on onXRFrame(time, frame):

I successfully reedit my code with the ChromaKeyMaterial.
The videos play and I see the video.
What I needed is to put the material.update function in onXRFrame()

I tried with a local file it works perfectly, with a external video too!
The camera freeze while loading the video and detecting plane surface though.
Does it exist to load video async?

.

#6

@fguill : Please post github demo. Would like to study the code.

Thank You! <3