Create video element in the three.js/editor

shikitos · May 28, 2021, 11:54am

I’m trying to add a video element to three.js / editor, but I can’t do it. Is it possible to add a video texture without getting involved in adding new elements to HTML? For example, using a link?

let video, videoTexture, movieScreen;

video = 'URL TO VIDEO';

videoTexture = new THREE.VideoTexture(video);
videoTexture.needsUpdate = true;

const videoMaterial = new THREE.MeshBasicMaterial({ map: videoTexture, side: THREE.DoubleSide });

const planeGeometry = new THREE.PlaneGeometry( video.videoWidth, video.videoHeight,	4,	4 );
videoElem = new THREE.Mesh(planeGeometry, videoMaterial);

this.add(videoElem);

I don’t think it’s going to work that way. But I don’t understand how to make this code work. Thank you all for your help!

Mugen87 · May 28, 2021, 2:04pm

This code does indeed not work since the constructor of VideoTexture requires a video element as an argument. A simple URL is not sufficient.

I’ve never tried it but maybe it works if you create the video element inline via Document.createElement()?

CHOIX · August 28, 2024, 4:09pm

I also tried to add videomaterial to the editor… it’s complicated than I thought, since you need to understand the structure/architecture of the editor code.

create a MeshVideoMaterial.js in editor/js/

import * as THREE from 'three';

class MeshVideoMaterial extends THREE.MeshBasicMaterial {
    constructor(parameters = {}) {

        // Create the video element
        const video = document.createElement('video');
        video.src = 'http://localhost:8083/editor/images/video.MOV'; //fixed video as example
        video.loop = true;
        video.style.display = 'none'; // Hide the video element
        document.body.appendChild(video);

        // Wait for the video to load and play
        video.play();
        editor.renderActive = true; //ugly hack

        // Create the VideoTexture
        const texture = new THREE.VideoTexture(video);
        texture.needsUpdate = true;

        super({ userData: {type:"videomaterial"}, map: texture, ...parameters });
    }
}

export { MeshVideoMaterial };

in the Sidebar.Material.js:

import { MeshVideoMaterial } from './MeshVideoMaterial.js';
//add this line in the materialClasses object:
    'MeshVideoMaterial': MeshVideoMaterial

//and this line in the meshMaterialOptions object:
	'MeshVideoMaterial': 'MeshVideoMaterial'

//and somewhere define the play and pause buttons to showup in the sidebar:
	const playButton = new UIButton('Play').onClick(() => {
		editor.selected.material.map.image.play();
		editor.renderActive = true; //ugly flag for renderer later
	});
	const pauseButton = new UIButton('Pause').onClick(() => {
		editor.selected.material.map.image.pause()
		editor.renderActive = false; //ugly flag for renderer later
	});
	container.add(playButton);
	container.add(pauseButton);

now in the Viewport.js:

//since I didn't know how to push tasks to the renderloop..
//more ugly hacks; add this somewhere:
	setInterval(() => {
		if(editor.renderActive){
			render();
		}
	}, 10);

Selectable MeshVideoMaterial (but since the type is MeshbasicMaterial, after selection the selected type will still show MeshBasicMaterial):

play and pause buttons for the video

It’s a prooooof of concept, I see a video material playing on the meshes… maybe we could use this as a “starting point”:

-be able to select a video directly on the “map” button; determin if it’s a video then create the video element and show play/pause button

-somehow make it serializable… so that you can save and load the scene with the video

I guess it’s in the Sidebar.Material.MapProperty.js - but I couldn’t find the click listener that opens the file browse option.

manthrax · August 29, 2024, 3:57am

afaik you don’t need to add the video to the dom?
The main videotexture sample does add it, but I’m not sure if that is required?

https://threejs.org/examples/#webgl_materials_video

You may have to wait until the user has clicked once, in order to begin playing the video…
Which might be what causes the perception that it has to be added to the dom…
To serialize it, you can stick the url in the texture.userData and rebuild the video after loading…

CHOIX · August 29, 2024, 5:37pm

You don’t need, maybe you can access the video data alao directly on the video element, but the example you send is doing it exactly like I am doing it, check the code in your link.

manthrax · August 29, 2024, 10:49pm

I modified that sample locally… replaced:


		//video = document.getElementById( 'video' );

with:
		video = document.createElement( 'video' );
		video.id = "video"
		video.setAttribute("crossOrigin","anonymous");
		video.setAttribute("src", "textures/sintel.mp4");

video.play();

and it works just fine.

CHOIX · August 30, 2024, 10:54am

what do you exactly mean?
how did you integrated that example into the three.js editor?

manthrax · August 30, 2024, 11:01am

I mean paste this script into an object script in the editor:


function init(){
let video = document.createElement( 'video' );
video.id = "video"
video.setAttribute("crossOrigin","anonymous");
video.setAttribute("src", "../examples/textures/sintel.mp4");
video.play();
let texture = new THREE.VideoTexture(video);
this.material.map = texture;
}

and it will replace the material with the sintel video.

CHOIX · August 30, 2024, 2:49pm

ooh I see what you are doing now! ok it really works out of the box like that yes.

and crazy part is you can save the scene as json in the editor and load the json and interpret the script again the the scene.

I am not sure if there are already existing solutions or I’m doing it right here:

Load the json scene and execute the init and update (this should be pushed into the animation loop):


function executeScriptForObject(object, script) {
  if (script && script.source) {
    try {
      // Create a new function that wraps the script source
      const scriptFunction = new Function('THREE', `
        return function() {
          ${script.source}
          if (typeof init === 'function') init.call(this);
          if (typeof update === 'function') this.update = update.bind(this);
        }
      `);

      // Call the function with the correct context (this)
      scriptFunction(THREE).call(object);
    } catch (error) {
      console.error(`Error executing script for ${object.name}:`, error);
    }
  }
}


function LoadedScene() {
  const sceneRef = useRef();
  const { scene } = useThree();

  useEffect(() => {
    fetch('/threeScene.json')
      .then((response) => response.json())
      .then((json) => {
        console.log("Loaded JSON:", json);

        // Create an instance of ObjectLoader
        const loader = new THREE.ObjectLoader();

        // Parse the scene data from the JSON
        const sceneData = json.scene || json.object;
        if (sceneData) {
          const loadedScene = loader.parse(sceneData);

          // Add the loaded scene to the main scene
          if (sceneRef.current) {
            sceneRef.current.add(loadedScene);
          } else {
            scene.add(loadedScene);
          }

          // Execute scripts associated with objects
          const scripts = json.scripts || {};
          loadedScene.traverse((object) => {
            const objectScripts = scripts[object.uuid];
            if (objectScripts) {
              objectScripts.forEach((script) => {
                executeScriptForObject(object, script);
              });
            }
          });

          console.log('Scene loaded and scripts executed:', loadedScene);
        } else {
          console.error('Scene data not found in JSON');
        }
      })
      .catch((error) => {
        console.error('Error loading scene:', error);
      });
  }, [scene]);

  return <group ref={sceneRef} />;
}

manthrax · August 31, 2024, 6:21am

Yeah that looks legit!

CHOIX · August 31, 2024, 5:14pm

I think the workaround with the video script is nice - but in the end I would like to have a simple to use solution for the endusers… being able to select the video in the map option of the material would be still nice.

also how would I push the update() function into the useFrame or the animation loop?

CHOIX · August 31, 2024, 5:23pm

ok found the final solution
add this:

else { // or you could also detect the exact video type
				reader.addEventListener('load', function (event) {
					const video = document.createElement('video');
				  
					video.addEventListener('loadeddata', function () {
					  const videoTexture = new THREE.VideoTexture(this);
					  videoTexture.sourceFile = file.name;
					  videoTexture.needsUpdate = true;
				  
					  cache.set(hash, videoTexture);
				  
					  scope.setValue(videoTexture);
				  
					  if (scope.onChangeCallback) scope.onChangeCallback(videoTexture);
					}, false);
				  
					// Assuming the loaded file is a video
					video.src = URL.createObjectURL(file);
					video.play(); // Start playback immediately (optional)
					video.loop = true;
				  }, false);
				  
				  reader.readAsDataURL(file);
			}

on the line 167 in editor/js/libs/ui.three.js

github.com

mrdoob/three.js/blob/8ca1d4dff992d45598e4280d86683dd3cce133d7/editor/js/libs/ui.three.js#L167


      
          						if ( scope.onChangeCallback ) scope.onChangeCallback( texture );
          
          					}, false );
          
          					image.src = event.target.result;
          
          				}, false );
          
          				reader.readAsDataURL( file );
          
          			}
          
          			form.reset();
          
          		}
          
          		this.texture = null;
          		this.onChangeCallback = null;
          
          	}

final question is: how to make this run in the “play” mode of the editor?

Topic		Replies	Views
How to manage video inside a class Questions javascript	1	697	May 13, 2020
How do I use YouTube IFrame with WebGLRenderer? Questions video , iframe , youtube , videotexture	6	2545	July 3, 2022
Trying to use video texture but not works Questions	0	120	May 10, 2024
VideoTexture with MeshBasicMaterial decreases performance immediately Questions video-texture	5	1003	July 31, 2022
Controls are not on videotexture Questions video-texture	3	773	October 14, 2020

Create video element in the three.js/editor

Related topics