Connect Audio from Three.JS to MediaRecorder

Questions
,
1

Question “forwarded” from here: https://stackoverflow.com/questions/57358937/how-to-connect-audio-to-mediastream-from-three-js-audiocontext

Any idea how to connect audio to a MediaRecorder canvas stream from Three.JS? Please let me know if I can provider any more code samples other than the ones in the the question. Help is greatly appreciated.

1 Like
2

I’m afraid your use case is not clear to me. Do you want to include the audio of a media stream into the three.js audio graph? There is currently no way to do this but you can try to enhance the THREE.Audio prototype like so:

THREE.Audio.prototype.setMediaStreamSource = function ( mediaStream ) {

    this.hasPlaybackControl = false;
    this.sourceType = 'mediaStreamNode';
    this.source = this.context.createMediaStreamSource( mediaStream );
    this.connect();

    return this;

};
3

@Mugen87 Other way around actually. I am looking to include the audio from Three.JS into the mediarecorder. Does that make sense?

4

Are you talking about this API? MediaRecorder - Web APIs | MDN

5

That’s the one. The video works well, but the audio is lacking entirely. Other SO answers haven’t really helped unfortunately.

6

For developers in the FuTuRe:

const context: AudioContext = ThreeAudioContext.getContext();
const destination = context.createMediaStreamDestination();
this.audioListener.getInput().connect(destination);
this.backgroundGainNode.connect(destination);
stream.addTrack(destination.stream.getAudioTracks()[0]);

In short, to connect audio to the MediaRecorder, call createMediaStreamDestination and connect the gainNode (volume node) to the newly created destination. Then, add the track to the stream.

Couple of hiccups I had:

  • All gain nodes you connect need to be under the same audio context.
  • If you want to connect audio to the stream, even though it may not be audible, you need to create an independent GainNode.

Of course, I say GainNode with the understanding you might be using a different type of AudioNode, but I assume 95% of people just want audio to play without any alteration besides volume.

4 Likes
7

@Kotamigo
Hi, I am looking for some solution like yours for some days now.
Recording the video from a canvas is doing well but getting the audio from the played video does not work.
Could you… widen your solution by telling where audioListener and backgroundGainNode are coming from or how they are related to you snippet, please?

Trying ```
ThreeAudioContext.getContext();

Getting a "normal" AudioContext() results in an error when I try to get a node from it that says that no node from an other context were allowed.


Thank you in advance.
8

What do you mean by 'getting the audio from the played video'? Are you trying to:

A) record video from your THREE.js canvas along with audio from a <video> element or THREE.VideoTexture

B) record video from the THREE.js canvas and audio from THREE.Audio?

9

Hello an thx 4 your time,

I would like to record a scene with a video texture in it and the sound of the video. I know I could record the sound of the video- element but I would strongly prefer the canvas’ sound as it plays 100% synchronized.