How to stream audio in a PositionalAudio with AudioLoader?

Hello,

Is there a way to use a stream of audio (such as an internet radio / a soundfile-loop stream on a server) as the source of a PositionalAudio instance?

Thnx,
J

You can stream an audio if you create a native HTML5 Audio element and then set it via THREE.Audio.setMediaElementSource(). This approach is used in the following official example:

https://threejs.org/examples/webaudio_visualizer

Doing this is recommended for stuff like background music since it’s not necessary to download the whole audio buffer at once via AudioLoader.

Would this approach work for you?

1 Like

This looks good yes!

I will try to see if I can find a streaming audio example and replace the audio-file used here with the stream.

This is probably not the correct forum to ask this but, I am looking for a way to loop and stream an audio-file on a server and pipe it to all clients in real-time (so everybody visiting the website would hear the same playback). Then I would use that as a source for thee.js.

Do you think this is possible? Is there a good node-js forum to ask for this? I asked Quora and Stackoverflow, but didn’t get a straight response, Ideally I would love to find a forum like this one with a lot of activity but for node-js :slight_smile:

if my question is irrelevant, I understand, and it can be ignored :wink:

Thank you for the answer!

If you don’t get a response at stackoverflow, there could be also an issue with your question. So maybe you have to write it differently in order to get more feedback. In general, stackoverflow should be a good space for such topics.

If you can get it work with the pure Web Audio API, we will probably find a solution with three.js, too. I’m not sure this is helpful but AudioContext.createMediaStreamSource() could be interesting for you. MediaStreamAudioSourceNode is not yet supported by three.js. But it could be part of your solution if HTML5 Audio is no option for some reasons.

Thnx! I will look into it.

FYI: With the next release R109, THREE.Audio provides a new method .setMediaStreamSource()for including Web RTC media streams into the Web Audio pipeline. Could be useful to you :wink:

2 Likes

Hi @Mugen87 !

Here I am again 2 years later! My project is now live and can be viewed here on this website:

https://www.subneu.live

It is working great on desktop, but on mobile it seems to not really work… The page refreshes at a certain point by itself. My intuition tells me this must be a RAM overload issue.

On the website I am presenting a contemporary electronic music audio piece with 14 10-minute-long mono sound files played back on 14 different Positional Audio instances inside three.js.

I have two questions.

  1. I am using the setMediaElementSource method. Does this mean that the audio is streamed? the buffer is loaded in chunks as opposed to loading the entire soundfile? This is of course what I want to accomplish. I can imagine that the RAM would get overflown pretty fast if it needs to load that many audiofiles on page-load.

  2. in the example you gave https://threejs.org/examples/webaudio_visualizer I see in the code that it is checking the user-agent for mobile and uses the AudioLoader if on mobile. why is this the case?

Thank you for your help

That is correct. The purpose of this method is to avoid downloading the entire audio buffer at once.

Ricardo added this code as an iOS workaround. However, I don’t why it was added and if the latest iOS/Webkit still requires it.

Ok,

Do you have any idea why my website doe snot work on mobile?
Since I am using the streaming method for positional audio does it matter how long the audio files are? or how many positonal audio instances there are in the scene? Are there any rules or limitations concerning this?

Thank you

Does the website work if you add no sound at all?

That should not be important.

I’m not aware of any defined limitations (e.g. max number of panner nodes).

Yes it works perfectly without the sound.
I am using SimpleFPControls to move the camera around ( GitHub - jsantell/three-simple-fp-controls: Simple, boring, first person controls for three.js</ti ). When on desktop I pass a positionalAudio source the camera sometimes hangs and in the next frame I am suddenly positioned somewhere else, as if something goes wrong with the controls. This is especially noticeable on slower computers and I think this must be the reason it does not work at all on mobile.

Once I “passed” all the PositionalAudio objects with my camera, everything seems “loaded” and it goes smoothly (on my macbook pro 2020)

I am also using DeviceOrientationControls to use the gyroscope on mobile, so maybe this could also be a cause for the issue

I am using these classes to play the sound

export default class Sound {
    constructor(url, durationCallback) {
        this.duration = 0
        this.mediaElement = new Audio(url);
        this.mediaElement.loop = true;
        this.mediaElement.preload = 'auto';
        this.mediaElement.crossOrigin = 'anonymous';
        var self = this
        this.mediaElement.addEventListener("loadedmetadata", function(_event) {
            var dur = self.mediaElement.duration;
            self.duration = dur
            durationCallback(dur)
        });
    }

    play() {
        this.mediaElement.play()
    }

    setTime(time) {
        let newTime = time / 1000
        if (!newTime.isNan && newTime != undefined && isFinite(newTime)) {
            this.mediaElement.currentTime = newTime
        }
    }
}

export default class Speaker {

constructor(scene, listener, soundURL, imageURL, position, durationCallback) {
    this.object = new THREE.Object3D()
    this.object.position.set( position.x, position.y, position.z )
    scene.add(this.object)
    
    this.spriteDimensions = 40

    this.sound = null
    this.setupSoundSource(soundURL, durationCallback, listener)
    this.setupSpeakerSprite(imageURL)
    this.setupSpeakerAvatarIcon(imageURL)
    this.object.layers.set(1)
}

setupSoundSource(soundURL, durationCallback, listener) {
    this.sound = new Sound(soundURL, durationCallback)
    var soundSource = new THREE.PositionalAudio(listener)
    soundSource.setRefDistance(5)
    soundSource.setRolloffFactor(1) //0.2
    soundSource.setMediaElementSource(this.sound.mediaElement)
    this.object.add(soundSource)
}

setupSpeakerSprite(imageURL) {
    this.sprite = new Sprite(imageURL, 
        {
            x: 0,
            y: 0,
            z: 0
        },
        {   x: 6,
            y: 6,
            z: 6
    })
    let color = new THREE.Color(1, 1, 1)
    this.sprite.setColor(color)
    this.object.add(this.sprite.mesh)
}

scale(value) {
    let scaled = this.spriteDimensions * value
    this.speakerAvatar.scale.set( scaled, scaled, scaled )
}

play() {
    this.sound.play()
}

setTime(time) {
    this.sound.setTime(time)
}

getTime() {
    this.sound.currentTime()
}

dispose() {
    this.speakerAvatar.geometry.dispose()
    this.speakerAvatar.material.dispose()
    this.object.remove(this.speakerAvatar)

    this.sprite.geometry.dispose()
    this.sprite.material.dispose()
    this.object.remove(this.sprite)
}
}

Hi, so I joined this forum because of this exact same issue. I cannot get the positionalAudio() or Audio() based sound to work if I use setmediaStreamSource() with an HTML5 element

I am using the positionalAudio() and Audio() for some positional and ambient sound respectively. I could give you the code example but my implementation is the same as @uvfauna (if not as elegant). if I use the AudioLoader() to load the sound they both positionalAudio() and Audio() work fine but that would cause memory issues that I would rather avoid. Any advice?

Hello @mudebbela.

For me I could fix the crashing of my website by reducing the amount of speakers.
But yeah. The audio still crashes after a while… like a crackling sound that eventually completely silences all audio output.
Unfortunately it is simply a known bug on iOS and there seems nothing we can do about it (I’ve spent enough time researching this).
The only thing we can do is pressure the webkit team to fix this goddamn bug!

Here: 211394 – createMediaElementSource not working

Ah yes, I found my people. Ive been trying everything to get positional audio to work on mobile to no avail. Thanks for the link.