PositionalAudio setMediaStreamSource with WebRTC Question (not hearing any sound)

Questions
21

Hi,

I’m having the exact same problem as @cat
Firefox works, Chrome does not.
I tried all possible scenarios I could think of, including

createElement('audio') and
new Audio() and assigning srcObj

22

But the above example works for me in Chrome. Can’t you just use the same apporach?

23

The above example works for me, but the same approach does not work with an incoming MediaStream from a WebRTC Transport. I wonder if this has to do with the preload tag used in the example. For the record, the incoming source audio does work with the HTML audio element.

24

Do you know any free WebRTC streams that can be used in a live example?

25

Hmm… Let me look into making a minimal setup. My source is available, but definitely not a minimal setup: https://github.com/AidanNelson/YORB2020

26

Okay, I managed to create a minimal example using SimplePeer. This example creates two peers in the same page (WARNING: wear headphones to avoid feedback!), initiates a WebRTC PeerConnection between them, pipes the incoming MediaStream into an audio object, then creates a positionalAudio object from it (arrow keys moves the camera for positional effect)

As above, this works in Firefox (with the audio element volume set to 0), but not in Chrome, as far as I can tell…

Any thoughts?

27

Before I dive into your code: Can you please verify if this demo works on your computer?

https://jsfiddle.net/sjvxf4rb/3/

It uses a media stream obtained via MediaDevices.getUserMedia().

This demo works for me in latest Chrome and Firefox.

28

Oh wait, you have already tested this approach earlier^^.

Anyway, if you use the same code and it does not work with your WebRTC stream, than please file an issue here:

https://bugs.chromium.org/p/chromium/issues/list

Looks like a browser issue.

29

:grimacing: okay

Thanks again for looking into this! It is super helpful to know I’m not making some silly mistake.

I think there may be a simple way to dynamically adjust volume of HTML audio elements to make some move towards positional audio that works cross-browser. Have you heard of such an effort? Would love not to rebuild the wheel, if possible.

30

ha, apparently we are working on two similar ideas!

We ended up implementing a workaround for chrome that only implements volume (no panning) based on the distance to each participant.
Works well enough.
It’s basically just

let volume = Math.Min(1, 1 / myPosition.distanceTo(participant.position))
1 Like
31

Good to know! Thanks for the tip. By the way, it appears that the Mozilla Hubs team is dealing with a different set of (potentially related) Chrome audio system issues:

And they have been developing an avatar audio control system, which may be helpful for anyone on this thread:

2 Likes
33

I just came across this post (and answer) on the babylonjs forum, where they seem to have got it to work:

I’m not familiar enough with babylonjs to know what magic happens here behind the scene, but then maybe it’s also doable for threeJS ?

1 Like
34

I figured out what both me and @cat were doing wrong here…
instead of
this.positionalAudio.setMediaElementSource(this.audioElement);
we need to use
this.positionalAudio.setMediaStreamSource(this.audioElement.srcObject);
to make it work in chrome…

1 Like
35

Did that really work @lukasIO? Added on the .srcObject made no difference for me - still no positional audio.

36

No it doesnt work like that for me too what @lukasIO mentions

37

has anyone found a fix to this? tried a few of the methods listed here and it seems to work occasionally, but not consistent at all

my current code:

        const audioElem = document.createElement("audio");
        audioElem.srcObject = stream;
        audioElem.muted = true;
        audioElem.autoplay = true;
        audioElem.loop = true;
        //@ts-ignore
        audioElem.playsinline = true;

        const posAudio = new PositionalAudio(listener);
        posAudio.setMediaStreamSource(stream);
        posAudio.play();
        posAudio.setRefDistance(3);
        posAudio.setDirectionalCone(200, 290, 0.2);
        posAudio.setVolume(0.75);

thinking there might be some race conditions here … maybe I need to mount the positional audio first? or wait for the stream to begin first? idk

38

found the issue here … for some reason an HTML audio element will play no problem with the code above, but the Audio Context of the PositionalAudio object will be in a suspended state unless you resume it after a user input.

Helpful tip … the three.js audio context is shared globally, so a little script like this will solve a lot of your silent audio issues

const onClick = () => {
    const context = AudioContext.getContext();
    if (context.state !== "running") context.resume();
}
window.addEventListener("click", onClick)
39

Its paused already about 5 years…

40

Thanks for this - I was able to fix my issues in react three fiber by using the same trick. For anyone else, the relevant bit of code looks like:

audioRef = useRef(new Audio());
audioRef.current.srcObject = new MediaStream([consumer?.track]);
audioRef.current.muted = false;

sound.current.setMediaStreamSource(audioRef.current.srcObject);
audioRef.current = null;

Using the audioRef to create the srcObject to pass to setMediaStreamSource seems to be the trick, because setting it directly to setMediaStreamSource(new MediaStream([consumer?.track])) doesn’t work.