Bird flight simulator with webcam gesture control — Three.js + MediaPipe

Just a follow up regarding your use of webgl2 with a Mac. I have just upgraded the demos of my webgpu and webgl Ocean modules to r184. I was pleasantly surprised to find that both webgpu versions (fragment shaders and computer shaders) work on my i-Phone. Oddly the webgl2 version does not. Now that Apple has decided to embrace webgpu, I wonder if they have abandoned webgl2?

Thanks Phil, good to know. My iPhone already lands on the Gerstner fallback right now. The iFFT path fails silently on Safari, so my auto-fallback kicks in (I added a little “iFFT /
Gerstner” pill in the top bar so I can tell which one is running at a glance). The Gerstner still looks decent, just without the FFT crest detail.

Your WebGPU observation makes me think I should upgrade the whole thing once you have a stable WebGPU Ocean module ready. If you ever publish that as a drop-in replacement for Ocean3.js, let me know :slight_smile: :folded_hands:

I have two WebGPU versions - one that uses fragment shaders and one that uses compute shaders. Although the compute version is supposed to be faster, the fragment version currently runs faster on my iPhone.

Here is a link to my discussion that has links to all 3 versions.

And here is a webpage that describes how to implement the modules in your program.

1 Like

Oh awesome — thanks for the links!
Super interesting that the fragment version is actually faster than compute on iOS — I would have assumed the other way around. Good data point. If we do the
WebGPU upgrade on my end I’ll start with the fragment version then.

Will report back when I’ve tested. Honestly, even if I don’t switch everything over immediately, it’s great to know the path forward exists. Appreciate you
keeping me in the loop!

1 Like

on it :slight_smile: birdybird

Idea is that the first person bird has to bring stuff from trees and worms to the nest. Open for Ideas / Input!

Hi @phil_crowther,

Ocean4 is now live in birdybird - WebGPU path kicks in automatically on modern Chrome/Safari (append ?renderer=webgl for the pre-Ocean4 Gerstner fallback). Attribution in the credits overlay and in src/vendor/Ocean4.js (CC BY-NC-SA 3.0 header preserved).

One small note for your README in case it helps others: your normal map is world-space (packed nrm3.x, nrm3.z, nrm3.y), not tangent-space - feeding it into TSL’s normalMap() helper flattens the perturbation and renders the ocean as a perfect mirror. Decoding directly in colorNode works: normalize(vec3(nTex.x, nTex.z, nTex.y)).

Thanks again — Ocean4 is the centerpiece of the WebGPU port and reads
beautifully at any height.

Cheers,
Mathias

I’m glad to hear that you were able to use it. Credit for much of the inner mechanics of the program belongs to Attila Schroeder. He basically wrote the code for the WebGPU versions.
I’m not sure I understand your comments about the normal map. My intent was simply to create a standard colored

Here is a program that shows the “inner workings” of the program (I had to adjust some of the values to make them visible). The last square shows the normal map. Although a bit hard to see, it is supposed to have the standard coloration of a normal map.

The command I am using in TSL to add this normal map to the texture is this:

normalNode: normalMap(texture(grd_.Nrm),grd_.NMS),

where grd_.Nrm is the address of the normal map and grd_.NMS is a variable used to invert the map - if necessary.

Are you saying that I should be using a different command?

I have not had problems with the waves being flat in my programs.

p.s. I just looked at birdybird (linked 2 messages above), which I assume is the latest version of the program. You have done a great job with the ocean. My biggest concern with using a non-cascaded wave generator the risk of visible tiling. But I did not see any of that in your program, even using a PC monitor.

Just for fun, I tried running it on my iPhone. The setup worked great. But, when I got to the Flugelschlag! screen, it stopped. (I have an older iPhone with the latest iOS).

1 Like

Hi @phil_crowther,

Apologies — I was overstating that. You’re right, your map has standard normal-map coloration and normalMap(texture(...), NMS) is the correct way to use it. What we actually hit was specific to our setup, not your module:

We were using MeshBasicNodeMaterial (unlit) for the water instead of MeshStandardNodeMaterial, and we UV-scaled the plane by ×4 for repeat-tiling across a larger horizontal area. In that combination the tangent-space decoding inside normalMap() didn’t drive our custom colorNode the way we’d expected - so we fell back to reading the RGB texels directly and treating them as axis-aligned world-space offsets (which works because the water plane is axis-aligned in world after the -π/2 X-rotation). The problem is on our side; nothing to change in Ocean4 or your docs.

Also — thank you for calling out Attila’s contribution. I’ll add him to the credits overlay in the next deploy; src/vendor/Ocean4.js already carries his line in the header but he should be named in the visible credits too.

On the iPhone crash at “Flugelschlag!”: thanks for trying it! That screen is the tilt-calibration prompt, which needs DeviceMotion permission. On older iOS versions that permission request sometimes doesn’t surface the dialog, or WebGPU init fails silently and we never recover to the WebGL2 fallback on the mobile path. I’ll dig in. If you’re up for it, could you tell me which iOS version + iPhone model? That helps a lot.

Thanks again for Ocean4 and for the pointer to Attila.

Cheers,
Mathias

Glad that my normal map is okay.

If you ever need an Ocean that is more complex, here is the full-featured version that Attila created. His GitHub Repository is here. As you can see, he has worked on a lot of complex projects - most of which I barely understand. :confused:

Regarding your program - on my iPhone:

  • I got the prompt advising me to turn the phone sideways;
  • I got the prompt requesting permission regarding motion capture - I said “yes”
  • It then went though a series of displays with arrows and other characters - which I assume was a description of commands.
  • After that, I got the Flugelschlag! screen and nothing more.

I know a bit of German so I don’t think I did anything wrong. But there may have been an instruction I missed.

My iPhone is an iPhone 13 with iOS 25.4.1. I have about 14GB of free memory.

Hi Phil,

exactly the info I needed. Bug found and fixed (deployed a few minutes ago):

On iOS 13.4+, DeviceOrientationEvent.requestPermission() and DeviceMotionEvent.requestPermission() are separate permissions. The
“Motion & Orientation Access” dialog you said yes to was granting only
orientation. Without the motion permission, devicemotion events never
fired → the shake-detection Promise on the Flugelschlag screen had no
fallback and simply hung. You didn’t miss an instruction; the app was
genuinely stuck.

Fix is two parts: (a) we now request both permissions explicitly, and
(b) the shake step has an 8-second timeout + a “Skip” button so no one
ever gets stranded there again. Could you give it another try when you
have a moment? Same URL - you may need to re-install the home-screen
icon since iOS caches aggressively.

Thanks also for the link to Attila’s full-featured Ocean — I’ll have a
look. And I’ve already updated birdybird’s credits overlay to name him
alongside you, with a link to his three.js forum profile.

The changes apparently helped. I got past all the screens and into the game.
However, once there, all I could do was spin in place with the bird standing vertically (like when on the ground.) I could stop the spinning by tilting my phone, but nothing else.
After awhile it froze and a little semi-transparent line appeared in the upper left hand corner - perhaps debugging data? Here is a screen shot.


Everything seemed to be working well in the background. I just couldn’t go anywhere.

1 Like

Great news that the calibration fix got you past the Flugelschlag
screen! And your “spinning in place, standing vertically” description
was again a perfect diagnostic — it told me exactly what was broken.

The bird spawns at altitude with velocity = 0. Without any flap (on
mobile: a shake gesture), it stalls instantly and falls to the ground,
transitioning into GROUNDED mode. In ground mode the tilt input maps to
yaw-in-place turning, and the bird’s model stands upright — which is
exactly the “spin in place, standing vertically” you saw. So the bird
never actually flew; it crashed within a second or two and you were
standing on the grass.

Fixed (deployed just now): the bird now spawns with ~18 m/s forward
glide speed. That gives you ~10-15 seconds of airtime before you have
to flap, which should be plenty to learn the shake gesture.

To flap: shake the phone firmly (the wrist-flick motion from the
Flugelschlag calibration screen). Each shake gives a burst of forward
thrust and a bit of climb — a real bird’s wingbeat.

Two other tips for mobile:

  • Tilt left/right to steer (roll)

  • Tilt forward to dive, back to climb

  • W key (desktop) = dive; on mobile, tilting forward past the calibrated
    rest angle does the same

Also thanks for flagging the faint line in the upper-left corner —
that’s our keyboard-debug HUD. It only appears if no DeviceMotion
events are firing within a window (so the code fell back to “assume
keyboard”). If your shakes ARE being detected it should be gone. If it
stays visible on retry, that would point to the motion-permission grant
not actually sticking on your iOS version, which would be worth knowing.

Could you try one more time? I also added a ?seed=N URL param so
you and I end up on the same procedural world — before this every
browser got its own random terrain and I couldn’t tell whether some
obstacle near your spawn was aggravating things. Here’s a known-good
seed I’ve test-flown:

https://pmmathias.github.io/birdybird/?seed=42

Should see a clear glide, and if you can shake the phone, proper
powered flight.

Okay, I got it to working. You are right, I didn’t know how to flap my wings.

I think one reason for the spinning is that the bank control seems backward - at least to me.
When I tilt the phone left, the bird goes right and vice versa. I would suggest reversing that.

It might need a bit more of a “dead zone” so the bird won’t respond to tiny motions. Alternatively, you could use non-linear controls - where the controls are less sensitive at small angles and more sensitive at large angles. Also, the part where bird turns completely sideways is too much. I would not let him turn that far.

I am impressed with how smooth it runs. All that scenery and no stuttering!

When the bird lands, I don’t think you have to change the orientation. Instead of stopping with his nose in the air, You could have him running/walking with his wings in.

To take off, you could tilt the phone downward to start him moving faster and, when he is going fast enough, tilt the phone upward to have him take off. Or shake the phone to make hi flap - your choice.

Landing could be a fun exercise. Get close to the ground, and let speed bleed off. And then tile the phone quickly in the updards direction so that he tilts his wings way back to slow down completely. This is similar to how you land a small airplane - you left speed bleed off and keep tilting the airplane back (“flaring”). If you do it just right the airplane will stall just inches above the runway. Birds have the advantage of being able to rotate their wings completely. So they can do that inches above the ground to slow themselves down quickly and to kill lift at the same time. Some even flap their wings while in that posture to help fine-tune drag and lift. But that would probably be too complex to implement.

Thank you sou much for testing!

The calibration wizard learned a wrong axis sign and there was no way for the user
to notice before takeoff (in my older iPhone the issue did not occur).

Just shipped a redesign:

  • Each calibration step now shows a live β/γ dot in a labelled
    gauge
    plus a phone-icon that mirrors your tilt — so you can see
    whether your phone is reporting data and which direction it reads
    as.

  • Final step before saving is a test-fly preview: a bird that
    banks/pitches with your live tilt through the just-computed
    profile. If left-tilt makes the bird bank right, you see it
    immediately and hit ↻ Recalibrate — no commit until you click
    ✓ Looks right.

One ask for the rerun: when you reopen the page you’ll likely hit a
“Calibration found — Use previous / Recalibrate” dialog. Please pick
Recalibrate so the old broken profile gets replaced.

Your other points (bank inversion vs reversed-axis tuning, dead zone,
non-linear curve, max bank cap, walking landing pose, tilt-takeoff,
flare landing) — all logged as tickets, will work through them over
the next sessions.

Sorry I missed this reply. I have been busy working on my own flight demo (a Sopwith Camel).
I tried the simulator again and I am able to fly okay. I suppose one gets used to the bank being reversed - like you can get used to pitch being reversed.

It is encouraging that you are able to get this kind of simulation to work with an iPhone. The level of detail is amazing.

Unfortunately, it still freezes after only a few minutes. Is there a chance of a memory leak?

Also, someday you will want to add instructions in English.