Hi everyone! Built this with my son — a browser-based bird flight simulator where you control the bird with your body via webcam.
Demo video + blog: 3D Bird Flight Simulator with Webcam Controls – Built with AI in One Afternoon (Three.js)
GitHub: GitHub - pmmathias/VogelSimulator: Browser-based 3D bird flight simulator with webcam gesture control & mobile gyroscope support. Fly, land, walk, catch fish! Built with Three.js + MediaPipe. · GitHub
Tech highlights:
- Custom GLSL terrain shader (5 texture layers blended by height + slope)
- GLTFLoader for animated stork model (from the Three.js examples — thank you!)
- InstancedMesh for ~70K trees and 800+ buildings
- Underwater world with fish sprites and coral
- MediaPipe PoseLandmarker for real-time gesture recognition
- Playwright-based automated testing and performance benchmarking
The gesture recognition was the hardest part — went through ~10 iterations before landing on a simple “hands above/below shoulders” binary state approach, which turned out to be much more robust than amplitude-based detection.
Built entirely with AI pair programming (Claude Code).
4 Likes
Nice demo - visually and technically.
However, watching the video reminds me of the old vaudeville joke:
“I just flew here from New York and, boy, are my arms tired!”
1 Like
Really nice project, especially combining gesture control with a full 3D environment. Using webcam pose detection to drive flight mechanics is a clever approach, and the simplified “hands above/below shoulders” logic makes a lot of sense for stability.
Integrating MediaPipe pose tracking with a Three.js simulation is a great example of how computer vision can directly control a browser based world. The terrain shader and large scale instancing for trees and buildings are also solid choices for keeping performance manageable while still giving the environment density.
The underwater layer and animated bird model loaded through GLTFLoader add a nice sense of depth to the world as well.
I’ve been experimenting with similar browser based environments where large scenes stream assets and interact with different input systems. Projects like this are interesting because gesture input opens the door to more natural interaction models for WebGL worlds.
Demo environment example
https://theneoverse.web.app/#threeviewer&&crateria
Would be interesting to see how the gesture system evolves. It could potentially expand into glide mechanics, banking gestures, or even flock style interactions if multiple birds were simulated.
1 Like
Thanks Phil! 
That joke is basically our user testing feedback in a nutshell — my son was exhausted after 2 minutes of flapping!
Btw, thats actually why we added the mobile version with gyroscope tilt controls — much easier on the arms
1 Like
Thank you for the detailed technical feedback, Umbawa!
You nailed it on the gesture detection — we went through ~10 iterations (amplitude-based, direction-change counting, arm-crossing, ducking…) before landing on
the simple binary state check. Lesson learned: the simplest approach often wins.
Banking and glide mechanics are actually already in there — one arm higher triggers proportional banking, and at sufficient speed the wing incidence angle
provides passive lift for soaring without flapping. The flock simulation idea is fascinating though — Three.js has the three.js webgl - gpgpu - flocking with boid flocking, and combining that with a gesture-controlled lead bird could be really compelling.
Definitely on the wishlist!
1 Like
@Umbawa_Sarosong flock interactions are in! 
Just shipped a V-formation system: 24 storks that periodically join the player bird. They drift in from behind, fly in formation for about 10 seconds, then scatter and disappear — repeating every ~30 seconds. Each boid has individual animation phase offsets and organic wobble, and they bank with the leader during turns.
The formation trails behind the player (not ahead), so the chase camera naturally reveals the whole flock during maneuvers. The lazy camera follow (position rate 2.5 vs lookAt rate 3.0) creates a nice cinematic drift effect.
1 Like
I played for 10 minutes and boy are my arms tired! 
Very cool work.
1 Like
When my wife was pregnant, she read “Chesapeake” by James Michener which discussed the migratory habits of Canadian geese, including how they would switch positions because flying lead position is so exhausting. She then had a vivid dream where she was flying with them and remembered how tired she got when it was her turn to lead the formation.
Once I saw a formation of pelicans flying just above the waves - so low that they would climb and descend to match the waves.
So I gave that a try on your simulation. It is pretty neat when the gets so low that you can see the reflection in the water. However, I had a hard time staying at low altitude. If you flap or turn, the bird climbs.
But I was also pleased to discover that you can dive into the water and then fly out again. If you ever try to expand this simulation, you might stock the ocean with fish and allow the bird to dive into the water and - if their aim is good catch a fish.
Also being able to land on the ground would be cool.
1 Like
@phil_crowther What a beautiful story about your wife’s dream. In real V-formations, geese do rotate the lead position for exactly that reason: the upwash from the bird ahead reduces energy cost by ~15%, but the leader gets no benefit. Formation flight is essentially a cooperative energy-sharing protocol. Your wife’s subconscious understood aerodynamics better than most textbooks explain it. Your low-altitude observation is spot on — right now flapping always adds vertical lift regardless of pitch angle, which makes sustained low flight nearly impossible. I’ll rework the flight model so that flapping at a shallow or negative pitch pushes you *forward* rather than *up*. That should make wave-skimming possible — and yes, the water reflection at low altitude is one of my favorite accidental discoveries too. The landing idea is going on the roadmap! I want to implement a proper touchdown + walking mode — so you can land, walk around as a bird, and take off again. Imagine exploring the terrain on foot, finding a good spot, and then launching back into the sky. That would add a whole new dimension to the experience.
Fish catching… that’s ambitious, but I love it. One step at a time: first land, then walk, then fish. 
Thank you for playing long enough to discover the underwater flight — that’s currently a bug I promoted to a feature. The fact that you came back up is the important part <3
1 Like
I look forward to those improvements. Landing is an interesting process for a bird. Like an airplane they “flare” to a soft landing. But, unlike an airplane, they are able to change the shape of their wings.
I think your ocean looks great. But if you want something with more distinct waves, you can use my ocean module. This module is designed to work with landscapes that use square grids (e.g. 1/2 mile). Here is a flying demo. If you drop down close to the water, you can look out on a “sea” of waves in all directions.
1 Like
@phil_crowther You’re fast — and I couldn’t resist. I just shipped a basic landing & walking mode!
How it works:
-
Slow down near the ground and the bird touches down automatically
-
Once landed, you control with WASD (W = forward, S = backward, A/D = strafe) and mouse left/right to turn
-
SHIFT = run faster
-
SPACE (or a flap gesture on webcam) = take off again
I’ll be honest: the landing animation is pretty basic. You’re right that real birds flare beautifully — they reshape their entire wing to maximize drag at the last moment. My current stork model doesn’t have the geometry for that, so the touchdown is more “controlled crash” than “graceful flare.” I think I need a different bird model with more articulated wings to do it justice. Work in progress!
Your ocean module looks absolutely stunning — those waves have real depth and the horizon line is gorgeous. I’d love to integrate it at some point. My one concern is performance: I’m developing on a MacBook (integrated GPU, no NVIDIA), so I need to be careful with shader complexity.
Works great. Since I am a wading bird, I went and stood in the water.
Now if he can just get his friends to land …
Did my ocean module work on your Mac? That is the WebGPU version which uses compute shaders. It is very simplified and does not use cascading or sea foam - just the minimum needed to create a single IFFT geometry. I use textures which cover several grids to add color to the ocean and to hide tiling.
1 Like
@phil_crowther OK so your messages have been dangerously productive for me. Here’s what shipped since your feedback:
Low-altitude flight (your pelican observation):
Flapping at level or negative pitch now pushes ~80% forward instead of up. Plus ground effect within 5m of the surface — 30% extra lift efficiency. Sustained wave-skimming is now stable and genuinely fun. Your pelican memory is now a flight mechanic.
Gerstner waves:
I looked at your ocean module and it’s beautiful work — but it’s WebGPU with WGSL compute shaders, and my sim is pure WebGL. So I couldn’t use your code directly. What I did instead: I added 3 overlapping Gerstner waves (0.15–0.35m amplitude) as vertex displacement in GLSL on top of Three.js’s standard Water class. The 16K vertices of the water plane now physically move up and down, which breaks the mirror-perfect reflections into diffuse, distorted ones on the undulating surface. The existing normal map still provides fine detail ripples. It’s nowhere near your IFFT quality, but the difference from the old flat plane is dramatic — especially during low passes.
What I did NOT use (and why):
Your IFFT ocean module uses WebGPU compute shaders — completely incompatible with my WebGL pipeline. I’d have to migrate the entire renderer to WebGPU to use it, which is a bigger project than the sim itself! So I rebuilt the wave concept in standard GLSL. Your ideas shaped every feature above.
1 Like
I will try out the new improvements. They sound exciting.
FYI - I also have a WebGL2 version of the ocean module (in a program updated to r182). Here is a static demo showing 4 adjacent grid squares. It should look pretty much the same as the WebGPU version.
Okay, I tried the change - why not make flapping add 100% forward thrust?
I realize that, in theory, adding thrust increases speed which increases lift, requiring minor adjustments to pitch. However, the controls in your simulation are not currently sensitive enough to allow the user to easily compensate. For example, flapping causes a large and immediate change to thrust and the controls (I used the arrow keys) do not appear to allow for small offsetting adjustments to pitch. I am not saying that anything is wrong. I am just saying that - with the current control setup - switching to 100% forward thrust would seem to make low flight a lot easier. If I want to climb (or descend), I can simply use the arrow key. Also the turns should not change altitude.
Together, these changes should allow the bird to easily skim and turn over the water. (I can also drown half of the birds in my formation - unless you prevent them from changing altitude with turns.)
All great fun!
1 Like
Thanks for the WebGL2 version — that broke my resistance and I integrated Ocean3.js this afternoon.
Hybrid setup: I kept Three.js’s Water class for the sun reflection and mirror reflection, and swapped its waterNormals for your normalMapFramebuffer.texture. The displacement map goes into the Water vertex shader via string replacement — sampling texture2D(dmap, uv).rgb and adding it to position before the gl_Position line. My plane is 24 km so your 2.4 km wave tile repeats 10× with RepeatWrapping, subdivision 256×256 so the displacement actually deforms the silhouette.
Result is what I was hoping for: FFT wave shape + Water-class sun path. Looking down from a dive, individual crests catch the sun separately — completely different feel from the Gerstner approach I had before.
On the license: kept the CC BY-NC-SA 3.0 header in Ocean3.js verbatim. The project is a hobby/portfolio thing and stays that way. I added a Credits panel in the UI that attributes you and links to http://philcrowther.com/Aviation/3JS_demos.html — happy to redirect to a different URL if you’d prefer.
On your flight-control feedback: 100% forward thrust on flap and altitude-preserving turns, both next on the list. Your formation is safe from drowning.
1 Like
Hi, I’d like to ask for a review of my reply in the bird flight simulator thread (post #16). It got auto-flagged as promotional — fair enough, my first version had blog links and blog-post-style formatting. I’ve since edited it to a plain technical reply to @phil_crowther, who invited me to integrate his Ocean3 module and we’ve been having a productive exchange.
Could someone please take a look at the edited version and unhide it if it’s okay now? Happy to shorten or adjust anything else that triggers the filter.
Thanks for keeping the forum clean.
1 Like
It’s an interesting project, but it’s also a good example of where Three.js projects can drift into being more about tech demos than actual games. The gesture control is impressive from a technical standpoint, but it looks like most of the effort went into proving that MediaPipe + Three.js can be wired together rather than focusing on long-term gameplay or systems design.
The custom terrain shader, instancing for trees and buildings, underwater effects, and gesture recognition are all technically solid, but it also highlights a common pattern in Three.js showcases: lots of impressive tech stacked together, but not much attention to core gameplay loops, optimization for mid-range hardware, or how this scales into something more than a demo.
It’s a strong technical demo, no question. But it also underlines why there isn’t a real “Three.js game ecosystem” yet. People build impressive experiments, but very few of them evolve into stable, scalable game frameworks or reusable systems. That’s the gap. Until more projects focus on performance, reusable systems, and long-term structure instead of novelty tech, Three.js will keep producing cool demos rather than fully fledged games.
1 Like
The author can easily add game elements, if he wants to do so. Some of which have been discussed above. For example, he could create a game which involves trying to raise little birds into big birds. This could be done by adding nests of birds and things like fish to the water which can be caught and brought back to the nest.
But a lot of us are doing technical things that we hope will be useful to three.js game developers, like the IFFT wave generator module that he used. I don’t mind if the author keeps perfecting the interface and mechanics for birds - an area which has not received a lot of attention in the past.
1 Like