Hi all,
I wanted to share something a bit different from the usual showcase. This is a robot arm that is actually being driven by a trained neural network, with real physics, entirely client-side. No server, no streaming, no video playback. The brain and the body both run in your tab.
Live demo: https://lerobot.binhph.am/
What you are looking at
The SO-101 arm at the top of the page is controlled by an ACT (Action Chunking Transformer) visuomotor policy I trained in LeRobot to pick up cubes. Every frame, the model reads the arm’s joint state plus two camera images (a wrist cam and a side cam rendered live from the scene), and predicts the next chunk of joint targets. Those targets drive the arm. So the motion you see is genuinely the policy reacting to what it sees, not a baked animation.
The physics is MuJoCo compiled to WASM. The rendered geometry is the physics geometry, so contacts, grasps and the cube tumbling are all real simulation rather than scripted.
The three.js side
three.js (via React Three Fiber + drei) handles everything you actually see: the lighting, the desk, the HDRI-based image lighting, contact shadows, and the two render-to-texture camera feeds that get fed straight into the policy as observations.
The stack
- three.js r184, React Three Fiber + drei (rendering, HDRI IBL, contact shadows, live camera RTT)
- MuJoCo 3.10 (WASM) for physics
- onnxruntime-web for inference. WebGPU + fp16 where available, multi-threaded WASM fallback otherwise (Safari path). The model is ~16M params, ~33MB fp16
- Cross-origin isolation (COOP/COEP) so SharedArrayBuffer threading works, which takes the WASM path from ~2.7s to ~0.4s per forward
- Trained with LeRobot; the dataset and weights are public on Hugging Face
Why I built it
The page is also a written survey of basically every visuomotor policy and inference regime in LeRobot (ACT, diffusion policy, pi0, SmolVLA, the VLAs, reward models, and so on). I wanted the top of the page to not just describe what these policies do but let you watch one run live, so the live arm is the hero and the writing is underneath it.