My project i2i is more or less working at this point: https://i2i.chat/
BodyPix / Seriously.js background removal places video chat in a “room”.
Lance / Three.js / cannon-es / three-to-cannon real-time multiplayer 3D physics table.
You can drag-and-drop models similar to threejs editor and they’ll appear on the table with physics/animation. Many thanks to @mrdoob @donmccurdy @Mugen87 for all this wonderful code and help. Comments / bug reports welcome!