How to overlay 3d scene over 2d video feed

Questions
1

i am to make a T-shirt try-on app , am using mediaPipe for this ,each render loop I give a frame to mediaPipe and it returns a array of points like this `

index 0 - nose
index 1 - left eye (inner)
index 3 - left eye (outer)
...and so on 

each point has three values like this 


x,y - normlize to video dimensions ,(think this might be in ndc space of the input frame)
z - depth

my goal is to move a rigged model using these points in 3d ,how can i overlay a 3d obj form scene to on top of point index 0 in screen scape , is there a way to transform the 3d scene as to match it with real life video feed captured by the camera

2

You’ll need computer vision for this to estimate human pose from images/video. Once you have that, come back, the Three.js part is the easy part.

3

I have the human pose estimate using media pipe it returns me an array of 33 pose Landmarks, but landmarks have x , y normalise to images/video res , whant to know how can i project this x,y on a 3D scene so it overlaps on x , y that i get form images/video world

4

Ah, for that you need have to do some math with the camera frustum. You need to calculate how far away from the camera one unit will equal to one pixel.

So, if your video is 1024x768 for example, then your scene will have the same aspect ratio, and you need to position objects far enough away from the camera so that 1 unit left/right/up/down will be 1 pixel in the video.

This might be of use:

There are probably other ways to do it, but if you use that function linked there, the object you will fit into the screen should be 1024 units wide, then you’ll have the result for the distance.

Basically its just doing trigonometry with the camera frustum.

That’s one thing. Another thing is, if you want to match 3D objects in the scene with 3D objects in the video, then you need to match the video camera frustum to the scene frustum so that the proportions and perspective will be the same, etc.

5

thanks for the help will try this

6

Hi, did you able do the conversion. Could you pls help with that. I am also trying to do the same

7

If you have sample code maybe someone can help. You could start by making a sample forked from this CodePen Three.js starter: