Map a object exectly on the frame of another object

Questions
1

Hi,

My Project is like a Virtual try on, I have trained a computer vision model to detect the human foot and provide me a segmented mask of the foot then I convert the segmented 2d pixels (outline of the foot) into a 3d points array and then create a buffer geometry of this array now I want to align my 3d shoe object on the exact way of the segmented mask which is converted into an object.

As both of the objects are buffer geometry their default mesh position and rotation are zero. how can I impose my shoe object to take the shape of the segmented mask?

I tried several methods:

  1. The foot-segmented mask object gives me a bounding sphere and from its center point I can position the foot on it but have no clue how to find rotation.

  2. Calculate three distinct points of the foot-segmented mask and try to set an object inside it. like this method: https://codepen.io/boytchev/full/GRzagBW
    image

But when I tried to do it for the shoe, the shoe direction was not set as from the mask we can not predict the toe point or heel point I don’t know how to set the direction of the shoe right:
image

I need to find a solution that I can use to map the shoe on the segmented mask and the shoe will set the direction position and rotation similar to it accordingly.

Any help in this?

2

The answer to your question depends on how you do this conversion. To align a 3D object to a 2D image, you’d need some depth estimation.

3

The first phase use some computer vision models like YOLO and Sagment anything Model (SAM MetaAI) which detect the foot on the scene and send me a segmentation mask or you can say an outline of the pixels of the foot
image

then I used these pixels array and did the conversion of the pixels for the 3d world using raycasting like this: for (let i = 0; i < arr.length; i++) {
mouse.x = (arr[i][0] / window.innerWidth) * 2 - 1;
mouse.y = - (arr[i][1] / window.innerHeight) * 2 + 1;
planeNormal.copy(camera.position).normalize()
//console.log(‘hello 23’,planeNormal,scene.position)
plane.setFromNormalAndCoplanarPoint(planeNormal, scene.position);
raycaster.setFromCamera(mouse, camera);
raycaster.ray.intersectPlane(plane, point)
positions.push(point.x, point.y, point.z);

}

Then pass this postions array to the buffer geomtry:

    <bufferAttribute
      ref={boxRef}
      attach='attributes-position'
      array={positions}
      count={positions.length / 3}
      itemSize={3}

    />
  </bufferGeometry>

I got the exact outline from this method:
image

How can I use the depth estimation method to figure out the point of intersection of the shoe and foot and the second point to the tip of the shoe as I don’t know how to set the directions?

4

Your AI solely relies on color. As mjurczyk stated, you need depth in order to correctly calculate (estimate) the perspective which you’ll need to apply an image. You simply don’t have enough data, unless your AI can also do this for you.

Also don’t forget that this applies to the shoe image as well. There’s a lot more to it than just “slapping an image on top of another image automatically” I’m afraid :man_shrugging:

5

Hi, As of my knowledge depth can give me near and far points from the camera but my problem is to find the positions of the foot tip and point of intersection with the leg when I have it I can perform the positioning and then draw a way for occlusion that is the next step for the AR effect. Or I am thinking of cutting the foot mask from the image and getting the whole array instead of just the outline of the pixel coordinates and then converting them into a 3d foot object like this:
image
Then If I use the UV mapping concept to wrap a shoe object around it is it possible? Or even for the UV map concept we need two distinct points?