Postion a 3d dress model on to the center of left and right shoulder keypoints detected from posenet

Questions
,
1

Hi, I detect human body pose from a webcam using TensorFlow pose. I want to place my 3d dress in the center of the left and right shoulder. I’m converting the pixel X Y coordinates to 3D coordinates but the 3d dress in not positioned at the exact location.
My calculations:
let x_avg = (leftShoulderX + rightShoulderX) / 2
let y_avg = (leftShoulderY + rightShoulderY) / 2
mapWebcamTo3D(x_avg,y_avg,0)

function mapWebcamTo3D(webcamX, webcamY, depth) {

const videoFrameWidth = webcamRef.current.video.videoWidth;

const videoFrameHeight= webcamRef.current.video.videoHeight;

const desired3DWidth = 100; // The width of your 3D space in your desired unit (e.g., meters)

// Define the desired 3D space dimensions

const desired3DHeight = (desired3DWidth / videoFrameWidth) * videoFrameHeight;

// Calculate the scale factors for X and Y

const scaleX = desired3DWidth / videoFrameWidth;

const scaleY = desired3DHeight / videoFrameHeight;

// Calculate the 3D coordinates

const x3D = (webcamX - videoFrameWidth / 2) * scaleX;

const y3D = (videoFrameHeight / 2 - webcamY) * scaleY;

const z3D = depth;
}

My 3d Model:

function Model({…props}) {
const group = useRef()
const { nodes, materials } = useGLTF(‘/Red_Blouse_GLTF.txt’)
console.log("model: ", props.dimension.x, props.dimension.y )
return (
<group ref={group} {…props} dispose={null} scale={10} position={[props.dimension.x, -props.dimension.y, 0]} >
// //

}

My rendering:

            <Webcam
      ref={webcamRef}
      style={{
        position: "absolute",
        marginLeft: "auto",
        marginRight: "auto",
        left: 0,
        right: 0,
        textAlign: "center",
        zindex: 9,
        width: 640,
        height: 480,
      }}
    />
         <Canvas style={canvasStyle}>
                  <Suspense fallback={null}>
                      <ambientLight />
                      <spotLight intensity={0.2} 
                                 angle={0.1} 
                                 penumbra={1} 
                                 position={[10,15,10]}
                                 castShadow />
                      <Model dimension={{x:XAvg, y:YAvg }}/>
                      <OrbitControls enablePan={true}
                                     enableZoom={true}
                                     enableRotate={true}/>
                  </Suspense>
               </Canvas>

Can someone help me to resolve the issue?

2

Need screenshots of issue. But you should perform a Box3 bounds sizing match. You compute the width of your pixels from camera to the width of you model and scale by that scalar

In the case that the garment is not symmetrical youll need to know what relative points are on the model and pull them to your pixel space and smoothly tweak the vertices like in a 3d mesh editor app

4

Can you suggest three.js built-in functions for doing this?

Can you suggest three.js built-in functions for solving this?

5

box3
https://threejs.org/docs/index.html?q=box#api/en/math/Box3
the other idea is math, youll have to figure it out. Observe soft selection mode in blender. There is no available function from three