Convert webcam pixel coordinates to 3d vector

Questions
1

Position a 3D dress model onto the center of left and right shoulder key points detected from the pose model.

Hi, I detect human body pose from a webcam using TensorFlow pose. I want to place my 3d dress in the center of the left and right shoulder. I’m converting the pixel X Y coordinates to 3D coordinates but the 3d dress in not positioned at the exact location.
My calculations:
let x_avg = (leftShoulderX + rightShoulderX) / 2
let y_avg = (leftShoulderY + rightShoulderY) / 2
mapWebcamTo3D(x_avg,y_avg,0)

function mapWebcamTo3D(webcamX, webcamY, depth) {

const videoFrameWidth = webcamRef.current.video.videoWidth;

const videoFrameHeight= webcamRef.current.video.videoHeight;

const desired3DWidth = 100; // The width of your 3D space in your desired unit (e.g., meters)

// Define the desired 3D space dimensions

const desired3DHeight = (desired3DWidth / videoFrameWidth) * videoFrameHeight;

// Calculate the scale factors for X and Y

const scaleX = desired3DWidth / videoFrameWidth;

const scaleY = desired3DHeight / videoFrameHeight;

// Calculate the 3D coordinates

const x3D = (webcamX - videoFrameWidth / 2) * scaleX;

const y3D = (videoFrameHeight / 2 - webcamY) * scaleY;

const z3D = depth;
}

My 3d Model:

function Model({…props}) {
const group = useRef()
const { nodes, materials } = useGLTF(‘/Red_Blouse_GLTF.txt’)
console.log("model: ", props.dimension.x, props.dimension.y )
return (
<group ref={group} {…props} dispose={null} scale={10} position={[props.dimension.x, -props.dimension.y, 0]} >
// //

}
<Webcam
ref={webcamRef}
style={{
position: “absolute”,
marginLeft: “auto”,
marginRight: “auto”,
left: 0,
right: 0,
textAlign: “center”,
zindex: 9,
width: 640,
height: 480,
}}
/>

<Model dimension={{x:XAvg, y:YAvg }}/>

2

Without a live example that can be manipulated and debugged, it is quite impossible to find the problem.

My only suggestion is the three items below. I’m afraid, I could not be of any further help.

1. First check whether mapWebcamTo3D is good

Find the 3D positions of both shoulders and make some red spheres there. Are the spheres placed correctly on the shoulders?

  • If yes → this is excellent, proceeded to step (2)
  • If no → you must fix mapWebcamTo3D(...)

2. Calculate the midpoint

Because of the properties of perspective projection, it is better to calculate the 3D positions of the shoulders and then to find the midpoint. Create a sphere in this midpoint. It should be exactly in the middle. If not, try to find the cause and fix it.

image
image1011×290 13.7 KB

3. Use your 3D model

Now you can use your model instead of the last midpoint sphere. If the model is shifted (i.e. not well centered), then your model origin point is wrong or it is not designed for your case. Thus, you must either fix the model, or make an adjustment of the midpoint.