Make a opaque mesh without rigid body properties

I want to create a space where the mesh is not going through another mesh.
But when I turn it into a rigid body, it’s governed by the physics rule and the mesh doesn’t seem to move the way I want to (like the below example).

I just want exactly this example (https://codesandbox.io/p/sandbox/mixing-controls-hf1cs?file=/src/App.js:19,2) where I am able to move the cube around the whole space but it should not go through another mesh.

How do i achieve this?

try this https://codesandbox.io/p/sandbox/drag-n-drop-with-physics-and-collisions-9pjzm6

Have tried applying the same property for another rigidbody, but the joint of the pointer and the mesh is not maintained.
https://codesandbox.io/p/sandbox/drag-n-drop-with-physics-and-collisions-forked-8prlmc

where am i going wrong?

and in the above code, if you remove the position from mesh, after the initial movement, the joint gets maintained.
but for initial position both the mesh are at the origin colliding with each other.

another issue, the light-blue box can go through the orange box, but vice versa that doesn’t happen. Even though both are same components.

is this some bug or am i doing something wrong here?