I was not able to grasp completely the logic in your code, so I simplified the dragging to fit to my way of thinking.
Is it something like this what you want to achieve (watch the yellow frame)?
The code is here: https://codepen.io/boytchev/full/NWVVvBz. I spent too much time on this. I hope it could at least give you some hints.
To be honest, if I were to drag an object in a room, I’d prefer to use navmeshes.