Looking for ideas on how to create this interaction with object limits and dragging

I hope someone will know of some solution for my game idea to work! :slight_smile:

I am recreating a simple game to get the block out of the closed area. The wall and other blocks are limits to each other and so if they have a collision they should stop. The user will be dragging the blocks to figure out the puzzle.

The issue is that I can drag and it stops at the other object limit but I can also just keep dragging through the object and come out the other side.

Any thoughts or direction would be greatly appreciated.