Raycaster drags 2 objects if at the same distance

Hello! I am trying to use the raycaster on one selected object.
I’ve attached a fiddle. There are 2 buttons, each creating the same geometry but with a different color. If you create a red object, you will be able to drag it while your cursor is on the object and while pressing down the left click. If you add a green object and drag the red one on top of the green one (or vice versa) they will overlap, sharing the same space thus making the raycaster able to move both at the same time. I don’t want this to happen. I don’t want them to overlap eachother.
How can i restrict the raycaster so you are only able to drag one object while the left-click button is pressed?