my way to solve this was to set controls.enabled = false when the canvas.on('selection:update') fired. The issue was because of the orbit controls still moving while you also move the fabric object. anyway take a look at my demo 3D Merch Configurator