Rotate a plane around a pivot point without using group

Hi, I am trying to rotate a plane around a pivot point. Things are not working good. When I change the pivot point with the object having some kind of previous rotation then the object moves with the pivot point in a weird manner. I think to some extent I know why this is happening but not entirely sure. The key point here would be: it only happens when there is some previous rotation.

fiddle: https://jsfiddle.net/2zqL17cx/1/
( use right mouse btn to change the pivot point and click on the silver thing to rotate/scale)

this is how I am doing it:

var matrix = new THREE.Matrix4();

  matrix.setPosition( mesh.pivot );
  matrix.multiply( new THREE.Matrix4().makeRotationFromQuaternion( mesh.quaternion ) );
  matrix.multiply( new THREE.Matrix4().scale( mesh.scale ) );
  matrix.multiply( new THREE.Matrix4().setPosition( mesh.pivot.clone().negate() ) );
  matrix.multiply( new THREE.Matrix4().setPosition( mesh.position ) );

  mesh.matrix.copy( matrix );