transformControls with orthographic view

https://jsfiddle.net/rquL46sb/1/
The above example does not include transformControls. I can add if needed.

I have an orthographic view of a cube that I can change (back,side,front).
When the cube is rotated the camera is updated via applyQuarternion to be always looking at the current view of the cube.

My issue is that the when I add transformControls the axis controls are messed up on rotation (when I resize in x direction, it will also increase in y) as the angle of my camera does not match the angle of the controls axis.

How do I update the transformControls axis so when I rotate the object and change move my camera the controls axis move with it?

Before rotate

enter image description here

After rotate

enter image description here