Is there a way to extract or create/update a coordinate system that will rotate along with the camera, like on the pic below, from the current camera position/quaternion etc.?
I can do it using up vector, (X = Up x lookAt, Y = lookAt x X) but it fails if the camera is currently looking down world Y axis.
Z → camera.getWorldDirection
Y → camera.up
X → camera.getWorldDirection.applyAxisAngle(camera.up, Math.PI / 2)
You should also be able to transform a Vector3 to local-space by applying Object3D.quaternion (or transform matrix / Euler rotation):
Forward → new Vector3(0.0, 0.0, -1.0).applyQuaternion(camera.quaternion);
Up → new Vector3(0.0, 1.0, 0.0).applyQuaternion(camera.quaternion);
Right → new Vector3(1.0, 0.0, 0.0).applyQuaternion(camera.quaternion);
(It does seem to work nicely when looking directly downwards.)