How to rotate camera between quaternions on a plane?

I have two camera positions/quaternions pairs that I know and I want to move/rotate camera between them on a plane.

In both positions camera is looking at the origin, so I made a quaternion out of two forward vectors and I can rotate positions.

This doesn’t account for the rotation of the camera on its local XY plane, so I thought I could use slerp between quaternions but the cameras do not look exactly at the origin.

I also built cameras w/o translation to look at the slerp and it seems like slerp transition doesn’t lie on a plane, so that small “fan” of cameras is not flat.

Is there something I’m missing here or is there a simple way to fix this problem?

big cameras - start / end positions/rotations,
yellow line - quaternion vector,
small cameras - transition steps,
“fan” of cameras at z < 0 hemisphere - transition cameras w/o translation based on slerp.