Orthographic Camera -- Zoom at Cursor

I’m trying to implement zoom to cursor with an OrthographicCamera and OrthographicTrackballControls. I disable zoom in OrthographicTrackballControls and use the following function on mousewheel:

zoomDirection  = new THREE.Vector3();
function mousewheel(event) {
            var amount = event.deltaY / 100;
            var zoom = camera.zoom - amount;
            var factor = 1;
            var offset = el.offset()
            var mX = amount > 0 ? 0 : ((event.clientX - offset.left) / renderer.domElement.clientWidth) * 2 - 1;
            var mY = amount > 0 ? 0 : -((event.clientY - offset.top) / renderer.domElement.clientHeight) * 2 + 1;

           zoomDirection.set(mX, mY, 0.5)
                .multiplyScalar(amount / zoom)


           camera.zoom = zoom;

This seems to work at first: the camera zooms into the mouse point, but then the camera starts to “jump” around after a bit of zooming, with the mesh no longer visible on screen.

Something that might help: I have an axis helper in the screen as well that “flips” when it stops working as expected. When the scene is loaded, the X-axis helper point due left, but when I get to the point where the camera jumps and I no longer see the mesh, the X-axis helper sometimes (but not always) flips to point due right.

Also, if I zoom OUT first, I can zoom in further before the mesh disappears. I’m not sure what this all adds up to but I would appreciate any help.

It’s hard to help you based on the information you’ve provided so far. Please add a live example that demonstrates the issue.