With some modifications to @Rabbid76 suggestions from here I solved the problem.
Since the camera left, top, right, and bottom values are in world units (see here) , and relative to the camera position, these values are also fixed.
In case of OrthographicCamera, the calculation of the world coordinates of the borders of the camera frustrum should factor in the zoom.
For example, the x world coordinate of the left border is computed with:
let leftBorderX = camera.position.x + (camera.left / camera.zoom);
In the working example here, the image always fills the view window.
The minimal zoom is set to 1, to prevent a case where the image is smaller than the view window
If the zoom is 1, the image covers the view window, and panning is disabled.
If the zoom is bigger than 1, panning is enabled as long as the image covers the view window.