Camera isn't transformed as expected

Hello! I’m trying to create a custom camera that is similar to a 2D renderer camera. This camera isn’t based on the Orthographic or Perspective cameras because I want the camera to take a width, height, and zoom level value (from 0 to 1) instead of a frustum. For some reason the camera translation isn’t working as expected but zooming is working. What’s interesting is that 3d geometries seem to account for translation but 2d geometries (try uncommenting the BoxGeometry in the demo).

To test out the camera I’ve built a small demo that lets you transform the camera coordinates by panning and zooming (via drag). I’m trying to get the camera to zoom and pan similar to this d3 example: https://next.observablehq.com/@d3/zoom-canvas-rescaled.

Any help is appreciated!

Codesandbox Demo:

The Code

import * as THREE from "three";

const scene = new THREE.Scene();
const camera = new THREE.Camera();

// PlaneGeometry zooms but translation seems to have no effect
const geometry = new THREE.PlaneGeometry(100, 100);
// Translate seems to have an effect on BoxGeometry
// const geometry = new THREE.BoxGeometry(100, 100);
const material = new THREE.MeshNormalMaterial({ color: 0xffffff });
const cube = new THREE.Mesh(geometry, material);
cube.position.set(0, 0, 0);
camera.lookAt(0, 0);
scene.add(cube);
camera.translateX(1);

const renderer = new THREE.WebGLRenderer();
renderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(renderer.domElement);

const origin = {
  x: 0,
  y: 0,
  scale: 1,
  transform: new THREE.Matrix3()
};

function updateTransform() {
  origin.transform = new THREE.Matrix3();
  const transform = origin.transform;
  const elm = transform.elements;

  // projection
  transform.scale(2 / window.innerWidth, -2 / window.innerHeight);

  // // translation
  elm[2] += origin.x * elm[0] + origin.y * elm[1];
  elm[5] += origin.x * elm[3] + origin.y * elm[4];

  // scale
  transform.scale(origin.scale, -origin.scale);

  const [a, b, c, d, e, f] = transform.elements;

  // convert to mat4
  const projectionMat4 = new THREE.Matrix4();
  // prettier-ignore
  projectionMat4.set(
    a,b,c,0,
    d,e,f,0,
    0,0,1,0,
    0,0,0,1
  );
  camera.projectionMatrix.set(...projectionMat4.transpose().elements);
}

function translateOrigin(deltaX: number, deltaY: number) {
  origin.x += deltaX;
  origin.y += deltaY;
}

function zoomTo(x: number, y: number, ratio: number) {
  origin.x += (x - origin.x) * (1 - ratio);
  origin.y += (y - origin.y) * (1 - ratio);
  origin.scale *= ratio;
}

function zoom(x: number, y: number, delta: number) {
  zoomTo(x, y, Math.pow(0.8, delta));
}

function draw() {
  updateTransform();
  renderer.render(scene, camera);
}

draw();

let mouseIsDown = false;
function handleMouseMove(deltaX: number, deltaY: number) {
  translateOrigin(-deltaX, -deltaY);
}
renderer.domElement.addEventListener("mousedown", () => {
  mouseIsDown = true;
});
renderer.domElement.addEventListener("mouseup", () => {
  mouseIsDown = false;
});
renderer.domElement.addEventListener("mousemove", (e) => {
  if (mouseIsDown) {
    handleMouseMove(-e.movementX, -e.movementY);
    draw();
  }
});

renderer.domElement.addEventListener("wheel", (e) => {
  if (e.ctrlKey) {
    zoom(e.pageX, e.pageY, e.deltaY);
  }
  draw();
  e.preventDefault();
});