For those who might be interested then definitely try using AVIF files as equirectangular background.
It is rather small file size and the effect appears to be rather good. Some HD image files might look as good as originals.
You could consider converting your EXR or HDR files to separate files by using free @monogrid gainmap-js online encoder to obtain JPEG file with gainmap, which you could then convert to AVIF by using either ezgif or even try my Image to 3D Mesh viewer / converter (whichever gives you a better result - just keep in mind that encoder in my viewer is limited to max 4k resolution).
As an initial test, maybe try using ezgif to convert spruit_sunrise_4k.hdr.jpg file from three.js repository - the AVIF should be approximately 1/4 of the original size.
THREE.TextureLoader should be able to load AVIF files but also try any other HDRJPGLoader like the one from @monogrid.
Cannot really suggest any memory or GPU wise comparison.