Is it possible to create a texture atlas that is composed of basis compressed images that would be usable by THREE, while still getting some memory savings on the GPU? If this is possible, how much memory would be saved?
Context: I have a bunch of images that are all already individually basis compressed. I also am currently constructing atlases on my server using uncompressed images, and then basis compressing the entire atlas. I’d like to move the atlas creation to the client. I know that it is possible to use render targets to create an atlas texture directly in THREE using uncompressed images. However, the output atlas is also uncompressed, which dramatically increases the memory load on my client GPU. I’m looking for ways to generate atlases on the client but retain the memory savings inherent to using basis compressed images.
Generating Basis-compressed textures on the client will be very slow and lossy — decoding, merging, and then re-compressing. three.js doesn’t provide tools to do this, but you can find WASM implementations in Basis Universal or KTX-Software. I don’t think two Basis textures can be merged without lossily re-compressing them both, but you could ask at the Basis repository to be sure.
In general the idea is to do compression offline, and here that would mean also generating the texture atlas offline.
I figured that there’s no world where we pursue decoding, merging, and recompressing. Either we do it all on the backend, or we find a way to merge basis textures (if that is possible).
(For completion, we COULD send the client uncompressed files and basis compress them client side, but I don’t think it would make a lot of sense for our use case; still, thanks for the links to the WASM binaries)
Just to compile some more of my team’s research, we did find a webgl api call (WebGLRenderingContext.compressedTexSubImage2D() - Web APIs | MDN) that allows for handling a subrectangle of a compressed texture, which could possibly be used for generating atlases?
There are some THREE examples of partial texture updates using uncompressed textures, e.g. three.js webgl - texture - partial update but I don’t know if this would work with compressed textures as well
Ah that’s a great point – you don’t necessarily need to merge the Basis-compressed data, because KTX2Loader will be transcoding it to something some target format with the client device anyway, like ASTC or ETC2. So something like this could work:
allocate a 4K ETC2-compressed texture in GPU memory
transcode N textures to ETC2
upload each of the N textures to specified regions of the atlas
The target formats will depend on the device — see the two developer guides below for details on that:
I ended up punting on this particular implementation – there were other bottlenecks in my code at the time. But a year later I’m back on the same question.
Looking back at the original question that I asked, I wanted to provide a bit more context. I have a large dataset of images that I want to render to the user. The dataset increases frequently (realistically capped at like 200k, but I would love to support unbounded data) and the coordinates that images appear at can change, potentially frequently (so we cannot easily rely on, say, octree-type implementations to load pre-baked data at varying resolutions).
My current approach to solving this issue is to precompute a bunch of atlases at different resolutions and basis compress all of them. This gets me surprisingly far – with basis compression I can have something like 50 4096x4096 atlases loaded in at a time – but there are a lot of obvious seams. The atlases themselves are pretty slow to create, and atlases as a data structure feel really unwieldy.
My original question was geared towards trying to avoid storing atlases as a pre-baked structure. The thought process – and the true problem statement – is something like: given a function that takes the camera position and returns a list of images that need to be rendered within a certain radius, can I generate atlases dynamically and load it into memory fast enough that it’s not super noticeable to a viewer (say, within a second or two).
So far all of my searching for an elegant solution has come up empty. Constructing an atlas takes a while just from image io; compressing that atlas takes a while; sending the bytes over the wire to the browser isn’t free either. But recently I noticed Add THREE.CompressedArrayTexture by RenaudRohlinger · Pull Request #24745 · mrdoob/three.js · GitHub was merged and I started wondering if creating and loading a texture array would be fast enough to pull this off
This is basically a very roundabout way of asking if any of the changes in THREE in the last year make this task easier, or to gut check if there are other trees I should be barking up in the graphics/rendering world that I might not know about yet. @donmccurdy hope you don’t mind if I tag you, since you responded above already and were pretty involved in the linked pr
Let me try to paraphrase the goals first: You have a very large number of textures available, and would like to be able to render as many of them as possible at the same time. Offline compression is OK, but you don’t necessarily know in advance which subset of the textures will be rendered, and so ideally compression would not pre-bake to a subset like an atlas.
Is that much correct? If so — I do think using THREE.CompressedArrayTexture sounds like a good idea. The general approach would probably be:
Compress each texture individually with the same compression type (e.g. BasisLZ ETC1S) and resolution
Load textures on the fly with THREE.KTX2Loader, which transcodes them to a GPU-compatible format
Assemble the transcoded textures into a THREE.CompressedArrayTexture object
The THREE.CompressedArrayTexture object should be pre-allocated for a fixed size if you plan to add images later. I’m not sure whether three.js currently supports updating specific indices of the array texture individually, or if the whole array texture is re-uploaded when it changes. You could inspect the KTX2Loader source code for an example of how to construct an array texture.
I think I understand how to follow the general approach – certainly 1 and 2 make a lot of sense. For step 3, ill definitely have to dive into the loader source, thanks for the pointer.
Do you have a sense of whether there is a size difference in terms of GPU memory usage here? I’m not as familiar with basis compression under the hood, but I assume that the output of basis_compress(array(png_list)) is smaller than array(basis_compress(png_list)). If it’s many multiples smaller, would this be prohibitive?
I believe the VRAM cost would be identical, since the basis textures must be transcoded to formats like ETC1 or ASTC on the GPU anyway. File size over the network might be more or less but I wouldn’t expect a huge difference.
I took a stab at assembling the transcoded textures into THREE, but I’m getting essentially just a black square. Code snippet below, embedded into the THREE CompressedTextureArray example:
// Set up the loader.
const ktx2Loader = new KTX2Loader();
ktx2Loader.setTranscoderPath('jsm/libs/basis/');
ktx2Loader.detectSupport(renderer);
// Load two ktx images separately. Both are the same size (128x128).
const im1Promise = new Promise((resolve, reject) => {
ktx2Loader.load('textures/im1.ktx2', function (im) {
resolve(im);
});
});
const im2Promise = new Promise((resolve, reject) => {
ktx2Loader.load('textures/im2.ktx2', function (im) {
resolve(im);
});
});
// Wait for both images to load, then construct the texture array.
Promise.all([im1Promise, im2Promise]).then((values) => { [35/22538]
console.log('values!', values);
// Try and merge the mipmaps of the values by just concatenating them...
// const mipmaps = values[0].mipmaps.concat(values[1].mipmaps)
// Or try to merge the Uint8Arrays of the two loaded ktx2 files by concatenating
// the actual bytes
// Based on: https://github.com/mrdoob/three.js/commit/32c1a67e2de98d54323975ce0fa63ce69b5998b3
const data = new Uint8Array(
values[0].mipmaps[0].data.length + values[1].mipmaps[0].data.length,
);
data.set(values[0].mipmaps[0].data, 0);
data.set(values[1].mipmaps[0].data, values[0].mipmaps[0].data.length);
const mipmaps = {
data,
height: values[0].source.data.height,
width: values[0].source.data.width,
};
// Actually construct the array texture by pulling the data from the source inputs
const texarray = new THREE.CompressedArrayTexture(
mipmaps,
values[0].source.data.width,
values[0].source.data.height,
values.length,
values[0].format,
THREE.UnsignedByteType,
);
// Copied from the original example.
console.log('texarray', texarray);
const material = new THREE.ShaderMaterial({
uniforms: {
diffuse: { value: texarray },
depth: { value: 55 },
size: { value: new THREE.Vector2(planeWidth, planeHeight) },
},
vertexShader: document.getElementById('vs').textContent.trim(),
fragmentShader: document.getElementById('fs').textContent.trim(),
glslVersion: THREE.GLSL3,
});
const geometry = new THREE.PlaneGeometry(planeWidth, planeHeight);
mesh = new THREE.Mesh(geometry, material);
scene.add(mesh);
});
I’m guessing that I can’t just merge the Uint8Arrays the two ways I tried above.
Hey hi, yes, it’s possible to create a texture atlas using basis compressed images for THREE.js and compress image by https://jpegcompressor.com/, maintaining memory savings on the GPU.
