Optimal way to load 16-bit data as a texture

Questions
1

Hello all,
Firstly, thank you all who have contributed to this project, it is nothing short of amazing.

I have been using Three.js for my little photography project, and I’m at a point where I would like to trim as much fat as possible.

Since camera raw files come in different formats that range from 10-bit to 16-bit, at first, I’ve been loading them as Uint16Array, then doing a conversion + normalization + adding alpha, and getting back a Float32Array. Then, creating a THREE.DataTexture with RGBAFormat and FloatType.

With some photos reaching 10 000px on the long edge, RAM usage becomes a problem, since I can’t seem to find a way to Garbage Collect quickly enough. So it all ends up storing a couple of copies of the same array in different states.

I try to ‘null’ the arrays as soon as I don’t need them, but that still relies on GC.

Currently, I’ve been experimenting with this:

Which helped me shave off quite a bit of RAM-usage off the peak, at a cost of longer loading times (creating and sampling 3 textures instead of 1).

So, I was wondering, maybe someone has come across any other ‘tricks’ that would allow me to use 16-bit data for textures? Or maybe a way to do the Uint16 RGB → Float32 RGBA conversion more efficiently with respect to peak RAM usage?

I’ve seen some applications using Tiling and IndexedDB storage, but before I attempt to go down that route, I wanted to ping you all and maybe get some lower-hanging-fruit ideas.

Thanks again!

2

You can use 16 bit integer, or 16 bit float (HalfFloat), directly as a texture source. If it’s a single channel you can use RedFormat or LuminanceFormat iirc.

If you do need to transform it.. you could make your loader load in chunks and process a chunk at a time, convert it, and then throw away the source chunk.. and when you’ve loaded and converted all the chunks.. concat them.

3

In the past I’ve converted uint16 RGB to float16 RGBA this way:

import { DataUtils } from 'three';

const UINT16_TO_FLOAT = 1 / (2 ** 16 - 1);

// rgb -> rgba
const rgb = new Uint16Array(/* <source data> */);
const rgba = new Uint16Array(width * height * 4);
for (let i = 0, il = width * height; i < il; i++) {
	rgba[i * 4 + 0] = DataUtils.toHalfFloat(rgb[i * 3 + 0] * UINT16_TO_FLOAT);
	rgba[i * 4 + 1] = DataUtils.toHalfFloat(rgb[i * 3 + 1] * UINT16_TO_FLOAT);
	rgba[i * 4 + 2] = DataUtils.toHalfFloat(rgb[i * 3 + 2] * UINT16_TO_FLOAT);
	rgba[i * 4 + 3] = DataUtils.toHalfFloat(1);
}

const texture = new DataTexture(rgba, width, height, RGBAFormat, HalfFloatType);
texture.colorSpace = LinearSRGBColorSpace;

In theory you shouldn’t need the float16 conversion, and could use uint16 directly from the RAW file with THREE.UnsignedShortType and texture.normalized= true (I’m assuming you are using mipmaps and wouldn’t want integer values), but that’s a newer option in three.js and not one I’d tested at the time.

***

A trick someone shared with me a while back was to transfer large arrays to a Web Worker (the Web Worker doesn’t actually need to do anything, it can just sit there) when you’re done with them, so that GC happens off the main thread. I don’t know if that would help to avoid a crash if you’re running out of RAM, but it should reduce the stalls associated with Major GC events.

4

Thank you both!

Weirdly, I could not get uint16 to work when I tried it. Double-checked, and I’m getting an internalformat warning (with no picture). Should I be setting internalformat manually?

texture = new THREE.DataTexture(rgba, width, height, THREE.RGBAFormat, THREE.UnsignedShortType)

&

texture = new THREE.DataTexture(rgb, width, height, THREE.RGBFormat, THREE.UnsignedShortType)

Returned

WebGL warning: texStorage(Multisample)?: internalformat: Invalid enum value RGB
WebGL warning: texSubImage: The specified TexImage has not yet been specified.

or

WebGL warning: texStorage(Multisample)?: internalformat: Invalid enum value RGBA
WebGL warning: texSubImage: The specified TexImage has not yet been specified.

Regarding uint16 → HalfFloat conversion, and please correct me if I’m wrong because my understanding is still very superficial, but isn’t that a precision loss?

Thanks again for the replies, they also help me to get a rough direction of where I need to be going.

5

For this case, think you’ll have to set:

texture.normalized = true;
texture.needsUpdate = true;

I’m not sure of the current state of supporting RGB (vs. RGBA), so just to get things working with uint16 unorm, I’d want to get RGBA working first.

You’re correct that uint16 to float16 represents some precision loss, though… I’m not sure if that’s still meaningfully true if the source data for the uint16 texture was 10- or 12-bit.

Related, fixing an example loading rgba16 unorm from a KTX2 texture here: KTX2Loader: Fix regression in rgba16 unorm support by donmccurdy · Pull Request #33662 · mrdoob/three.js · GitHub .

6

Very cool, grabbed the three.js off GitHub, can confirm that 16-bit RGBA now works with texture.normalized = true.

Sadly, the GL extension necessary for that is not supported in Firefox.

It seems that there won’t be a workaround that avoids that one per-element array operation (which is quite costly time-wise), it’s either adding Alpha channel, splitting the array or converting to different array type. Or a mix of those.

Anyway, thanks again for the detailed information! This has been very useful to me.