One idea: toTransferable API (for working across Worker thread boundaries)
Problem:
Getting things like ColladaLoader and GLTFLoader to function in a web Worker is not too difficult: it requires mocking (or polyfilling) some APIs on the worker side (f.e. document.createElement('img')) to make those loader classes believe they are using DOM APIs, and filling in the mocks to use what is available in a worker (f.e. fetch for fetching image data).
That’s the easy part!
Where it gets tricky is when we want to send the loaded objects to another thread (f.e. back to the main thread if that is where Three.js is running).
What I’ve been doing so far is calling toJSON on collada.scene or gltf.scene and passing that through postMessage (using Google’s Surma’s fabulous Comlink library makes it easier), then on the receiving end using ObjectLoader to convert the payload back to usable Three.js object.
But this has costs:
- On the sender side, in
toJSONcalls,TypedArrays are converted to regularArrays (cpu time cost in sender). - Transferring of the object over
postMessagemeans that the regular arrays will cause a mem copy (CPU time cost during postMessage). - On the sender side, Textures will have their images converted into
Blobs, and finally converted into object URLs (CPU time cost in sender). - On the receiving side,
Arrays will be converted back intoTypedArrays inObjectLoader(CPU time cost in receiver). - On the receiving side, images need to be fetched and loaded from the blob URLs (CPU time cost in receiver).
What we really want to avoid is cpu time cost during postMessage and CPU time cost in receiver, because those are the ones that will pause (jank) the UI thread. We don’t really care if some processing time takes longer in the worker, as long as we eventually get the result without janking the user’s experience, although saving any time wherever possible is still better.
But, it works! With minimal polyfill, we can load objects in a worker, and actually get some performance gain, but nowhere near as much as could be possible.
Solution:
What I think would be super nice is an API that is basically identical to toJSON, perhaps called toTransferable, that does exactly what toJSON does, except it will
- On the sender side, leave
TypedArrays as typed arrays (CPU savings in sender). - Transferring
TypedArrays asTransferables is super fast, no mem copy (CPU savings during postMessage). - On the sender side, convert images to
ImageBitmap(CPU time cost in sender, but savings during postMessage), or update the Texture class to automatically always load images into ImageBitmap by default (cpu savings both in sender and during postMessage). - On the receiving side,
TypedArrays remainTypedArrays, simply passed around byObjectLoader(or perhaps a new class calledTransferableLoader) (CPU savings in receiver). - On the receiving side,
ImageBitmaps remainImageBitmaps, simply passed around (CPU savings in receiver).
Similarly, what really matters is CPU savings during postMessage, CPU time cost in sender, but savings during postMessage, and CPU savings in receiver, all of which cause savings on the important receiver side where UI is running (even if it means more time cost for ImageBitmap on the sender side), where making the UI-side experience Jank-free is the most important goal.
Note that the result of toTransferable would no longer be compatible with JSON.stringify, but the performance gain will be very nice for use with web workers.
toTrasferable would return a tuple: [object, transferables] where object is the object output (similar to toJSON, but containing TypedArays, ImageBitmaps, etc), and a list of all Transferable objects.
Transferable objects are objects that get sent as a pointer copy instead of as an array copy. Passing an Array over postMessage means all the array data will be copied, while transferring a Transferable (f.e. a TypedArray or ImageBitmap) means that only a pointer will be copied to the other side (plus some minimal object wrapper created on the receiving side, but the cost is very cheap compared to copying all the array memory).
Implementation:
toTransferable would simply be a matter of copy/pasting all toJSON code, and modifying it a little so as to preserve all the Transferable objects in the output object, and instead of returning the output object, return a tuple with that object as well as a list of Transferables.
The list of Transferables is needed because of how postMessage needs to be called:
const [object, transferables] = superDuperScene.toTransferable()
self.postMessage(object, transferables)
// or
self.postMessage(...superDuperScene.toTransferable())
where object contains (anywhere inside of it at any level deep) the transferables that are listed in transferables.
jank 