THREE.ComposedTexture - play GIF, APNG etc as texture

[esnho]

Hi, this looks cool!
I’m new to gumroad, how do you distribute the class?
It’s an npm module? A tag script?
Which is the license model?

Thanks!
Luca

[Fyrestar]

Thanks you get the source files from gumroad, containing the example as well, further updates will be sent out directly to your mail you used. You’re free to use it in commercial and non-commercial projects, just minify it with your final bundle.

[cabada]

This looks great!

I have a question regarding optimization, I’m looking forward to fill a scene with multiple animated textures.

Would you say using this approach with GIFs/APNG is a better at optimization than using VideoTexture with .webm?

[Fyrestar]

Thank you.

@cabada

I have a question regarding optimization, I’m looking forward to fill a scene with multiple animated textures.

Would you say using this approach with GIFs/APNG is a better at optimization than using VideoTexture with .webm?

That depends on multiple things of your scenario, how long/how many frame and how large the animations are, if you need clean transparency, and if there are many different animations.

[cabada]

Just got the script from your Gumroad, and I gotta say it works amazing!

@Fyrestar

That depends on multiple things of your scenario, how long/how many frame and how large the animations are, if you need clean transparency, and if there are many different animations.

They are very simple and small, but will have lots of different files. I’m using it for environmental purposes. I tested it without re-using the texture to see if it can candle multiple GIFs without lagging and it works good. (I’m attaching a video) (Butterflies are the animated sources)

I think I’ll stick with this solution instead of VideoTexture

Thank you!

[Fyrestar]

Looks great!

Yes for sprite purposes like that it’s perfectly suited.

[uvfauna]

This works great but is somehow blowing up the memory of my website
I am displaying 4 gifs in my scene (their filesizes are 74 MB combined).
My website’s heap memory (chrome dev tools) goes from 55MB to 1GB when I compare the scene without versus with the 4 gifs.
Is there a known memory leak somewhere?

[Fyrestar]

74 MB for a single GIF is quite huge, usually they’re optimized to be around 1-10 MB. There is no known memory leak, when decoded/encoded they’re also in a compressed image format if you used one of the example ones.

With the dev tools you should be able to trace down where the heavy memory usage comes from more precisely. Also make sure to drop references to the original loaded files after they’re decoded.

[uvfauna]

Thank you fyrestar for the wise words. I also became aware that some of those gifs were just too heavy, so I removed those ones and I also compressed all of them and reduced their size by half.

Can you explain a bit more about the reference dropping after decoded? Is this already done in your example file?

[Fyrestar]

It’s just about making sure that you don’t keep variables; or better properties in objects still being used -holding the blob files loaded in your app referencing them. if it’s never referenced outside of the loading procedure it will get collected by the GC.

Sometimes people just try to misuse something to carry the data along assigning the blobs to a unrelated object just to pass it along, but as long as that object is used the property holing the blob stays in memory.

In short: if you (would) assign the blob to some app abstract class like “Resource”, as a property resource.blob you would need to null the property for it to get collected, or if you made an URL for a blob calling URL.revokeObjectURL, it’s more about general JS than a THREE.js specific subject.

[benwest]

Hi! This looks promising for my project. Can I ask if frames are sent individually from the CPU during playback, or is all the data uploaded to GPU up front?

[Fyrestar]

Since playback composition runs through canvas it should be technically all on GPU side with the frames compressed by the ComposedTexture class, in any case there is always only 1 texture buffer to be rendered. I consider adding stripe generation in the next update, however having one composition texture is more scalable and flexible for different number of frames, required for partial frames due to optimizations of the gif/apng etc, and more natively compatible as no shader side specific playback related treatment is required.

[png]

ran into some issues with the gif not update if using threejs ^141 and above and fixed them via…

• call this.needsUpdate = true at the end of the _render function so threejs knows to update the texture.

• rename THREE.Math functions to THREE.MathUtils since its since been renamed

[Umbawa_Sarosong]

load the GIF separately, with its own image tags, see virtual tour the map pings loads animated transparent nodes using GIF image with libgif.js

[Fyrestar]

Thanks for the notice i’ll check it and make an update.

[Fyrestar]

New update is done

It will be sent to your email you provided on Gumroad.

• Fixed: for ^141 releases (needsUpdate)

• Added: SpriteTexture for using sprite-sheet textures and playback just like ComposedTexture

• Added: toSheet method on ComposedTexture create a baked sprite-sheet of the frameset
  Creating a sprite-sheet from ComposedTexture will automatically use a ideal layout stripe or atlas and if exceeding a given max resolution scale down to fit

• Added: timeScale for ComposedTexture and SpriteTexture to alter general playback speed (default 1.0).

Things you may want to configure

// May be set by renderer.capabilities.maxTextureSize (recommend default unless needed), sheets going above will be scaled down to it
THREE.ComposedTexture.MaxSpriteSheetResolution = 4096;
// Sprite-sheets below this resolution will be a stripe which is more reasonable for uneven number of frames
THREE.ComposedTexture.MaxStripSize.MaxStripResolution= 2048;

THREE.SpriteTexture

This new texture class uses a sprite-sheet (atlas or stripe). The animation interface is the same as on ComposedTexture. There are cases to use either this or ComposedTexture. The sprite-sheet can be derived from a ComposedTexture. via simple call .toSheet or provided from a regular texture.

Benefits

• Only 1 initial texture upload (asked here)
• Texture can be shared for a massive number of instances with individual animation state (related question)
• Can be more memory efficient on JS side by it’s resource being a single texture and the frames not being needed anymore
• You can load an assembled sprite-sheet texture and assign it directly, either you provide the sprite-sheet meta info or let it generate it:

const sprite = new THREE.SpriteTexture( {
	texture: spriteSheetTexture,
	padding: 0,
	columns: 3,
	count: 7,
	delay: 80
} );

Notice: the benefits of sharing the spritesheet texture are only available for THREE release R138 and higher, since a concept for sharing textures has been added.

When to use SpriteTexture

• You got a smaller amount of frames or/and resolution, typically like 2D game asset sprite-sheets
• You want to render a huge amount of it with different animation states
• You don’t need to use repeat wrapping of the texture or texture transformations

When to use ComposedTexture

• Your resource files for your project/use case are GIF, APNG etc
• You got a lot / larger frames that are too many / too large (or unpredictable in that regard) for a sprite-sheet
• Your main concern is GPU memory, ComposedTexture has fixed memory usaged of 1 frame on GPU
• You need repeat wrapping of the texture or texture transformations

The outer left showcases a GIF turned to spritesheet and used with SpriteTexture then.

Next i’ll probably add a animation mixer API to easily define and deal with takes/sequences, as well as probably a tool for you to create and manage them with a dedicated interface.

Also a demo to use with THREE.InstancedMesh.

[niccolofanton]

Hello, I’m using the same code in your example, a renderer with outputEncoding: sRGBEncoding, and when I use the same encoding with your ComposedTexture fps drops to 20/30 for some reason.
If no encoding is specified on the texture everything runs smoothly, I have also tried with other types of texture with the same encoding and I’m not having issues.

This is the gif I’m loading I’ve also noticed that with your gif (magic.gif) the problem doesn’t appear

[Fyrestar]

I can’t reproduce or notice a performance drop using your gif with different output encoding, does your problem appear in your app only or when using the gif in the example too? It’s also a really giant one with 2k, you should pass the container.downscale = true option so it will scale it down rather than up to next PO2 to 1024.

[niccolofanton]

Ok, thanks for the quick reply, I switched to SpriteTexture and the issue seems fixed.

I’m going to try with container.downscale = true

Edit: container.downscale = true fixed the issue also with ComposedTexture thanks a lot!

[Fyrestar]

Yes i certainly would suggest downscale anyway.