Have you has a look at the new WebGPURenderer / NodeMaterial / TSL architecture?
I feel like all the stuff you’re working on should be able to work well in the new setup.
For example, here’s the new SSRNode:
https://raw.githack.com/mrdoob/three.js/dev/examples/index.html?q=ssr#webgpu_postprocessing_ssr
I’ve had a look at all of those, didn’t see the SSRNode before though. I do have some thoughts.
I think in general the approach is sound, and I like how minimalist a lot of the code is. It’s very few lines for the benefit that the API provides. I admire that.
To me, the issue is in the overall architecture. Three.js is going for a fairly traditional rendering architecture with WebGPU. Which is fine, and you can build other architectures on top of it, but that’s not what I wanted to build.
My goal was to build a GPU-resident renderer, something that can render millions of instances in real-time without much CPU overhead.
Let me paint a rough picture:
in 2014 DirectX 12 was released. It provided a much lower-level graphics API.
Vulkan came soon after, now we also have Metal. The primary problem that these APIs came to address was this:
pushing commands to GPU is slow, let’s make it faster.
To do that, command queues were introduced and we were given the ability to record command buffers in different threads. This was huge, you could now push almost 2 orders of magnitude more commands to the GPU each frame. The “bottleneck” of CPU ↔ GPU communication was widened.
At the time, it was believed that this widening would solve the issue for good, but GPUs got faster and CPUs didn’t really.
As AI started to adopt GPUs and started to take off, money started to poor into the manufacturers’ hands from clients that wanted to do general purpose compute on the GPU. They didn’t need “shaders” they had compute. So compute shaders started to develop and with time - dominate.
Graphics programmers saw this happening, so they started to move more and more of traditionally CPU-based work loads to the GPU. A lot of that trend was driven by the console market as their hardware architectures incorporated compute more readily, and the hardware architectures were more… fluid, where you had a bunch of not-quite-CPU cores that could do a lot of work, but required special programming model, just like compute shaders.
Let’s fast-forward to today, a lot of graphics engines are idling on the CPU side, there’s little to do for them there, as most of the work is happening on the GPU. All you do is build descriptors of change, and sometimes even that is handled on the GPU.
With this, the paradigm on drawing one object at a time and sending a bunch of commands to the GPU for each object is dying. I don’t think it’s dead, the graphics APIs like Direct X, OpenGL, Vulkan etc are still centered around the concept of attributes, indices and traditional draw. But these are getting less and less use.
Granted a GPU-resident renderer is much harder to build, there’s more complexity, there’s less educational material and the APIs allow you to do that, but don’t really encourage you or help you in any specific ways.
So, long story short - WebGPURenderer the way it is now, and the direction it’s going is not compatible with what I’m doing. On a pretty fundamental level.
Three.js is fast-enough, it’s much faster to get into, the programming model is very clear and straight-forward. I’m creating something, arguably not for today but for 2+ years from now.
I like the idea. I think Node-based languages are really powerful. I wrote a few in my days. Heck, meep has a few node-based languages inside of it.
The problem I see with node-based languages is the user interface. A node-based language can be useful as an API, but only if it is sufficiently high-level. A low-level node-based API is just a pain with no gain. You can sort-of see it in Unreal and Unity, they have node-based shaders, but they offer very complex nodes for you to use there. I have used Unity’s shader editor a fair bit, and I came to realize that it’s a massive pain. Because it doesn’t offer a good user interface. The UI is slow, there’s no search, grouping is non-existent etc. So, in my view a node-based language without a great UI is a bad investment.
Can you write an SSR shader in a node-based language? - yes, as evidenced by @Mugen87 's work that you linked. But,
I know I’m cherry picking here, but I hope this clarifies my view a bit.
Do I think GLSL is great, or even WGSL - no, I think they are pretty bad languages. Especially WGSL is a massive pain. But it’s a standardized pain, with a lot of reference material. I was missing module system in WGSL - so I wrote a bare-bones one like so:
/**
*
* @param {CodeChunk[]} dependencies
* @param {string} code
* @returns {CodeChunk}
*/
CodeChunk.from(code, dependencies = []) { /* ...*/ }
This isn’t perfect, but it’s good enough for me. I looked at a bunch of different language abstractions for WGSL in particular, or, let’s say SPIR-V, the problem is that they all sacrifice expressiveness and specificity for the sake of compatibility. Just like, say TypeScript is dominant because it only compiles to 1 language - JavaScript, so it’s able to capture every aspect of JavaScript perfectly. As soon as you start to target multiple languages in translation - you’re playing a losing game.
Here’s a simple example: Unreal engine can target WebGL as a compile target. And it works! It looks like 
though. Why is it? Do guys at epic not know how to use a graphics API? It’s because the compiler is forced to target the lowest common denominator, the actual compiler doesn’t know about WebGL in particular, not really. It targets OpenGLES 3.0, and then disables all features that don’t translate directly to WebGL.
There are some successful examples out there, such as C, or LLVM. But they have the benefit of C not being a target language at the time, and actually providing more conciseness and expressiveness, and LLVM not being oriented towards programmers.
I think NodeMaterial concept is not bad, and can be great, but it needs to come with an amazing set of tooling, specifically a UI.
TSL - I don’t dislike it, but I don’t love it. It’s WGSL with extra steps, and you’re using it as a declarative language written in a functional language (JavaScript) which makes awkward on top of what I mentioned before.
I’m not sure this is speicifically valid or not. But SSRNode is a toy. If you read the original code - you can see the step-wise ray marching through the depth buffer, you can see the basic denoising with the edge-preserving blur pass, but it’s not a practical tool.
The traversal is way too slow, and it considers everything to be a mirror surface. You can use it to produce some pretty pictures, but it’s not physically grounded in the slightest, it doesn’t respect StandardMaterial and it’s not energy-preserving.
I don’t think that’s a problem, it’s a good teaching tool and there are use cases where you have glossy everything where it behaves close to “realistic”.
Isnt the point of a system like this to just be able to add a node and have it work automagically? It doen’t seem that the target user is concerned with any of this?
I wouldnt be surprised if there was a react solution today, such as:
<MyMaterial>
<PBREffect/>
<SSREffect/>
</MyMaterial>
Or what in the old days of three.js would look like:
myMaterial.ssr = true
This is the first take on TSL that i see like this, but you also seem to be doing much more lowe level work than the people that have been raving so far. Any chance you could elaborate on these cons?
It might, and I think it would open up a new market for 3d on the web. It would make the engine less approachable though, and require some pretty good documentation. All of this would put a strain on development resources and increase barrier to contribution.
That’s eseentially what I have, there’s just a mesh, no instancing. It makes sense when all of your data in on the GPU and you’re building your draw calls there as well. Instancing basically becomes the default and the cost of 1-instance draw overhead disappears.
Ha, I like to think about the future, but honestly I’m not that good at guessing the hardware future. I think that we’re not going to have something radically different. It’s true that we’ve had decidated hardware for various things over the years that have been absorbed into the CPU/GPU or even the chipset. But the overall architecture hasn’t actually changed that much. We still have an ALU, and we still have tiered memory model. As long as you have distance between ALU hardware - time is going to be a factor, and moving code as close to the data will be a trend.
GPU-resident architecture is essentially an observation that your data is on the GPU, so instead of controlling the logic from the CPU that’s far away, why not control it directly on the GPU as well.
I find that many “educators” like to sensationalize the topics they describe. I think that’s a really sad fact, because things like GPU-resident architecture sound scary and arcane - but they are really trivial to understand, it just takes a few more words and a little bit more time. When we present topics as being “super complex”, it makes others think that there’s no hope, they are too dumb to ever understand it.
It’s true that some topics are very very complex, but typically it’s because there is a lot to learn, not because each individual thing is hard.
Yes, you’re right. However, when you do this - a shader still need to be compiled. And TSL will add a lot of overhead to that compilation process. The compiler for TSL is not nothing either, so you’ll have to pay the cost of (down)loading that as well. Now you have the same functionality, but your bundle is larger and your app takes longer to start.
That’s on the user’s side. On the programmer’s side you’re affected more directly, most of what I wrote was in that key, about the developer experience.
In software engineerign we work with abstractions. Your CPU runs instructions, those instruction are created from compiling x86 assembly, which was compiled from C code in case of Chrome, for example.
In the same way, TSL is an abstraction on top of GLSL. An abstraction is not free, you typically sacrifice performance, because you lose some semantic power. You also add compilation/translation overhead.
It’s a trade-off. What does TSL give you for the negatives it incurs?
I think that’s about it. You could argue that it provides static analysis as well, but I think this is really debatable, as DAWN (the WebGPU backned in Chrome) and driver vendors already do a ton of static analysis for you and have way more resources to do so than our three.js project here.
Again, I think nodes make sense, but not as an API, not in this case. ← only my opinion though.
Is 1 feasible without 2 in the real world? If one is using things that are available on WebGPU but not on WebGL, does it defeat the purpose of running on multiple APIs? Would one still assemble different programs, but by the virtue of 2 this should be easier? (Provided one is willing to pay the price that you mentioned).
Specifically on TSL, as people seem to care about the topic.
The idea of a “common” language for the GPU is not new at all. In many ways that’s what GLSL or HSL are. They are common languages to vastly different GPU architectures.
The idea of having a language that compiles to multiple different API-specific languages like HLSL and GLSL is not new either. Most of the big companies out that that EA with their Frostbite engine had a node-based shader language that compiles to HSLS and GLSL for a long long time now.
Unity guys thought to themselves that they are very clever, so they invented their own oh-so-special shader language. Even godot has their own shader language.
NVidia created their own language not so long ago, called slang. Well, if nothing else - I like their sense of naming.
AMD’s famous Fidelity-FX has a kind-of light-weight language abstraction called ffx, very unimaginative. It targets GLSL and HLSL mostly.
Unreal tried a few different options and settled with HLSL and using cross-compiler toolkit from Intel to target other languages.
My point is - TSL seems like a new shiny thing, but only in the very narrow scope of a browser and WebGL/WebGPU APIs.
Just to make sure I don’t get more flack for this than I deserve - I don’t think that TSL is a bad idea. I think it’s an idea that need many more pieces to be good, in my view.
I think you pretty much got it right. Imagine the case of using a compute shader. These don’t exist in the WebGL, so what does this abstraction layer do for you?
It can do one of two things:
transform feedback feature of WebGLThe first option is kind of useless, as it means you don’t really support both APIs, and the second has extra cost, as performance will likely be bad and you will be pulling a massive engine for correctly translating compute to WebGL.
How about atomics, WebGPU has atomics, WebGL does not. How do you reconcile that?
How about memory address spaces in WebGPU that give us groupshared memory?
You get the picture. You have two languages that are inherently different.
With that said, there are more commonalities than there are differences, and you can share a lot of code. So in your shaders you might be able to share, say 95% of the codebase between WebGPU and WebGL. The problem now is that you have to have disclaimars like:
“This TSL module will only work with WebGPU”
Aaand we’re back to asking the question: Why not just write WGSL instead then?
( silently noting that the discussion now has the potential to slip off into off-topic )
That did not sound very silent lol. Is it regarding mrdoobs post? I think its super valid to gather user feedback.
@Usnul thank you for the explanation!
Way to go @PavelBoytchev, this sounds not only feasible but also a great deal.
I’ve been myself searching for GPU data structures as a more efficient way to work, specially for geometry, about a month ago I found RX Mesh from Autodesk, and there are others exploring the same concepts. Somehow your post gives a very vivid picture of “totally doable ideas”, trully inspiring.
And lets just appreciate @Usnul efforts in sharing his advances and receiving feedback, this forum is pure gem.
Rxmesh looks really promising! Just seen this also https://youtu.be/3EMdMD1PsgY?si=G2JXCCJQLPYVtiuu
Yep, that’s pretty much what Shade does as well. I watched that presentation and the referenced one from Remedy, my architecture is very close to theirs.
Mesh shaders are cool, but by many accounts compute shaders are close-enough in performance that it makes small amount of difference.
Also, you’d need full-fat drawIndirect API if you work with mesh shaders, which we don’t have in WebGPU. We have a severely lobotomized version which is practically useless.
Therefore - compute shaders
I have dealt with this intensively. I made a PR last week about this. I owe thanks to Sunag and RenauldRohlinger for their suggestions for improvement an review for this.
I worked very intensively with the node system to understand it better because it was very important to me to have drawIndirekt in the WebGPUBackend.
With this PR, three.webgpu.js r170 will supports drawIndirect/drawIndexedIndirect. We then really have the opportunity to use INDIRECT drawBuffers in compute shaders and assign them to geometries. TSL also always allows the option to use raw WGSL if one prefer that like I do. In any case, I’m really looking forward to r170 with the drawIndirect extension 
A small update. Added bloom post process effect.
Here’s a shot with some emissive materials
and here it is with bloom on:
The light intensity is ~2.2 here, and emissive coefficient on the traffic light LEDs is ~10.
Here’s a shot with different light direction to put most of the scene in shadow:
and with bloom on
for comparrison, here’s a version without any post process (SSR, SSAO, TAA, Bloom all off)
Here’s a shot of Sponza with high directional light intensity
here’s Bistro scene, I cranked up emissiveness to 10 on all emissive materials
For those who are interested in details, the technique I use is a hybrid of a few different existing techniques.
Because the filter size is quite small, with only 13 taps - the technique is almost free in terms of performance. The luma filtering helps get rid of temporal instability and fireflies.
Overall I’m very happy with the results.
I was on the fence about bloom before, but I think if it’s implemented well - it does enhance the image in a very significant way, and virtually for free too.
A while a go you suggested you may have some live demos, have I missed them?
Touch controls are incidental, but should work at least to some degree ( if you’re lucky enough to have WebGPU support on your touch-screen device)
Just to preempt some of the expectation:
Shadows are going to take up anywhere from 50 to 95% of render time, this is due to the lack of RTX API in WebGPU, RTX shadows are just expensive unfortunatelly. I’m working on an alternative solution, so this is a known perf limitation right now.
This is not open-source, and the code is under copyright. I know that sucks for some of you, but it’s literal months of my work out of pocket, so please respect that.
Would you consider just contributing this to the main threejs repo? Under “examples” or “addons”?
Hey @dubois ,
No, I don’t think so. My stance is relatively simple:
As it related to three.js specifically, the architecture is drastically different, so it doesn’t really fit.
But the main thing is that this is intended to be a commercial product at the end of the day, so open-sourcing would not align with that.
I still think there’s a lot of value in just seeing the tech run in the browser.
A lot of the times you think to yourself: “I don’t know if this technique would even be viable or not”. Then you spend weeks prototyping, only to arrive at the answer “no”, or you arrive at something that kind of works, but you give up because you don’t believe that you could get to something truly viable.
Seeing a complete solution eliminates a lot of the guesswork. For example, Epic’s Nanite has been pretty much solved in late 90s and early 2000s, I know this because I researched the topic extensively when working on my own implementation. But why hasn’t anyone made a commercial version of this before? Because it’s a complex thing that would take a large time expenditure and it’s not clear at all whether it’s viable at the end of the day.
Once Epic published their own work, suddenly you see versions of it popping up left and right, because guesswork is eliminated.
Anyway, I do hope this this will serve as an inspiration and a proof of viability to others.
