Shade - WebGPU graphics

[Usnul]

Yet another tangent, tried integrating atmospheric transmittance and scattering:

without:

with:

without:

with:

There’s transmittance affecting the light itself, changing the color of the light that hits the object. Scattering scatters some of the light as it travels through the atmosphere.

The effect is quite pronounced for larger scenes and helps with depth perception.

For smaller scenes like the one in the second screenshot it’s not as relevant. We can see a bit of darkening due to light absorption which makes the scene look a bit more interesting.

Sources

MinimalAtmosphere by Felix Westin

I spent more time than is reasonable researching the topic.

In particular I found Sucker Punch presentation on Ghost of Tsushima very useful.

Also, Epic Games EGSR 2020 presentation “A Scalable and Production Ready Sky and Atmosphere Rendering Technique”.

Here are links to the Epic Games work by Sebastian Hillaire:

• paper
• repository (I’m amazed they released it under MIT)

[Usnul]

Continued working on the path tracer, took a while to connect various parts of the engine, but finally got the ray tracing part working.

The main purpose of the path tracer is to build irradiance cache for global illumination, so it’s not intended for viewport rendering.

But to make sure that the path tracer works correctly, you need to test and validate it somehow. You could do that in code, but our eyeballs are incredibly good at it already, so that would be a waste.

Therefore, I decided to build a basic viewport path tracer for testing/validation purposes first.

So far I got the RTX api implemented (nearest hit query + any hit query). Here’s a screenshot validating the nearest hit:

It’s not very exciting, as we just draw distance of the hit from the camera into the buffer. That’s just the depth buffer. But, it’s a ray-traced depth bufffer.

Next, I worked on material and texture sampling inside the ray tracer, tool a little while to sort things out but got it to work:

Bloopers

There’s still a lot of work to do, and if you look closely you’ll be able to spot some errors quite easily.

[Usnul]

Added light sampling and bounce, here’s the current state (note, no tonemapping):

64 samples, 3 bounces

4 samples, 3 bounces

1 sample, 3 bounces

There are a number of issues, UVs are a bit messed up, material is only using the diffuse channel etc. But the bounces are good. Here’s the same render without any textures:

and here’s the normal rasterized render with just the direct light:

I’m pretty happy with it already. It doesn’t even need to be perfect yet, it just needs to be able to create a good-enough approximation of diffuse light bounces.

[Usnul]

Working on GI and shadows. They are quite tightly bound really. To do GI you need to trace shadow rays.

Added radius parameter to lights, which lets us have soft shadows:

Ray Traced

Default

Here’s the version with disk radius = 0:

Ray Traced

Default

The sampling noise needs mroe work, and shadows are too noisy to be used in production, so I’ll need to add denoising and some kind of temporal reprojection them.

Here are a few more shots with soft shadows:

I also worked a bit on performance optimization for path tracing and got ~ 2x uplift on ray queries by being more careful about memory access.

Here are a couple more renders from path tracer as a bonus

[Usnul]

Fixed a few issues in intersection math and material sampling of the path tracer:

Textures are in the wrong color space, so everything is too blown out, but other than that - I think I cam move onto other things, the tracer is in a good-enough state already.

[x1911]

This is a very impressive project. Do you plan to make this engine available for everyone to use? Recently, I have been trying out Three.js with WebGPU, and as you mentioned, its performance improvement seems quite limited compared to WebGL.

[Usnul]

@x1911

This is a very impressive project.

Thank you!

@x1911

Do you plan to make this engine available for everyone to use?

Yes, eventually. But most likely not for free and not as open-source.

---

Since last time, I made a number of improvements to the path tracer:

After that I’ve been working on the temporal denoiser, most of the problems boil down to my lack of knowledge, but I’m finally making progress, here are a few shots with basic elements of denoiser in play being used for ray traced soft shadows:

here are a few screenshots of debug data while working on the denoiser:

temporal accummulation is not finished yet, all of the smoothing is from a single frame spatial denoising and TAA, which is already pretty good, but there’s visible flicker if you look closely and shadows are not as smooth as they can be.

Planning to make a release a demo once denoising is done. After that global illumination is the only major part that’s left.

[Usnul]

Was busy with other work recently, so progressed slowed down quite a bit.

However, there is progress. Introduced light probe volumes that I worked on previously. It turned out to be relatively painless process, porting code from WebGL and JS over to WebGPU so far.

The process was made easier by the fact that a lot of my performance-critical code is written using very simple pure functions, which are incredibly easy to port over to shaders.

I re-visited my path tracer along the way, fixing certain inaccuracies and improving the shading model. Here is Sponza with light probes:

Here it is without:

And here are just the probe contributions:

You can see a lot of color bleeding on columns and hanging banners, overall the image is a lot warmer.

One very important, and I think cool, thing here is that all of the probe rendering is done incrementally in real-time. So if you chance the lighting - probes will update over time. There is no “baking” as such, you load a scene and it progressively gets updated each frame with a few more rays per probe.

There are a bunch of technical issues to fix still though, as there is light bleeding all over the place. I haven’t ported deringing code for probes yet either. Once those problems are addressed - I believe this will be an awesome turn-key solution for pretty much every use case.

Here are a few more comparissons:

---

---

---

For testing purposes I’m using an 8x8x8 grid of probes for each scene, but again, it’s a tetrahedral mesh, so in real-world scenarios probes will likely be sparsely placed, with more probes near surfaces and fewer in empty space.

[Usnul]

Still working on global illumination. Ported most of the DDGI implementation over from JS side to GPU. Still struggling with various bugs. Here are a few screenshots of current progress:

Here are some of the artifacts:

You can clearly see the probe boundaries and there are various leaks.

Oh yeah, I’ve spent a bit more time on the sky. Realized that it’s pretty important for lighting, especially for outdoor scenes. It’s a fascinating topic, how the sky color is all about absorption and scattering due to interplay of molecules and photons. I’ve incorporated sky into both the direct lighting ad well as path tracer that powers the GI. Here are a few pretty renders from the path tracer with the sky enabled.

Path Tracer

![Screenshot 2024-09-04 054511|500x500]

(upload://3pZDBmN1hn0WNKBr2cj7YyMIYvW.jpeg)

for those who are interested all renders are 32 samples only (hence the noise).

Global Illumination is Hard

The difficulty with GI is that it’s a multi-variate optimization problem. That is - there are multiple things you want to maximize, like realism, shadows, highlights. In reverse sense, things you want to minimize like noise and leaks (shadow/light where they aren’t supposed to be).

One one end of the spectrum you have a full path tracer, which is slow as sin and can’t really be made “fast”. There are tricks that use various functions to guide rays, and caching strategies, but inherently it’s a statistical simulation and those are notoriously non-linear in excution time. For things like solving a differential equation with a couple of variables using gradient descent - we know we can get a good result in just a few steps, but for path tracing the “number of variables” or the conditionality of the problem is huge, really really huge. So path tracing will never be the go-to technique for real-time rendering where performance is concerned. There, I said it.

My path tracer can trace ~9,000,000 (9 mil) random paths per second, with up to 3 bounces. That’s on RTX 4090. It’s slow, too slow for real-time. 1080p resolution is typically 1920*1080, or 2,073,600 (2mil) pixels. I can trace ~4 frames per second, in reality it’s quite a bit better as paths are not random if we trace from the screen, most of the rays will have excellent alignment so GPU cache usage will be good.

Say we can trace 20 frames, say you’re running a proper RTX api and not a custom software implementation as I do, say you get 100 frames, or even 200. That sounds good, except the actual image quality will be awful. For a relatively clear image in good lighting conditions you need ~500 paths (samples) per pixel, so that 200 budget brings us squarely back to 0.4 FPS.

A good denoiser with temporal component can make do with ~16 samples per pixel. We’re back in real-time range at 12.5 FPS. But this will only hold true as long as we’re in good lighting conditions. As soon as lights get smaller and further away - required number of samples grows exponentially.

I believe a path tracer with tricks can be used in specific lighting conditions, eventually. But that’s probably a couple of generations off. But even then, we’ll be relying on tricks.

Famously unreal’s lumen is ray tracing right? Wrong, it’s a clever mix of SDF ray marching and surface caches. Even then, it’s pretty demanding on hardware and is unstable in low-light conditions and in cases of small light sources.

On the other end of the spectrum, we have irradiance fields, such as what I’m working on. The idea is to capture lighting in 3d space using potentially expensive techniques like path tracing and sample that “field” during shading. No path tracing necessary once the field is captured.

Irradiance Field

Neat, except that field will be of low resolution, it kind of has to be, otherwise we’re back to path tracing the image, the thing we wanted to avoid. So, that low resolution creates problems for us, just like if you have a very low resolution texture and try to represent accurate details with it - you’ll see more and more problems as resolution goes down, same with the irradiance field.

Here’s a very interesting set of screenshots from the original DDGI paper:

It’s very clever and subtle, so subtle you might not catch it. DDGI paper promises a robust solution to global illumination, no tweaks, no fiddling. But let’s have a look here:

classic probes, we see the light leakage all over - bad, yeah.

biasing via normal makes things smoother, kind of, but there are more leaks now. The smoothing comes partially from oversampling probes that have nothing to do with us. Okay, let’s see where this goes.

Using a low-resolution depth map we can figure out which probes are behind the surface we’re trying to shade, it’s not perfect, you can see the spikes indicating so-called self-shadowing, similar to errors in standard shadow maps. I mean, it’s better now in the sense that we have less light leackage that we started with, but I think every sane person will agree that this looks ugly and is way more distracting than the “classic”. Here it again for comparisson:

So how do we solve it? Ah, well, with a fudge factor of course!

el voilá

Here’s the relevant bit from the paper

Let’s take a look at the screenshots from the paper itself:

looks good, I can see some errors, but nothing major.

Looks fine as a screenshot, but if you look closer there are leaks, and quite major ones:

here’s the image without DDGI for comparisson

Here’s a more interesting one:

lets enhance first row

If you look closely you’ll notice that none of these are the same, there are artifacts in every one of them and different ones at that.

And if you think I’m cherry picking, here is the second row:

in case it doesn’t pop out to you:

This doesn’t mean that probes have no chance, they still have excellent performance characteristics. This does mean that probe placements are quite important and that pesky self-shadowing bias needs to be addressed.

DDGI in Shade

Back to shade. To establish a baseline here’s what render looks like without GI:

And for comparrison, here is the path traced version with 512 samples/pixel, we’ll assume that to be our ground truth.

Let’s try different grid resolution for probe placement:

Pica Pica

2x

4x

x8

x16

x32

Kitchen

off

ground truth

x2

x4

x8

x16

[Usnul]

Few screenshots with shadows, no denoising, so just 1 sample per pixel

All of the above are with the GI off, here’s with GI on just for reference

2 caveats:

• TAA is on
• SSAO is on

[Usnul]

Decided to work on SSR, got surprisingly far in just one day. Here are SSR targets (not composited with the main render, so just the reflections).

A few things here:

• reflections are recursive, meaning that things reflect multiple times.
• reflected rays disperse based on GGX pdf, that is - more diffuse something is - less mirror-like the reflections.
• roughness cutoff, this is a pretty standard optimization where surfaces that are “too rough” are skipped, the only note here is that there’s a fading curve applied towards that cutoff to hide any artifacts
• edges of the screen fade out. Again, this is a very standard trick to hide bogus reflections at the edges that tend to flicker with tiny camera movements.

Anyway, still have to add denoising here and composit, but I’m pretty happy with the overall look. Also the noise function for GGX sampling is rather poor and will need to be replaced.

The issue with SSR, at least in my mind, is that the reflections are adding more energy, and tonemapping is already done, so we somehow have to do SSR before tonemapping.

[Usnul]

Switched from hash-based random to a dedicated 3d blue-noise (well, it’s actually scrolling 2d, but who cares).

The point is - blue noise is pretty great. Who knew? Nah, not really, I use it everywhere else already.

There are some other visible differences, because I messed up the normals before. Also added more ray termination checks.

I set out to create this (SSR) thinking that the ray tracing is the hardest part of all of this and then you just sort-of blend it with the rest. Turns out it’s far from being the case.

You trace the rays, then you have to resolve the hits using blurred color mip chain, sort of like with the pre-filtered environment map. Then you need to do reprojection which is tricky because reflections live in a different plane, and only then you can start thinking about compositing. A pain basically.

[Usnul]

As it usually goes, found more problems with BRDF code and ray marching, here’s the latest:

Added some very shoddy compositing into the mix, no denoising yet:

The effect here is very subtle as intensity of reflections is currently modulated by metalness, so less metallic surfaces produce fainter reflections.

More pronounced effects are visible in metal, here’s a bike shot for comparrison:

here’s without SSR:

here’s a scene from blender 2 splash:

and here’s without SSR:

you can see a bunch of highlights disappear. Here’s just the SSR for clarity:

A bit disappointed with the overall impact, that is - a lot of work for relatively few pixels on the screen changing color. Granted, it’s cool and important but I’m going to look for the ray way to blend reflections with diffuse and dielectric surfaces as well.

[Usnul]

Still working on postprocess.

Added spatial denoise to SSR pass.

A few screenshots with SSR on/off

Spending most of the time on reprojection. Turns out to be the hardest part of this whole thing.

For now working in the context of TAA, here are a few screenshots of TAA up-close (4.6x zoom).

Without

With

Without

With

Without

With

[manthrax]

Is there a reason that the screenshots are overall relatively dark? Is there no ambient term or sky contribution to lighting, or is it a tonemapping thing?

[Usnul]

Heya, there is tonemapping going on, it’s the standard ACES one.

As for ambient, I wanted to avoid handwavy lighting as much as possible. So the intention is for lighting to be as physically realistic as I can get it. So no ambient term.

To demonstrate a bit better what I mean, here’s what the bike looks like without the shadows, SSR and anything else, just straight direct light:

with shadows

and SSR and SSAO

and Light probes

Light probe solution is not robust enough to just be switched on yet though.

I’m thinking to add an environment map as a stop-gap solution for now to add ambient lighting

[Usnul]

So after way too much time spent working out kinks in SSR (screen space reflections). I discovered FidelityFX stocastic SSR, which was close-enough to what I already had to integrate with.

Key takeaways:

• reflections are now traced using hierarchical depth buffer, which means that even mirror reflections converge in ~20 taps, without any tunneling or smearing.
• a bunch more validation metrics for rays to ensure we don’t get gibberish results

And just as before, we’re still working with surface roughness to decide which way to cast a ray. This means that when roughness is low - we get very sharp reflections and when roughness is high - we get very smooth reflections.

Here are some spheres to demonstrate:

Here’s a close up:

and with SSR off

Here’s the same bike scene from before:

With SSR off:

And here’s a zoom in on the side mirror, what’s in the mirror is purely SSR:

Here’s the PicaPica scene:

there aren’t many smooth surfaces here, so the effect is more subtle, here’s with SSR off:

Other than the SSR, I finished work on environment support, and integrated it with the sky simulation, so the scenes are lit with realistic sky, integrating the following:

• current altitude
• mie/rayleigh/ozone scattering/absorption (multiscattering, not just single)
• sun position (taken from directional light)
• sun color/intensity

The sky is based on Epic Games paper “A Scalable and Production Ready Sky and Atmosphere Rendering Technique” by Sébastien Hillaire

The pre-filtering part for the environment map is worth a few words. I didn’t like equirectangular projection, and I didn’t like using a cube map. Both felt unpleasant for different reasons. So I ended up going for an octahedral projection instead. I already had all of the code for that in the engine. You get less distortion, you utilize the entire set of texels and it’s a single image, unlike a cube map, so filtering is much easier and efficient.

For now I’m concentrating on removing the noise from reflections.

[PavelBoytchev]

Reflections are nice. Is the max number of reflections configurable? Or your approach does not need such configuration?

Also, do you have plans to tackle issues with borders of shiny metal objects? Sometimes artifacts like these may ruin all the efforts spent for making good shadows, illuminations and other light-related prettiness.

[Usnul]

Turns out it was specular antialiasing at fault, fixed

thanks for the motivation

In case anyone is interested, Shade implements specular antialiasing based on “Filtering Distributions of Normals for Shading Antialiasing” by A.S. Kaplanyan et al (NVIDIA) 2016

The problem was in the filter region getting too large at the edges of geometry, since we’re sampling right next to the background.

Also, to illustrate the difference that this makes, here’s blender’s scene in Shade:

and here it is in three.js (dont’ mean to pick on three.js )

if we zoom in on cog teeth:

and three

you can see a ton of aliasing in three.js, the problem gets worse under motion

[Usnul]

Working on SSR reprojection.

Without SSR

Here’s raw SSR resolve:

Here’s SSR with 1 spatial denoiser pass

2 passes

3 passes

Throwing temporal denoiser into the mix:

Better variance guiding metrics for temporal reprojection

Further biasing history mix based on variance:

Enabling TAA

Still have some fireflies under fast motion, but pretty incredible what you can achieve with a really noisy input.