Having Geometric Specular Antialiasing would be incredible

[Arecsu]

Hello! How’s going?

I’m working on a offsceen canvas model visualizer for fine art prints that we make in our creative studio. It’s a PBR workflow and so far, it looks really good. However, at certain angles, the aluminum material from the back of the object will cause a lot of specular aliasing on it.

It was a lot worse before because the front face of the aluminum object (just behind the artwork plane) would be lit by the environment map, so I had to set a black 1.0 roughness material to it. Good workaround! Still, what’s left is the specular aliasing from the sides of the aluminum back.

Things that I’ve tried so far:

• FXAA. It only made everything a bit blurrier. Specular aliasing kept being the same.
• MSAA. It does improve things a bit, but not that much. Going above x4 not only did not improve specular aliasing at all, but it also became a resource hog. My implementation runs a small benchmark to determine if MSAA should be enabled, and also if resolution should be lowered. MSAA would be disabled in a lot of mobile devices, resulting in worse aliasing. Interestingly enough, geometry aliasing is not that bad compared to the specular ones, which are really distracting and prominent.
• SMAA. Compared to FXAA, targets the edges a bit better, leaving the rest of the image looking sharper. Still, like FXAA, it didn’t help at all.
• Setting the HDRI map to work with PMREM. AFAIK, it should work out of the box without really calling PMREM manually. It did nothing, it looked the same.

Things that will make it worse:

• Beveling and curving the edges of the glass and aluminum in Blender. While it does make it more visually realistic at certain angles (really, it is an upgrade in realism) and I would love to have it, it caused soooo much specular aliasing, fireflies and sparkling at the silhouette that I can’t justify to have. I also made sure the normal map of the aluminum didn’t map in the bevels, but still the problem persisted. It even happened with the glass. The current demo in the website is the model without bevels because honestly, it was that bad. Not even MSAA x8 could save it.

Here is where Geometric Specular Antialiasing comes in

I do believe this is a problem that should be mitigated from the shader pipeline. Valve explored this concept years ago when they began exploring VR. Highly recommended to watch their talk. They had to tackle aliasing as best as possible in both performance and visual terms, because having aliasing + low framerate in VR it just degrades the experience so much.

They came up with a cool math which, to my understanding, will increase the roughness values of the pixels that will be prone to have high specularity values when viewed at specific camera + geometric angles. Correct me if I’m wrong about it.

Here is their code:

// Dense meshes without normal maps also alias, and roughness mips can’t help you!
// We use partial derivatives of interpolated vertex normals to generate a geometric
roughness term that approximates curvature. Here is the hacky math:

float3 vNormalWsDdx = ddx( vGeometricNormalWs.xyz );
float3 vNormalWsDdy = ddy( vGeometricNormalWs.xyz );
float flGeometricRoughnessFactor = pow( saturate( max( dot( vNormalWsDdx.xyz, vNormalWsDdx.xyz ), dot( vNormalWsDdy.xyz, vNormalWsDdy.xyz ) ) ), 0.333 );
vRoughness.xy = max( vRoughness.xy, flGeometricRoughnessFactor.xx ); // Ensure we don’t double-count roughness if normal map encodes geometric roughness

// MSAA center vs centroid interpolation: It’s not perfect
// Normal interpolation can cause specular sparkling at silhouettes due to over-interpolated vertex normals
// Here’s a trick we are using:

float3 vNormalWs : TEXCOORD0;
centroid float3 vCentroidNormalWs : TEXCOORD1; 

// In the pixel shader, choose the centroid normal if normal length squared is greater than 1.01

if ( dot( i.vNormalWs.xyz, i.vNormalWs.xyz ) >= 1.01 ) {
  i.vNormalWs.xyz = i.vCentroidNormalWs.xyz;
}

The talk slides where they mention this are here and the specific timing in the video presentation is here.

My (failed?) attempt to implement it

I’m just getting started in the 3D world. I know pretty much nothing about shader code. All I could do is dump the compiled PBR shader code in three.js and tried to inject Valve’s code where material.roughness is calculated. Although I could load properly load the shader, it didn’t seem to antialias anything.

Valve’s code to Three.js looked like this:

vec3 normalW = inverseTransformDirection( normal, viewMatrix );
vec3 nDfdx = dFdx(normalW);
vec3 nDfdy = dFdy(normalW);
float slopeSquare = max(dot(nDfdx, nDfdx), dot(nDfdy, nDfdy));

float geometricRoughnessFactor = pow(saturate(max(dot(nDfdx, nDfdx), dot(nDfdy, nDfdy))), 0.333);

material.roughness = max( roughnessFactor, 0.0525 );
material.roughness = min( max(material.roughness, geometricRoughnessFactor ), 1.0 );

Note that I might be completely off in my assumption of what vec3 normalW and the rest of the values are because as mentioned, my knowledge about shaders is almost non existant. I also skipped the centroid part because I have no idea how and where to place that code.

Here’s geometricRoughnessFactor dumped directly using gl_FragColor = vec4(geometricRoughnessFactor * 6.0, 0.0, 0.0, 1.0);

Keep in mind I’ve amplified the value x6 to make the lines at the edge look better in the screenshot.

Moving the camera results in smooth edges reveal with little to no aliasing. But for some reason this doesn’t really translate into further specular calculations at all. Seems like the values are not sufficient, although I’m not sure.

Thoughts? Can this be implemented in three.js?

I believe specular aliasing is one of those problems people would quickly point out TAA as the solution to it. I don’t think it is. If the problem lies in the shader pipeline, TAA would be a rough patch on top of it. It won’t solve the problem completely. Furthermore, proper TAA with motion vectors (if I understand correctly) would be a complement to proper geometric specular antialiasing implementation, creating smooth pleasing edges when objects and cameras are in motion.

I do think Valve’s approach is pretty clever when it comes to good performance + lovely visuals. It is a low hanging fruit! Tackling this from the shader pipeline would be a good idea, or at least to have it as a toggle.

And please if someone knows how to properly modify the shader code to make this to work, it would be incredible <3. Thank you!

[manthrax]

That aliasing looks it might be related to the Bloom applied?

There’s are a bunch of different antialising techniques available in various forms for threejs… (as a post process)

MSAA, SMAA, TAA, FXAA, to name a few…

all of these are really sensitive to having the renderer set up correctly… renderer dimensions being set up correctly. Post procesing pipeline being set up correctly… passes appllied in the right order…

sample count settings on the rendertargets…

There are a lot of little parts to get right.

So there may still be some hope that you can get an existing solution…

r.e. “modifying the shader code to make it work”… there is
https://ycw.github.io/three-shaderlib-skim/dist/#/latest/standard/fragment
site (and other similar sites) that let you browse the unrolled shader code for the built in materials. This can help a lot when trying to modify the built in shaders.

You can perform global modifications to all shaders by modifying THREE.ShaderChunks (iirc… forget what its called) to edit the chunks globally…

Or you can use material.onBeforeCompile = (shader)=>{

to intercept an instance of a materials shader before it gets compiled, and perform surgery on its vertex and fragment shaders.

[Arecsu]

Thanks for your answer. However, I managed to set up all the steps correctly, including the order of the passes and their respective resolution, with the appropiate pixel ratio and all that. Bloom is unrelated, it is basically the last pass and disabling it won’t change the specular aliasing. Even more, just rendering directly without any effects will still have the aliasing problem.

Also, as mentioned too, I didn’t have problems injecting vertex or fragment code into the materials. My method dumps the exact compiled shader of three.js (this is the fragment one) so I can modify it anywhere I want without having to manually monkey-patch. Nonetheless, this is not really the point I wanted to discuss. I can edit the shader but unfortunately I know pretty much nothing about shader programming, that I don’t know how to implement Valve’s solution to three.js PBR shader pipeline.

The purpose of this post is to share have a discussion about specular aliasing, and to explore ways to properly address it. Antialiasing methods are not efficient in tackling it. MSAA is the best when it comes to it, and yet there’s work to be done. Supersampling the image is very expensive and also won’t guarantee to fix it completely.

If someone knows or have an idea on what to do to at least improve things in a notable way, it would be great. I do believe a good solution lies in the shader pipeline when it comes to roughness and specular calculations

[gkjohnson]

If you’d like to see this in three, your best bet would be to create a minimal example showing the specular aliasing and making a three.js issue linking the paper as well as a compelling before / after image from the paper or other reference implementation showing the improvement. Ideally your minimal example would use the same or similar model to your comparison screenshots but I know that may not be possible. At least someone should be able to point you to the right area where your fix code can be added or provide some more insight into whether there would be interested in adding this to the project.

I’d also recommend keeping your issue in the repo short and to the point. Detail can be nice but longer issues become more difficult to follow and I’ve noticed can lessen engagement. I might also take a look at what else is out there since it looks like there are some more recent papers on the topic.

[kpachinger]

Remove sub-pixel artifact (no background):

• render 2x and downscale… stencil ROI?
• bitwise not some shallow 45° edges mask
• depth test

[aaaidan]

Hi @Arecsu. I’m just catching up here, so forgive me if I miss (or misunderstand) something.

From what I’m reading it seems like you’re mostly wrestling with the limits of MSAA, rather than specular aliasing in particular. For example, I’m imagining there’s no “curvature” or “geometric detail” getting squashed on these edges that are giving you trouble. It’s just that there are angles where it’s “random” how many sub-pixels intersect with the side of the object, and there are only a small handful of coverage ratios that 4 samples can give, leading to the classic “dashed MSAA line”.

Perhaps we’d see a very similar artifact with an unlit cuboid, when viewing the top face from a grazing angle, even with MSAA enabled? If so, I’d recommend you lean into the “fake and bake” philosophy, and try to find a way to represent that detail with textures.

For example, two slightly offset planes facing the user, but masked with alphaMap, would probably give you much smoother borders (because of texture filtering) than MSAA could ever do. You may also need to add a pretty extreme normal map to make sure the lighting appears to be bouncing from the sides rather than from straight ahead.

If you look at this close up, or from weird angles, the illusion fails, so I’m not sure if this exact approach is suitable for your project. I just mean to illustrate the lengths of charlatanism you might need to go to.

Here’s a rough demo of what I mean. Using a custom alpha map allows edges to be drawn using texture filtering, rather than MSAA. Antialiased geometry on left, alphaMaps on right.

And this is the sleight of hand, shown visually.

Again, apologies if I misunderstand what you’re wrangling here. This might serve merely as an illustration of a “realtime mindset”, rather than a specific solution. Good luck, interested to know what you think.

[aaaidan]

Here are some more screenshots from an iteration of the above demo, but without quite as much skullduggery. In this iteration, the sides “actually exist”, but because most of the edges are masked with alphaMask, the silhouette has smooth, alias-free edges in almost all situations.

While texture filtering doesn’t always yield perfect antialiasing, but it is clearly a lot smoother than the typical 4x MSAA.

This view shows how this version of the illusion bears closer scrutiny, and the whole trick doesn’t rely on the sides being not much thicker than a pixel.

[kpachinger]

I believe the facts of smoke and mirrors will persist despite whichever smoke and mirrors arguments are proposed.

[Arecsu]

Hey @aaaidan, thank you so much for your workaround. It’s been a long time since I’ve finalized the 3D implementation I’ve discussed here but I will definitely take into account what you’ve described, as it may be handy if I find some time in the future to wrestle with it again. It has potential to serve my project due to the fact there’s no really anything in the background (just black).

Nonetheless, I find important to share some findings since I’ve made this post that can potentially light upon this issue even further. I’ve noticed that if I render the entire pipeline in SDR, MSAA will be much, much more efficient at smoothing these bright edges, even at 2x, making it completely usable. However, in SDR, the whole model and lightning looks very flat. I don’t remember the exact details but there was no way to make it look better by changing tonemaps or exposures. The only way to make it look good was to use HDR in the whole pipeline.

This is due to the fact that antialiasing techniques struggle to get proper values to smooth the jagged lines under high dynamic range conditions, such as this particular case where we have very bright values over a black background around them.

There is this blog post which goes deeper about it: Graphics blah blah: HDR inverse tone mapping MSAA resolve . It also showcases a “inverse tone mapping resolve” technique to get around the issue. I didn’t try to implement it though.

Another related read:

A Quick Overview of MSAA

Quoting:

Before HDR became popular in real-time graphics, we essentially rendered display-ready color values to our MSAA render target with only simple post-processing passes applied after the resolve. This meant that after resolving with a box filter, the resulting gradients along triangle edges would be perceptually smooth between neighboring pixels3.

However when HDR, exposure, and tone mapping are thrown into the mix there is no longer anything close to a linear relationship between the color rendered at each pixel and the perceived color displayed on the screen. As a result, you are no longer guaranteed to get the smooth gradient you would get when using a box filter to resolve LDR MSAA samples. This can seriously affect the output of the resolve, since it can end up appearing as if no MSAA is being used at all if there is extreme contrast on a geometry edge.

[…]

A much simpler and more practical alternative to performing tone mapping at MSAA resolution is to instead use the following process:

1. During MSAA resolve of an HDR render target, apply tone mapping and exposure to each sub-sample

2. Apply a reconstruction filter to each sub-sample to compute the resolved, tone mapped value

3. Apply the inverse of tone mapping and exposure (or an approximation) to go back to linear HDR space

So, what I originally though was pure geometric specular antialiasing, seems more like HDR high contrast radiance values which MSAA is unable to properly deal with.

Quoting again the first link:

Usually when mixing HDR rendering with MSAA, the super-simplified pipeline looks something like this:

1. Render the scene with MSAA
2. Compute lighting with sub-sample accuracy where it’s needed
3. Resolve MSAA texture to a normal HDR texture
4. Postprocessing
5. Tone mapping
6. Bloom

However, we actually want to do the tone mapping per sample, not on the already resolved sample. If we don’t, we get this (at 4x MSAA):

And then using this HDR inverse tone mapping MSAA resolve technique:

[aaaidan]

Oh of course, I didn’t consider that at all! But you’re totally right that MSAA only works reliably with SDR (device colorspace) output colors. Tonemapping the average of four linear samples (wrong) is not the same as tonemapping four samples and then averaging that (right). Tonemapping is not linear, which means it also can’t be commutative (that’s a ten-dollar word, right there).

You seem to already grasp that very well, but I found it very helpful to work through four imaginary samples it out with a pencil (or even a spreadsheet). I’d highly recommend doing this if you haven’t, because it doesn’t take long and can make the problem a very concrete. Imagine four grayscale HDR subpixel samples like 25.0, 0.12, 0.10, 0.05, and consider what the final color would be if you tonemap before averaging them, vs after. (Spoiler, one way is correct, and the other way gives a very different, and very wrong, number).

This non-commutativity seems to be why ThreeJS (and other renderers) implement tonemapping within the fragment shader by default, despite the many benefits of deferring tonemapping to the end (such as a post process on the buffer).

This inverse tonemapping solution is interesting, and plausible. It might even work pretty well in practice but, because tonemapping is destructive - we literally throw away color information - I don’t think it is possible to get it working reliably. Reconstructing linear color from tonemapped (sRGB) color seems like a hard problem, if not an impossible one. As an extreme example, consider two linear colors, 15.0 and 25.0. Most tonemappers will output 1.0 for both inputs. There’s no way to tell (or “inverse tonemap”) what the original number was, without just straight up guessing. If you think about it, there’s usually an unbounded range of linear numbers that could have resulted in 1.0 in sRGB.

This “inverse tonemapping” concept is very clever and insightful, and may be the first step on a path to solving this. But on its own, it doesn’t seem like a complete solution. (Although I would be delighted to be proven wrong).

But back to your scene. I assume this means your renderer is tonemapping at the end, using something like the renderpass composer? Perhaps you are keeping linear colors until the very end, when you convert them to sRGB with a tonemapping/encoding pass. A great thing about linear color is that it can be especially useful for bloom (and similar effects), since that extra dynamic range can often yield more realistic highlights. Other passes might also benefit from linear color, but I can’t think of any where it’s technically necessary to get the most out of them (color grading might be an exception, not sure).

If you’re keen, I’d recommend experimenting with your composer pipeline, reworking it (perhaps on a branch). Try moving tonemapping back to the beginning by having it run on the render pass (in the fragment shaders) before MSAA happens. Unfortunately, this will mean outputting boring old sRGB for your post processing, which will probably break everything initially. You’ll need to make significant artistic adjustments to get the bloom looking good again with only sRGB colors, which may be challenging but I believe it should be possible. You may also have to adjust other passes (eg color grading?). A fair bit of work, but after doing all this, you should get much better MSAA quality. This smoother render will also give a post process antialiasing pass a pretty big boost, since it now has more info about edges. This improved MSAA alone may even let you decide to ditch FXAA/MLAA entirely.

I’d love to hear and see how this works out if you get time to try this out.

Having said all this, I also just want to caution that, even with MSAA working perfectly, it is never going to look excellent (like, raytraced). You can only expect “very good” edge quality when it’s working well, and it will always be possible to break it very badly, especially when rendering very thin or detailed geometry.

Hopefully pipeline changes are enough to fix all your aliasing problems to your satisfaction. But I suspect the edges of your picture frame are going to continue to show those dreaded “crawling worms” (although they should be a lot less obvious).

Creating a truly picture-perfect render (comparable to a raytraced image) will probably require a combination of these techniques. And, yes, some of which are definitely smoke and mirrors.

[donmccurdy]

@aaaidan

Tonemapping the average of four HDR samples (wrong) is not the same as tonemapping four samples and then averaging that (right).

Agreed that these results will be different — but I’m not aware of any reason that one or the other is more correct, assuming the render pipeline is making the right assumptions elsewhere.

On the other hand, consider a scene with alpha blending transparency. Transparency models a physical rather than perceptual effect, and we should be blending the linear open domain RGB values. With tone mapping in the fragment shader, that isn’t possible.

This non-commutativity seems to be why ThreeJS (and other renderers) implement tonemapping within the fragment shader by default, …

TBH I think the reason is primarily about performance. Requiring a post-processing pass can be very expensive on mobile devices, and we don’t want that cost to be mandatory for something as essential as tone mapping. Perhaps ideally we’d offer both options but there are some complexity/maintenance considerations there. Currently WebGLRenderer does tone mapping in the fragment shader and WebGPURenderer defers it to a later pass.

***

I try to think of tone mapping and the distinction between use of the Linear-sRGB working color space and the sRGB output color space this way. Before tone mapping we’re working with open domain (0–infinity) values with some direct relationship to scene light, including the RGB stimulus values received by our “camera.”

Tone mapping, color space conversion, and (optionally) color grading collectively represent “image formation,” the process of creating an image suitable for a particular medium. For example, that medium might be an HTMLCanvasElement (generally implies sRGB), displayed on an SDR monitor (implies peak brightness) in a darkened room (might imply color grading choices).

After image formation, we have a closed domain (e.g. 0–1), often non-linear encoding targeted for some real or hypothetical medium. The operating system or display device will do some further adaptation beyond this. I tend to reserve the terms “SDR” or “HDR” for describing the formed image, or the display hardware on which it is shown, and not the pipeline prior to tone mapping.

***

Reconstructing linear (aka HDR) color from tonemapped (SDR) color seems like a hard (if not impossible) problem.

I think your instincts here are correct that it’s an ill-formed problem, in the sense that working backward from the formed image to the original RGB stimulus received by the camera is both (1) very hard, and (2) greatly compromising your image formation choices. That said, it is possible to work backwards to some RGB stimulus value that would result in the same tone mapped color in the final image, and there might be some cases where that is useful, just be aware that it’s not a 1:1 inverse, and does not restore the original / ground truth stimulus at the camera. Smoke and mirrors, as you suggest.

And, not all possible sRGB values are “reachable” by tone mapping. Depending on the tone mapping, there will probably exist some sRGB values that no open domain input can produce as output from the tone mapper.

***

Anyway, just some thoughts about the render pipeline. Whether this is helpful for MSAA, or whether certain “liberties” need to be taken with regard to the theory, to get the results you want practice, is another question.

[aaaidan]

Fascinating read, Don. Thanks for the insights.

Your alpha transparency counterpoint is right on. That’s for sure the fatal flaw of tonemapping before MSAA: all polygon blending is done in output color space (aka sRGB), as though each mesh were on a separate photoshop layer.

In OPs scene, I don’t see very much transparency or blending effects, if any. In this kind of fully opaque render, this flaw doesn’t really show up. It only manifests as tiny and subtle inaccuracies around the antialiased edges of meshes and is practically invisible.

But yes, introducing any significant amount of partial transparency (or additive blending) this often sends the whole render very obviously and quickly off the rails. If changing the pipeline isn’t an option, this can require remedial smoke and mirrors. Otherwise you’re just left with a strong “PlayStation 2 era” vibe. (If that’s what you’re going for, maybe you can just enjoy it!)

Instead of these smoke and mirror fixups (which are bespoke and can be expensive to produce), deferring tonemapping to post process as OP has done can certainly be a better approach. Since this means blending can now happen in linear color, transluscent objects generally look a lot nicer and are more physically accurate. That’s great!

But this is a tradeoff. It swaps one problem for another. While blending now works great, this often utterly obliterates MSAA, creating lots of jaggies and worms (just as OP is seeing). In extreme cases, it can seem as though MSAA is not even enabled.

So, deferring tonemapping to a separate pass, after MSAA, can be a good call, but is also a tradeoff. Personally, I have come to try to accept this flaw in the default blending as “part of the medium”, rather than fighting it, solely to keep MSAA working at its best.

When there is no opacity/blending at play, early tonemapping is no problem. And that unlocks the detail, stability, and performance of MSAA. And you can always stack additional grading and AA passes. As long as you don’t need robust, linear blending, that’s hard to beat.

(As an aside, I’ll for sure stop using “HDR” when I really mean “linear color”. Thanks for that callout.)

[aaaidan]

One vague idea I’ve been turning over in my head for a while: would it be possible for the fragment shader to output the linear and tone mapped colors? Possibly using “multiple render targets” feature of webgl2?

I haven’t thought this through, but I wonder if this could mean we could somehow get the best of both worlds. Using linear color to do all the blending with the “behind” colors, but keeping MSAA intact by continuing to feed it tonemapped color.

This is kinda what I was thinking about when I said “untonemapping” seems incomplete, but could be a step toward a good solution

Does anyone know if anything has suggested or investigated this yet?