Most notably, the AI took liberty with shapes, but there is something useful in there. For example, the light is much stronger and warmer in the AI’s version. I was using standard D65 for my lighting, but AIs version is closer to 3500K. AI’s version has very strong Fresnel on the near column, which suggests boosting glossiness and perhaps making the stone a bit less of a dielectric. Finally, the “fuzz”/“sheen” on the cloth helps sell the visual.
I have some classical art education and I worked in game development for a long while, so I feel like I have a pretty good eye for aesthetics already, hence the value of this technique is more limited for me personally. But I thought that this is a really cool idea, and wanted to share it in a broader context.
Yes, thank you for sharing. I also saw that presentation, and I think it’s an interesting idea. Just one point: in the video, all of this points to asking the AI to suggest aspects of a scene that would make the final finish of an image look more “realistically believable”, things that one might have forgotten when constructing the scene from scratch, that is, an environment created from your imagination.
When working with real-world scenes, this is a little different. At best, it might suggest revising one or two roughness maps, but in general, the suggestions will be more along the lines of changes in the hue of some materials, colour temperature in lighting, and weather conditions, for example. In other words, if the rendering engine you are building is already governed by PBR rules that are oriented towards ‘physical correctness’ in image generation, then what this AI suggestions approach can help with is not the “fidelity” of the final image, but how to make them ‘more aesthetically pleasing’.
I think this idea was already mentioned in your post, but I wanted to emphasise the application of this approach in a reality capture context, which could be different from when you are creating a world from scratch and you don’t know the PBR parameters of the materials, for example.
Just my two cents. Many thanks for your inspiring posts.
Totally agree. And yes, the presentation was in the context of artistic renders, and not real-time graphics.
Consider it like this, you’re building a new three.js app for some reason. Could be a cool idea, could be a customer project.
You could go nuts and just throw everything you possibly can at it, pull in SSR, SSAO, some kind of light probe solution, volumetric fog, multiple directional lights for nice backlighting, ultra-high-res environment map etc etc. However, it takes time to bring in every one of these techniques.
And it takes performance headroom. So, what if you could instead set up something very basic, take a screenshot and throw it at the AI.
You get something back, so you pull it apart in your mind and figure out which strategic pieces you need to make the end-result look close to the AI’s rendition.
At the very basic level, I think lighting and colorgrading will be a part of the puzzle that AI can help solve for you.
Physical correctness, I find, is not always desirable. Here’s a random picture I pulled off the internet (upscaled)
The tree looks boring and flat, because there’s very little microshadowing going on, the roughness of the bark is very high so the entire surface looks incredibly uniformly lit.
The grass is so short that it looks painted on. The trees in the distance on the left alias into oblivion and look like a gray mess.
The building has virtually no definition to it and look like it’s made out of paper. The shadows are very dark, which makes it look like a very low-end rendering.
I used to think that realistic renders will make everything looks good by default, but there’s a lot of artistry involved. Realistic rendering tends to give more complex interactions. A red wall will reflect red light onto other objects in the scene, a shiny surface will reflect other objects in it and shadow casters will block the light etc. But in the end, color grading, bloom and things life DoF tend to carry way more “feeling” for the user than GI and excellent BRDF
And back to the AI, here’s that same photo from before pushed through an AI
