I think the only workaround for that pixelation at the moment is to reduce or increase roughness. Roughness values in the middle of the [0,1] range tend to have more pixelation. Or (more advanced) you could modify the three.js library locally to have higher resolution for the transmission render target, but this reduces performance and needs to be a power-of-two size.
the roughness seems to help! I am noticing though, that with a meshStandardMaterial with a color map or color applied, the pixelation occurs, but if you apply a canvasTexture to this material instead, there is no pixelation issue, which I find pretty interesting! It seems like having a high roughness ( maybe a value higher than 0.5 or so?) with a base color or color map kind of disperses the texels at the edges outward (i have no idea what I am talking about here but thats what it kinda looks like is happening – like a spray paint effect almost) anyhow many thanks for the pointer! – also, a value of 0.01 roughness seems to tame the pixelation to a near non noticeable level!
I have no idea why a CanvasTexture would change the perceived transmission resolution, maybe something to do with the minFilter/magFilter mipmap settings? Could be worth filing a bug about if it’s simple to reproduce.
still haven’t made a repro but will once I get some free time. I noticed on your gltf viewer, the pixelation issue doesn’t exist (though the pixelation occurs when I use the same model locally). Is that due to the script you wrote about in your blog post on volumetric refraction?