Projecting texture onto mesh and applying result to mesh texture (for mesh painting)

Questions
, ,
1

Hello folks,

I’m looking to implement a simple sort interactive painting thing for a single mesh, as in, a user would click and drag, and paint would be applied to the meshes’ texture at the mouse position in a circular shape.

Should be straightforward to do a raycast from the camera to the object with the mesh, and from there get the corresponding uv coordinate on the texture, that’s the easy part I suppose.

Now I could just draw a circle directly on the texture, centered at that uv position, I think that’s how the people who made this example did it:
http://www.cartelle.nl/deathpaint/

However of course the circle would be distorted, because it is not actually being projected onto the mesh before being applied to the texture.

So I’ve looked into a couple threads on this forum about projecting textures on meshes, like this one for example: Texture Projection

However they all seem to be about doing this projection ‘at render time’ by using a shader, which makes sense but doesn’t really match my usecase, I’d like to calculate the projection once when the user clicks to draw, and then apply it directly to the texture every time – I can’t figure out a great way to do this, does someone have any ideas? Would appreciate it :slight_smile:

I’ve also looked at https://threejs.org/docs/#examples/en/geometries/DecalGeometry which looks interesting, but I’d have to create a new DecalGeometry on every brushstroke, which Ideally I would like to avoid

2

You can use a CanvasTexture with a second material layered over the main material (Update the geometry.groups to ensure both materials work correctly). On pointerdown, use a Raycaster to get the UV coordinates of the intersection. Convert the UV coordinates to canvas pixel positions, draw on the canvas at those positions, and update the CanvasTexture.

Here is a working demo to illustrate the concept: Three.js: Painting on a Mesh Using a CanvasTexture

You can also check this example with fabric.js:

1 Like
3

Sure, as I wrote, that’s the easy part :slight_smile:

My question is about projecting the brush texture (a circle or whatever it may be) onto the geometry from the perspective of the camera or a normal, and applying that to the texture, instead of just drawing directly on the texture, so that the brush is not stretched like it is in the demo you linked, and the one I linked in my post.

Another approach I’m considering if I can’t figure out how to project onto the mesh properly, would be doing several raycasts in certain radius and then drawing the brush shape from the resulting uv points instead of just a circle.
That might be ‘good enough’ for counteracting the distortion, will have to try and see :person_shrugging:

4

Sorry, I don’t understand what you mean by “not stretched”? AFAIK, unless you’re using some optical illusion technique, the texture will naturally follow the curves of non-planar geometries. Just like in real life, if you project a circle’s shadow onto a non-planar surface, it will get distorted.

Can you share any practical examples of the effect you’re looking for? Has it been implemented in any 3D software or game engines?

5

Like this? MonkeyPaint

3 Likes
6

@manthrax, impressive example as always!

After playing around with it, I think I understand what the OP is looking for. When you spray Suzanne’s eyebrow, there’s a curve behind it, but the spray doesn’t seem to apply the texture to the surface behind it. Instead, it’s drawing over the radius around the intersection point.

I’m not sure if the effect could be achieved with the SpotLight’s projected texture?

7

Yeah there are a number of approaches for resolving that. I have more advanced versions in commercial work that use a depth buffer and normal buffer to restrict the brush to visible geometry. The technique I developed/used here can be generalized to support painting with textured brushes, and for things like lightmapping or baking.

3 Likes
8

Sure! Sorry if my description wasn’t accurate enough, to put it in simple terms using the Demo you linked:

Screenshot 2024-12-30 at 11.55.38
Screenshot 2024-12-30 at 11.55.381526×782 58.5 KB

Because you are drawing a circle directly on the texture, and that texture is then mapped onto the 3D surface, the circle will be warped as in the screenshot when going towards the top of the sphere.

What I would be interested in, would be achieving a drawing mechanic that lets you draw a circle that will not appear warped like that from the perspective of the camera, or from the surface normal at that point (not sure which is better), so there would need to be some kind of projection onto the geometry, and I’d like the result of that so that I could ‘bake’ it into the texture.

A practical example of this would be these examples for projecting textures onto geometry:
https://github.com/marcofugaro/three-projected-material
https://discourse.threejs.org/t/texture-projection/3224

That exact effect is what I want, I want to project a circle onto the geometry from a specific angle – however these examples are doing that projection in a shader at render time for each frame, and the result is being shown in the viewport directly – I’d like to do the projection once when drawing, and then have it applied to the texture.

Yeah exactly, that’s kind of what I’m after! It’s just that I don’t want to create a bunch of SpotLights (or decals, or such) for each brush stroke that do the work of projecting the texture on each frame. After you make a brush stroke, the projection will never change, so I’d like to ideally calculate it once and then store the result in the texture and just render the texture as is.

That’s exactly it yeah! Seems like you are using the brush normal to do the projection (or whatever you may call it), I also just found this example: three.js - texture - paint which seems to project from the camera instead, but I like the effect with the normal more I think.

I haven’t looked into your ScenePainter.js too deeply but looks like you’re still using a shader and rendering that to a rendertarget which I assume is then used to texture the model being painted or something along those lines?
I’ll look into it more, thanks for linking the example, that helps a ton already!

1 Like