Three-mesh-ui ⏹ VR UI (user interface)

Hi,

I’m making a small add-on for three.js to help with creating VR user interfaces.

Repository : https://github.com/felixmariotto/three-mesh-ui

It’s not a framework, just a small library to be plugged in the last three.js. It has no other dependency than three.js.

Please post your suggestion in issues on the repo, it will help me decide what new feature to prioritise.

currently support :

  • word-wrap on defined or default characters
  • “flex-box” (like CSS)
  • justify content, align content (like CSS)
  • automatically merge characters geometries for better performance
  • different text sizes and colours in one paragraph
  • inline blocks (principally to include images like emojis)
  • state management (buttons)
  • MSDF font for large text rendering
  • texture mappings like CSS ‘contain’ and ‘cover’
  • virtual keyboard with various language layouts
  • hidden overflow (like CSS)

nested_layout_opti


big_text_opti keyboard_opti

12 Likes

Sweet! :heart_eyes:

1 Like

Really great stuff!!

1 Like

@DolphinIQ @makman Thank you guys !

It’s going forward, now there is a simple API to add dynamic buttons, which you can style however you want :

teaser_button_optimised

Live demo (you can use your mouse)

1 Like

@felixmariotto This is great! Awesome to see someone is working on this – I had a need for something like this recently and was thinking about putting something together myself. This will make my life a lot easier.

A few thoughts and things I noticed when looking through the repo and examples:

  • I know you’ve mentioned that you’re building this for VR but there are a lot of use cases in which this can be used for conventional three.js, non VR applications (mine included!). It would be nice to keep this agnostic and not make any assumptions about VR in the library. Not that there are any being made now just thought I’d mention it.

  • I see that there’s a 0.01 unit offset being applied for every level of nesting in nested block. Is this to avoid z-fighting when rendering two planes that render to depth? Have you considered using Material.polygonOffset for this? That should allow you to keep all layers on a single plane if the user desires.

  • Have you considered using signed distance field fonts rather than the mesh ones being used now? Then you could just use a plane per character – the SDF type may be more complicated to implement, granted…

  • Have you looked into Yoga.js, at all? It’s a basic layout engine built to support some flex layout behavior which could be leveraged for 3d UIs. It probably doesn’t belong in your library, though, and could instead be used in tandem.

Thanks again and great work!

1 Like

@gkjohnson thanks again for your support and feedback !

I know you’ve mentioned that you’re building this for VR but there are a lot of use cases in which this can be used for conventional three.js, non VR applications (mine included!). It would be nice to keep this agnostic and not make any assumptions about VR in the library. Not that there are any being made now just thought I’d mention it.

  • Yes I guess that there is other needs for it, so I’m trying not to tie this to VR, but on another hand I don’t precisely know what could be these use-cases. I would be super interested to hear about your use-case and why you don’t use CSS3DRenderer or other available solutions (it would help me understand what features should be prioritised).

I see that there’s a 0.01 unit offset being applied for every level of nesting in nested block. Is this to avoid z-fighting when rendering two planes that render to depth? Have you considered using Material.polygonOffset for this? That should allow you to keep all layers on a single plane if the user desires.

  • I know Material.polygonOffset but so far I’ve made the choice of letting the users create their own materials. I think I could set polygonOffset parameters to the materials they pass, but I’m not very fan of doing this, I think people expect the material they pass to be used as-is. However I’m realizing that working with users materials is a huge limitation for my lib, and there will probably be an update shortly, so that the user can pass parameters to apply to a library-created material. At this point the default offset will be 0, and materials will have polygonOffset default values set to avoid z-fighting. I think users should still have the possibility of using a custom material though.

Have you considered using signed distance field fonts rather than the mesh ones being used now? Then you could just use a plane per character – the SDF type may be more complicated to implement, granted…

  • I still don’t fully understand that but this is definitely something I’m planning on implementing soon !

Have you looked into Yoga.js, at all? It’s a basic layout engine built to support some flex layout behavior which could be leveraged for 3d UIs. It probably doesn’t belong in your library, though, and could instead be used in tandem.

  • I didn’t know this lib, do you use it ? I guess you mean that the users could generate three-mesh-ui code with Yoga ?

Very exciting… grabbing my popcorn to watch! Keep up the good work.

1 Like

but on another hand I don’t precisely know what could be these use-cases… I would be super interested to hear about your use-case and why you don’t use CSS3DRenderer

For my specific use case I’m labeling engineering data in space and I’d like to do a perform a draw-through effect for the labels which isn’t possible with CSS3D or HTML approaches. It’s definitely a simpler case but for games or other art applications being able to render a layout and apply shader effects to it would be useful. Having an option for SDF or 3d text meshes for the latter case might be nice, too.

However I’m realizing that working with users materials is a huge limitation for my lib, and there will probably be an update shortly, so that the user can pass parameters to apply to a library-created material.

Having a smart default material would be nice or reserving the right to modify certain parameters of the material might be good alternative solutions.

I still don’t fully understand that but this is definitely something I’m planning on implementing soon!

:tada:

I didn’t know this lib, do you use it ? I guess you mean that the users could generate three-mesh-ui code with Yoga ?

I haven’t used it but I was looking at Yoga to handle layout when I was thinking about this so I thought I’d share. And yeah I’m imagining that a user could prep a layout with yoga and then set the x, y, width, and height of three-mesh-ui components.

Anyway I didn’t mean to throw a bunch of feature requests at you – I’m looking forward to seeing how it evolves!

1 Like

This is some nice work! I’m glad to see someone taking on this challenge in the context of pure Three.js.

I’ve implemented something very similar to fulfill the exact same needs, but specifically within our own scene management framework Troika. It works very well, but I haven’t publicized it to the community because (1) it requires buying into using Troika as a framework for the whole scene, which isn’t something I’m comfortable pushing, and (2) it’s almost entirely undocumented, which is my own failing. But, I’ll point you to its source and describe some of the differences, in case you want to steal any of its ideas. :slight_smile:

Here’s the source code for reference: https://github.com/protectwise/troika/tree/master/packages/troika-3d-ui/src

And a couple examples using it:
https://troika-examples.netlify.app/#globeConnections
https://troika-examples.netlify.app/#ui


This does use Yoga, as others have suggested. This gives it full flexbox capability, including sizing that depends on text length. Yoga is delivered as ASM-compiled code which makes it run fast, though I do still run the entire layout process in a web worker to avoid frame drops which is very important in VR.

The main downside of Yoga is that it’s big. Like >350K big. I’m hoping that improves with Emscripten compiler improvements but can’t count on it. I haven’t found any smaller solutions that have anywhere near its capability.

For building the UI structure I’ve chosen an abstraction where you can “decorate” any component as a FlexNode. I provide builtin decorated FlexNode components for 2D panel blocks, but you can also wrap any 3D object this way to let it participate in the overall flexbox layout. So for example you could have a globe of the earth that flexes within the layout and is sized to fit in its resulting box, but isn’t constrained to a 2D plane.

For common flat 2D panels, it supports backgrounds and borders with solid colors or image textures via the material. It also supports border-radius for each corner. Currently it does this by modifying the fragment shader, which works well enough but has the downside of requiring the material to be transparent and strictly controlling the renderOrder to get proper antialiasing. I’ve got ideas to improve that in the future.

I’ll echo others and suggest using the polygonOffset material properties for the coplanar layers to avoid z-fighting without visible z shifts. Sometimes a z shift can be nice in VR as an extra affordance for buttons and the like, but often you don’t want that. Your point about letting users supply their own material instance is a good one, and I get around that by inheriting from their material via Object.create(baseMaterial) so I can tweak individual properties without affecting their instance. That may be something you could try too.

For the text, I use our own SDF text renderer troika-3d-text – out of all these pieces I’d suggest you take a look at possibly using that for your library. It’s got a standalone Three.js class you can use independently of the Troika framework, and it has some significant advantages over any other SDF text solutions I’ve seen, such as being able to use web font files directly and using standard Three.js materials. It’s fast and folks like Mozilla MR are using it successfully, and I’m actively improving it, so give it a look.

Something yours does that mine doesn’t is mixing text styles within a single block. That’s a great feature and something I’ll definitely consider adding. :slight_smile:


Again, this is great work! I hope something in here is helpful to your effort, but if not, keep up the good job!

2 Likes

big-text-reddit

three-mesh-ui now has its own implementation of MSDF text, dependency-free.
It is slightly better than SDF text, more about this here.

live demo

2 Likes

Added support for backgroundSize, based on CSS background-size.
You can choose “stretch”, “contain” or “cover”.

Also added rounded edges

const block = ThreeMeshUI.Block({
	height: 1.1,
	width: 0.6,
    borderRadius: 0.05,
	backgroundMaterial: textureMaterial, /* any three.js material with a texture */
	'cover' /* or contain or stretch */
});

Live demo

3 Likes

inline_block_opti hidden_overflow_opti

Support for inline blocks and hidden overflow !

Inline block live demo

Hidden overflow live demo

2 Likes

Wow this is some amazing stuff! Great work!

1 Like

Thank you ! I’ve made a tutorial here if you’re interested in giving it a try.

This library is what I need! Thanks!
I have made a quick test by HTML tag, but I get “Uncaught ReferenceError: module is not defined at three-mesh-ui.js:1”
Do you know why?

1 Like

Sorry about the inconvenience, I did actually messed up something with the build.

It’s fixed with version 4.2.1 :

<script src='https://unpkg.com/three-mesh-ui@4.2.1/dist/three-mesh-ui.js'></script>
<script type="text/javascript">

// the lib is here
console.log( ThreeMeshUI )

</script>

Thank you!