Is there a library available in threejs that can optimize the arrangement of a group of meshes within a specified boundary? By nesting, I mean an algorithm that efficiently fits these meshes into the boundary, maximizing the use of available space while minimizing any unused or open areas.
for example open nest in rhino
Here is the website for OpenNest:
Would XAtlas do what you’re looking for? 
Or do you mean packing 3D models within a 3D volume?
Hmmm. i mean packing 3d meshes within a 3d volume/boundary. The goal is to fit the meshes within a boundary so there is minimal wastage. (ideally used for cnc and lazer cutting)
No 3d solution that I’m aware of, but you can build a 2d version for laying things out on a build plate using potpack.js
https://mapbox.github.io/potpack/
I’ve used it in the past to lay out shapes for laser cutting.
