You can set something like this: renderer.domElement.style.borderRadius = "10vh";
(or do the same with CSS).
Clipping also works: Tamalék [Catmull-Rom in vertex shader]
.hex{
/*border-radius: 200px;*/
-webkit-clip-path: polygon(50% 0%, 95% 25%, 95% 75%, 50% 100%, 5% 75%, 5% 25%);
clip-path: polygon(50% 0%, 95% 25%, 95% 75%, 50% 100%, 5% 75%, 5% 25%);
}