If you want a precise selection, this implementation of a lasso selection might help you: three-mesh-bvh - Geometry Collect Triangles
It’s meant for three-mesh-bvh (a specialized version of raycasting for single meshes) but the principle shouldn’t change too much.
If you are happy with a more rough one: isn’t it enough to calculate top x, top y, bottom x, bottom y of your selection and then using the same logic as in the box selection? The frustum is squared anyway so you still need to use some sort of square to select with it.