If the point cloud is static, you can benefit greatly from employing a spatial index (still on CPU!). You will need to replace the raycast
method, as I did with Mesh.raycast here: Octree injection for faster raycasting/raytracing
The principle is that you build the spatial index once, and then use it to accelerate multiple searches afterwards. An octree is just one among several possible options, and my implementation above is a bit naïve.