Using raycast to allow mouse selection of points is extremely inaccurate. Looking at the code, the Points raycast method models a point as a sphere, but the Points rendering seems to be more like a sprite: a point is effectively a flat object, always facing the camera.
More specifically, a sphere mesh near the edge of the near plane is elongated rather than exactly round, but a Point with a disc map in the same location is exactly round. So a raycast near but not on a Point will return an intersect with the Point due to the elongation. With many overlapping points, it is unusable.
This seems like a bug. Is there some reason the raycast method models a point as a sphere? Seems like it should instead model a point more like a circle mesh.