The problem is that I cannot cast rays to all directions (only limited degree and distance).
Yes, three.js cameras are based on projection so they can always see much less than 180 degrees, but you could use 4 x 90deg cameras, or 3 x 120deg, or rotate one camera to cover 360 degrees. That would be the easy approach.
Or make your own special camera (I had made a distortion-free 180deg camera once in Flash, then combined two of them for 360 degrees for 3D spherical view, but I have yet to translate it to three.js code).
Or it might be possible to make a shader for speed.