Hey abd 
I think I had a similar problem that the guys in this forum helped me a lot with. Please see this post, it got actually solved and still runs like a charm.
The basic idea I hade to face was to figure out which objects use the same material. And all those meshes with the same material-properties (except color, wich can be set individually) get merged together into one object. Therefore I only have 1 draw-call for each object and just about 5-10 objects left and that works so good.
Another idea ist to put objects into group and then raycast only these groups. Each none raycasted group can be switched to invisible. That way you can probably reduce the amount of objects to be rendered. Though I have to admit, I also have never done this before.