can you make a repro, using codesandbox? a chrome devtool performance tab snapshot would also be interesting so that you see where it spends time processing. im guessing it will be THREE.RayCaster.intersectObjects that takes the most time.
raycasting can be expensive depending on the model and how many vertices, but fiber optimises events, it shouldn’t cast every frame (which would be pointermove) if only click is registered.