displacementMap works only on the GPU, while raycaster works only on the CPU - to raycast against a mesh that’s using a height map, you’d need to apply the heightmap on the CPU for each vertex (mind, this can’t really be done in real-time for extended periods of time, so best to do it once during the loading of the map for example).
Quite a few topics with sample implementations around - example.