Is it even possible or do I have to use a different material than ShadowMaterial? I noticed that it works with MeshStandardmaterial for the ground but this leads to a different color of the ground due to the light.
ShadowMaterial does not work since it is a transparent material. The transmission pass only works with opaque and transmissive objects so meshes with ShadowMaterial are discarded.
You can disable transparency and forcibly set the options automatically set for transparent objects which enables the shadow to render with blending in the opaque objects pass. As you can imagine it’s a bit more complicated to understand and maintain, though.