Enabling cast shadow on meshes makes frustum culling less efficient

I noticed the same thing and was told this was normal behavior. Was I mislead?