Directional light flickering issues with shadows

I am trying to simulate sunlight using a directional light. In the process I am setting the size of the shadow camera values such that it fits the scene as tight as possible. However, I am facing a peculiar problem of light flickering at certain angles(when light behind meshes, occluded etc). Please refer to the attached picture below for an idea of the problem.


The below code reflects the settings of my renderer.

        let renderer:WebGLRenderer = new WebGLRenderer({ antialias: true, alpha: true, preserveDrawingBuffer: true });
        //set to false and clear manually. This is to make the viewhelper work
        renderer.autoClear = false; 
        renderer.shadowMap.enabled = true;
        renderer.shadowMap.autoUpdate = true;
        renderer.useLegacyLights = false;
        renderer.shadowMap.type = PCFSoftShadowMap;
        renderer.localClippingEnabled = false;
        renderer.outputColorSpace = SRGBColorSpace;
        renderer.toneMapping = ACESFilmicToneMapping;
        renderer.toneMappingExposure = 0.5;
        renderer.setClearColor(0x000000, 0.0);
        return renderer;

Shadow map size is the default value of 512x512 and Shadow map bias is -0.0005. The camera near and far values are 1 and 5000 respectively. I want to know what could be causing these flickering issues at certain camera angles.