When adding Physijs to my threejs, it brakes my rendering

Never used Physijs before. But you could try enable3d, which uses ammo.js as well.