I think it’s fair to say that three.js and babylon.js are both full-featured WebGL frameworks, and neither is going to impose an upper ceiling on the performance of your project. But in any ambitious graphics project, the best performance usually requires some amount of measuring, tuning, and ‘tricks’. And all of those things require time and work.
Whether it takes more time/work to get great performance from three.js or babylon.js will depend a lot on what you’re doing.
Common performance techniques — like frustum culling, instancing, batching, LODs, spatial indexing, … — will available in both frameworks. Most of these require some work to set up. I’m not sure what a “3D visualization monitoring system” involves, to make a guess about which performance tricks are required, or whether one framework is better for that purpose.