Hi!
Are you looking for something like this?
https://codepen.io/prisoner849/pen/ROdXzw?editors=1010
Just another idea is to use polybool.js (related topic [Closed] How to split/separate a single ShapeGeometry into individual Geometries base on faces?) to work with polygons: find intersection of the polygon of your floor and the polygon of your four point rectangle, if the result (its area or coordinates of vertices or amount of veriteces) is not the same as your original rectangle, then your rectangle is out of the floor.