How to plot bounding box with respect to a point cloud

Hi, I am loading a Point cloud file using PCD loader.
I have also loaded certain bounding boxes based on some given coordinates, but I am not able to make these boxes properly aligned with the point cloud coordinates. The boxes are not in the correct position and are not grounded and seem to be floating above the point cloud.

Here is a snippet with which I loaded my point cloud and the bounding boxes:

private loadPCD() {
    const loader = new PCDLoader();
    loader.load(
      this.url,
      (points: THREE.Points) => {
        if (this.pcdLoaded) {
          return;
        }
        points.userData.boundingBoxAnnotationContent =
          this.boundingBoxAnnotationContent;
        const geometry: THREE.BufferGeometry = <THREE.BufferGeometry>(
          points.geometry
        );
        const material: THREE.PointsMaterial = <THREE.PointsMaterial>(
          points.material
        );

        geometry.center();
        geometry.rotateX(Math.PI);
        material.side = THREE.DoubleSide;
        material.color.setHex(this.defaults.hex);
        this.pointCloud = points;
        this.grid = this.setGridHelper();
        this.scene.add(this.pointCloud);
        this.populateBoundingBoxAnnotation(); // here I am populating the bounding box after //adding the point cloud to scene.
        this.scene.add(this.grid);
        this.render();
        this.saveCameraControlState();
        if (this.cameraControlsEnabled) this.animate();
        this.centerOfLidar = this.getVisualCenterOfPolygon(geometry);
        const count = geometry.attributes.position.count;
        this.numberOfPointsLoaded = count;
        this.numberOfPointsShowing = this.numberOfPointsLoaded;
        
        
        this.isLoading = false;
        this.event.emit({
          trigger: "lidar-data-loaded-trigger",
          loaded: true,
          referencedAnnotation: {
            resourceHasAnnotatedJSON: this.resourceHasAnnotatedJSON,
            annotationContent: this.boundingBoxAnnotationContent,
          },
        });
        this.pcdLoaded = true;
        this.initStats();
        this.evaluateDistanceFromOrigin(geometry);
      },
      (progress) => {
        this.progress = Math.round((100 * progress.loaded) / progress.total);
      },
      (err) => {
        this.isLoading = false;
        this.event.emit({
          trigger: "lidar-data-loaded-trigger",
          loaded: false,
          referencedAnnotation: {
            resourceHasAnnotatedJSON: this.resourceHasAnnotatedJSON,
            annotationContent: this.boundingBoxAnnotationContent,
          },
        });
        this.loadingError = true;
      }
    );
  }

 private populateBoundingBoxAnnotation() {
    if (this.resourceHasAnnotatedJSON && this.boundingBoxAnnotationContent) {
      const _g = new THREE.BoxGeometry();
      _g.center();
      _g.rotateX(Math.PI);
      const tempV = new THREE.Vector3();
      this.boundingBoxAnnotationContent.forEach((bb: BoundingBox) => {
        this.createBoundingBoxInstance(
          _g,
          this.defaults.boundingBoxConfig.color,
          this.getBoundingBoxCoords(bb.coordinates),
          tempV,
          bb.class_name
        );
      });
    }
  }

createBoundingBoxInstance(
    geometry: THREE.BoxGeometry,
    color: string | THREE.Color | number,
    coords: Coordinates,
    vector: THREE.Vector3,
    label: string
  ) {
    const material = new THREE.MeshBasicMaterial({ color ,wireframe:true});
    const cube = new THREE.Mesh(geometry, material);
    
    cube.position.x = coords.position.x;
   cube.position.y = coords.position.y;
   cube.position.z = coords.position.z;

    cube.rotation.x = coords.rotation.x;
   cube.rotation.y = coords.rotation.y;
   cube.rotation.z = coords.rotation.z;

    cube.scale.x = coords.scale.x;
   cube.scale.y = coords.scale.y;
   cube.scale.z = coords.scale.z;
    this.scene.add(cube);
    cube.updateWorldMatrix(true, false);
    cube.getWorldPosition(vector);
    vector.project(this.camera);
  }

With this I get a result like this:


But the expected result needs to be like this where the bounding boxes are rightly aligned with the points and are grounded:

Any help in this regard is appreciated.

pcd-loader.component.html (3.0 KB)
pcd-loader.component.scss (1.5 KB)
pcd-loader.component.ts (46.7 KB)
2.pcd (7.7 MB)

If anyone needs to recreate this, you can do the same in a boilerplate angular project and have a look.
Dependency : “three”: “^0.138.2”
Dev Dependency:
@types/polylabel”: “^1.0.5”,
@types/three”: “^0.138.0”,
@types/webgl2”: “0.0.6”,

Any insights into this problem? Anyone ? @prisoner849 @Mugen87

Any chance to provide JSON with the data for a couple of boxes?

Sure @prisoner849 here you go with the JSON data for the bounding boxes:

{
    "annotations": [
        {
            "class_name": "Vehicle: Truck:2",
            "class_id": 7,
            "coordinates": {
                "position": {
                    "x": -28.44411040997511,
                    "y": -10.908647499411018,
                    "z": 1.5639627284249258
                },
                "scale": {
                    "x": 6.527588845825132,
                    "y": 2.7053958469647084,
                    "z": 3.088753093568613
                },
                "rotation": {
                    "x": 0,
                    "y": 0,
                    "z": 1.1588591747591366
                }
            }
        },
        {
            "class_name": "Vehicle: Truck:3",
            "class_id": 7,
            "coordinates": {
                "position": {
                    "x": 73.14449593032623,
                    "y": 30.126947760894836,
                    "z": 2.020717003637529
                },
                "scale": {
                    "x": 12.30766479140657,
                    "y": 3.3607588655188474,
                    "z": 3.998681604635891
                },
                "rotation": {
                    "x": 0,
                    "y": 0,
                    "z": -0.5487445375123619
                }
            }
        },
        {
            "class_name": "Vehicle: Truck:1",
            "class_id": 7,
            "coordinates": {
                "position": {
                    "x": 25.01690032775514,
                    "y": 2.263714672830247,
                    "z": 1.9643663735558619
                },
                "scale": {
                    "x": 12.745152036697608,
                    "y": 3.4765292357818036,
                    "z": 3.7974907141831835
                },
                "rotation": {
                    "x": 0,
                    "y": 0,
                    "z": -0.5331106969643833
                }
            }
        },
        {
            "class_name": "Vehicle: Truck:4",
            "class_id": 7,
            "coordinates": {
                "position": {
                    "x": 52.67422455341148,
                    "y": 27.45185899677017,
                    "z": 2.0037544675777923
                },
                "scale": {
                    "x": 12.555134377146214,
                    "y": 3.065328958835853,
                    "z": 4.1331076282010315
                },
                "rotation": {
                    "x": 0,
                    "y": 0,
                    "z": 2.7345906594565985
                }
            }
        },
        {
            "class_name": "Vehicle: Truck:5",
            "class_id": 7,
            "coordinates": {
                "position": {
                    "x": 11.870985735571594,
                    "y": 38.88986719108652,
                    "z": 2.4780189122414527
                },
                "scale": {
                    "x": 12.492040840733727,
                    "y": 2.9599363257716074,
                    "z": 3.277454879289394
                },
                "rotation": {
                    "x": 0,
                    "y": 0,
                    "z": 2.6185516904570356
                }
            }
        },
        {
            "class_name": "Vehicle: Truck:6",
            "class_id": 7,
            "coordinates": {
                "position": {
                    "x": 6.586469112473424,
                    "y": 40.04654127165122,
                    "z": 2.418127306912737
                },
                "scale": {
                    "x": 2.0797496300919427,
                    "y": 2.2058505020229275,
                    "z": 3.2304734340850843
                },
                "rotation": {
                    "x": 0,
                    "y": 0,
                    "z": 2.592521994924967
                }
            }
        },
        {
            "class_name": "Vehicle: Truck:7",
            "class_id": 7,
            "coordinates": {
                "position": {
                    "x": 5.285188266563637,
                    "y": 43.54250093062001,
                    "z": 2.4308194215461043
                },
                "scale": {
                    "x": 2.570057617341567,
                    "y": 2.9026862045568005,
                    "z": 3.369490628625357
                },
                "rotation": {
                    "x": 0,
                    "y": 0,
                    "z": 2.618420575003002
                }
            }
        },
        {
            "class_name": "Vehicle: Truck:8",
            "class_id": 7,
            "coordinates": {
                "position": {
                    "x": 4.144257860119978,
                    "y": 47.78841189356899,
                    "z": 2.422031004705218
                },
                "scale": {
                    "x": 1.5056279558793502,
                    "y": 0.9114685402080243,
                    "z": 2.574374945033881
                },
                "rotation": {
                    "x": 0,
                    "y": 0,
                    "z": 2.6322886166299746
                }
            }
        },
        {
            "class_name": "Vehicle: Truck:9",
            "class_id": 7,
            "coordinates": {
                "position": {
                    "x": 1.3205089375042007,
                    "y": 53.195234767081274,
                    "z": 2.2787420009242396
                },
                "scale": {
                    "x": 4.810655340916085,
                    "y": 2.9795990582530196,
                    "z": 3.7451833488407686
                },
                "rotation": {
                    "x": 0,
                    "y": 0,
                    "z": 2.636120184277253
                }
            }
        },
        {
            "class_name": "Vehicle: Truck:10",
            "class_id": 7,
            "coordinates": {
                "position": {
                    "x": -2.3372585059914854,
                    "y": 55.35458816772734,
                    "z": 2.309349808891241
                },
                "scale": {
                    "x": 1.7584739344099969,
                    "y": 2.6945024844782703,
                    "z": 3.540029356579609
                },
                "rotation": {
                    "x": 0,
                    "y": 0,
                    "z": 2.6140092792938177
                }
            }
        },
        {
            "class_name": "Pedestrian: Police_officer:1",
            "class_id": 14,
            "coordinates": {
                "position": {
                    "x": -22.220091903131106,
                    "y": -5.721974975152989,
                    "z": 0.7716872369746568
                },
                "scale": {
                    "x": 1.1649735821544855,
                    "y": 0.6528685587950898,
                    "z": 1.7922125170513628
                },
                "rotation": {
                    "x": 0,
                    "y": 0,
                    "z": -2.183734612557144
                }
            }
        },
        {
            "class_name": "Pedestrian: Construction_Worker:1",
            "class_id": 12,
            "coordinates": {
                "position": {
                    "x": 47.87786579108797,
                    "y": 45.039920239716594,
                    "z": 1.1778544721776854
                },
                "scale": {
                    "x": 0.8775186301537322,
                    "y": 0.4993571979217251,
                    "z": 2.0770678551218342
                },
                "rotation": {
                    "x": 0,
                    "y": 0,
                    "z": -2.7086425568669847
                }
            }
        }
    ]
}

@prisoner849 any lead on this ? :slight_smile:

Is this looking correct?

1 Like

@prisoner849 yes absolutely. The boxes are just missing their text labels from the json which is represented by the key class_name, but I think the you have achieved the major part.

I don’t do any transformations to the geometry of points. Then apply values of position, rotation and scales to a box, with the difference, that rotation on Z-axis has to be negated. That’s all.

@prisoner849 the solution worked. Thank you so much for sticking to this problem with me. Cheers. Here is how it looks, I made the bboxes transparent.

Any lead or idea or direction how to plot texts alongside with these boxes
Here is how generate my boxes :-

createBoundingBoxInstance(
    geometry: THREE.BoxGeometry,
    color: string | THREE.Color | number,
    coords: Coordinates,
    vector: THREE.Vector3,
    label: string,
    wireframe?:boolean
  ) {
    const material = new THREE.MeshBasicMaterial({ color, wireframe: wireframe, transparent:true,  opacity: 0.4,depthWrite: false,side: THREE.DoubleSide });
    const cube = new THREE.Mesh(geometry, material);

    cube.position.x = coords.position.x;
    cube.position.y = coords.position.y;
    cube.position.z = coords.position.z;

    cube.rotation.x = coords.rotation.x;
    cube.rotation.y = coords.rotation.y;
    cube.rotation.z = coords.rotation.z;

    cube.scale.x = coords.scale.x;
    cube.scale.y = coords.scale.y;
    cube.scale.z = coords.scale.z;
    this.scene.add(cube);
    // cube.updateWorldMatrix(true, false);
    // cube.getWorldPosition(vector);
    // vector.project(this.camera);
  }