Raycasting works perfectly before i rotate OrbitControls

Questions
,
1

I have seen this similar issue posted so many times but i have been trying to solve my use case of it for alsmost a week. I have 7+ years of writing mission critical software for the DoD yet this simple javascript code has caused me more grief and trouble than anything in my entire career. I will try and describe the issue below.

My app is simple, take TLE satellite data from Celestrak and display it around a globe. The user should be able to select a satellite and see its data.

The selection logic uses the raycaster to detect intersections. It works PERFECTLY when the app first launches. I can select whatever satellite i wish, no matter how big, small, close to other satellites, etc… I always get the correct satellite. The second I rotate my OrbitControls the satellite selection functionality breaks down and the raycaster either returns 0 or a completely incorrect satellite. The interesting thing is that if i rotate the OrbitControls exactly to where they started at when the app first loads than the raycaster starts working again.

This seems so simple to me. Here are some thoughts i had:

  • The fact that the raycaster works perfectly at the initial load and when the orbit controls return to their original position suggests that the raycaster might not be updating correctly with the new camera positions.
  • Raycasting relies on accurate world coordinates to detect intersections. If the camera or scene transforms aren’t being correctly accounted for, it could lead to the observed behavior where intersections return a size of 0 or select the wrong object.
  • One possibility is that the raycaster is not correctly considering the updated positions of the objects and the camera after the orbit controls are used. This could be due to improper updates in the render loop or misalignment between the camera and the raycaster’s coordinate system.

Everything i have found online more or less backs up that theory. DESPITE the seemingly obvious cause of the issue I am now officially SEVEN days into trying to fix this one tiny bug. PLEASE. ANYONE. SHOW ME WHAT I AM DOING WRONG.

Here is my current code. I am failrly new to Three.js and this has been a very long week of debugging so if you see ugly code its probably a result of the battle ive been fighting through. This is just one of the many solutions i have attempted.

import * as utils from ‘./utils’;
import React, { useEffect, useState, useRef } from ‘react’;
import * as THREE from ‘three’;
import { OrbitControls } from ‘three/examples/jsm/controls/OrbitControls’;
import { GLTFLoader } from ‘three/examples/jsm/loaders/GLTFLoader’;
import ‘./App.css’;

const EarthViewer = () => {
const mountRef = useRef(null);
const modelViewerRef = useRef(null);
const [selectedSatellite, setSelectedSatellite] = useState(null);

const logIntervalRef = useRef(null);
const camera = useRef(new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 1000));
const controls = useRef(null); // Ref for OrbitControls



useEffect(() => {
    const currentElement = mountRef.current;
    const scene = new THREE.Scene();

    camera.current.position.z = 20;
    const renderer = new THREE.WebGLRenderer({ antialias: true });
    renderer.setSize(window.innerWidth, window.innerHeight);
    currentElement.appendChild(renderer.domElement);

    const handleResize = () => {
        camera.current.aspect = window.innerWidth / window.innerHeight;
        camera.current.updateProjectionMatrix();
        renderer.setSize(window.innerWidth, window.innerHeight);
    };
    window.addEventListener('resize', handleResize);

    const ambientLight = new THREE.AmbientLight(0x404040);
    scene.add(ambientLight);

    const directionalLight = new THREE.DirectionalLight(0xffffff, 0.5);
    directionalLight.position.set(10, 10, 10).normalize();
    scene.add(directionalLight);

    const textureLoader = new THREE.TextureLoader();
    const earthTexture = textureLoader.load('8k_earth_daymap.jpg');
    const earthGeometry = new THREE.SphereGeometry(5, 32, 32);
    const earthMaterial = new THREE.MeshBasicMaterial({ map: earthTexture });
    const earth = new THREE.Mesh(earthGeometry, earthMaterial);
    earth.scale.set(1, 0.96, 0.96);
    const axialTilt = 23.5 * (Math.PI / 180);
    earth.rotation.y = axialTilt;

    scene.add(earth);

    controls.current = new OrbitControls(camera.current, renderer.domElement);
    controls.current.enableDamping = true;

    const satelliteGeometry = new THREE.SphereGeometry(0.1, 8, 8);
    const generalMaterial = new THREE.MeshBasicMaterial({ color: 0xffffff });
    const generalMesh = new THREE.InstancedMesh(satelliteGeometry, generalMaterial, 1000);
    generalMesh.instanceMatrix.setUsage(THREE.DynamicDrawUsage);
    scene.add(generalMesh);

    let generalIndex = 0;
    const generalSatellites = [];

    const starlinkMaterial = new THREE.MeshBasicMaterial({ color: 0x00ffff });
    const starlinkMesh = new THREE.InstancedMesh(satelliteGeometry, starlinkMaterial, 1000);
    starlinkMesh.instanceMatrix.setUsage(THREE.DynamicDrawUsage);
    scene.add(starlinkMesh);

    let starlinkIndex = 0;
    const starlinkSatellites = [];

    const fetchData = async () => {
        try {
            const response = await fetch('http://localhost:8000/api/satellites/');
            const data = await response.json();
            return data;
        } catch (error) {
            console.error('Error fetching satellite data:', error);
            return [];
        }
    };

    fetchData().then((data) => {
        data.forEach((sat) => {
            const { x, y, z } = utils.computeSatellitePosition(sat.name, sat.tle_line1, sat.tle_line2);

            if (isNaN(x) || isNaN(y) || isNaN(z)) {
                console.error(`Invalid position for satellite ${sat.name}: x=${x}, y=${y}, z=${z}`);
                return;
            }

            const dummy = new THREE.Object3D();
            dummy.position.set(x, y, z);
            dummy.updateMatrix();

            if (sat.name.startsWith('STARLINK')) {
                if (starlinkIndex < starlinkMesh.count) {
                    starlinkSatellites.push({ ...sat, instanceId: starlinkIndex, x, y, z });
                    starlinkMesh.setMatrixAt(starlinkIndex, dummy.matrix);
                    starlinkIndex++;
                }
            } else {
                if (generalIndex < generalMesh.count) {
                    generalSatellites.push({ ...sat, instanceId: generalIndex, x, y, z });
                    generalMesh.setMatrixAt(generalIndex, dummy.matrix);
                    generalIndex++;
                }
            }
        });

        updateBoundingVolumes(generalMesh, generalSatellites);
        updateBoundingVolumes(starlinkMesh, starlinkSatellites);

        animate();
    }).catch((error) => console.error('Error loading data:', error));

    const updateSatellitePositions = () => {
        generalSatellites.forEach((sat) => {
            const { x, y, z } = utils.computeSatellitePosition(sat.name, sat.tle_line1, sat.tle_line2);
            const dummy = new THREE.Object3D();
            dummy.position.set(x, y, z);
            dummy.updateMatrix();
            generalMesh.setMatrixAt(sat.instanceId, dummy.matrix);
        });

        starlinkSatellites.forEach((sat) => {
            const { x, y, z } = utils.computeSatellitePosition(sat.name, sat.tle_line1, sat.tle_line2);
            const dummy = new THREE.Object3D();
            dummy.position.set(x, y, z);
            dummy.updateMatrix();
            starlinkMesh.setMatrixAt(sat.instanceId, dummy.matrix);
        });

        updateBoundingVolumes(generalMesh, generalSatellites);
        updateBoundingVolumes(starlinkMesh, starlinkSatellites);
    };

    const updateBoundingVolumes = (mesh, satellites) => {
        const boundingBox = new THREE.Box3();
        const dummy = new THREE.Object3D();
        for (let i = 0; i < satellites.length; i++) {
            mesh.getMatrixAt(i, dummy.matrix);
            dummy.matrix.decompose(dummy.position, dummy.quaternion, dummy.scale);
            boundingBox.expandByPoint(dummy.position);
        }
        mesh.geometry.boundingBox = boundingBox;
        mesh.geometry.boundingSphere = boundingBox.getBoundingSphere(new THREE.Sphere());
    };

    const logBoundingVolumes = () => {
        console.log(`Updated bounding volumes for general mesh:`);
        console.log(generalMesh.geometry.boundingBox);
        console.log(generalMesh.geometry.boundingSphere);

        console.log(`Updated bounding volumes for starlink mesh:`);
        console.log(starlinkMesh.geometry.boundingBox);
        console.log(starlinkMesh.geometry.boundingSphere);
    };

    const logCameraInfo = () => {
        console.log(`Camera Position: ${camera.current.position.x}, ${camera.current.position.y}, ${camera.current.position.z}`);
        console.log(`Camera Matrix World: ${camera.current.matrixWorld.elements}`);
    };

    logIntervalRef.current = setInterval(() => {
        logBoundingVolumes();
        logCameraInfo();
    }, 2000);

    const transformNormalToWorldSpace = (intersect, mesh) => {
        const normal = intersect.face.normal.clone();
        normal.transformDirection(mesh.matrixWorld);
        return normal;
    };

    const onMouseClick = (event) => {
        // Ensure controls update the camera's matrix

        const mouse = new THREE.Vector2();
        mouse.x = (event.clientX / window.innerWidth) * 2 - 1;
        mouse.y = -(event.clientY / window.innerHeight) * 2 + 1;

        const raycaster = new THREE.Raycaster();
        raycaster.setFromCamera(mouse, camera.current);
        raycaster.params.Line.threshold = 0;

        // Check intersections with general satellites
        const generalIntersects = raycaster.intersectObject(generalMesh, true);
        if (generalIntersects.length > 0) {
            console.log("intersection found");
            const instanceId = generalIntersects[0].instanceId;
            const selectedSat = generalSatellites.find(sat => sat.instanceId === instanceId);

            // Transform normal to world space
            const normal = transformNormalToWorldSpace(generalIntersects[0], generalMesh);

            setSelectedSatellite({ ...selectedSat, normal });
            loadSatelliteModel(selectedSat);
            return;
        }

        // Check intersections with Starlink satellites
        const starlinkIntersects = raycaster.intersectObject(starlinkMesh, true);
        if (starlinkIntersects.length > 0) {
            console.log("intersection found");
            const instanceId = starlinkIntersects[0].instanceId;
            const selectedSat = starlinkSatellites.find(sat => sat.instanceId === instanceId);

            // Transform normal to world space
            const normal = transformNormalToWorldSpace(starlinkIntersects[0], starlinkMesh);

            setSelectedSatellite({ ...selectedSat, normal });
            loadSatelliteModel(selectedSat);
            return;
        }

        setSelectedSatellite(null);
    };



    const loadSatelliteModel = (satellite) => {
        const modelViewerElement = modelViewerRef.current;
        if (modelViewerElement) {
            while (modelViewerElement.firstChild) {
                modelViewerElement.removeChild(modelViewerElement.firstChild);
            }

            const modelScene = new THREE.Scene();
            const modelCamera = new THREE.PerspectiveCamera(75, modelViewerElement.clientWidth / modelViewerElement.clientHeight, 0.1, 1000);
            modelCamera.position.z = 5;
            const modelRenderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
            modelRenderer.setSize(modelViewerElement.clientWidth, modelViewerElement.clientHeight);
            modelViewerElement.appendChild(modelRenderer.domElement);

            const modelAmbientLight = new THREE.AmbientLight(0x404040);
            modelScene.add(modelAmbientLight);

            const modelDirectionalLight = new THREE.DirectionalLight(0xffffff, 0.5);
            modelDirectionalLight.position.set(10, 10, 10).normalize();
            modelScene.add(modelDirectionalLight);

            const loader = new GLTFLoader();
            loader.load(`/starlinkhigh.glb`, (gltf) => {
                modelScene.add(gltf.scene);
                const animateModel = () => {
                    requestAnimationFrame(animateModel);
                    gltf.scene.rotation.y += 0.01;
                    modelRenderer.render(modelScene, modelCamera);
                };
                animateModel();
            });
        }
    };

    window.addEventListener('mousedown', onMouseClick);
    

    const animate = () => {
        requestAnimationFrame(animate);
       
        updateSatellitePositions();
        
        renderer.render(scene, camera.current);
        
    };

    return () => {
        renderer.dispose();
        window.removeEventListener('resize', handleResize);
        window.removeEventListener('mousedown', onMouseClick);
    
        clearInterval(logIntervalRef.current);
    };
}, []);

return (
    <div ref={mountRef} className="threejs-container">
        {selectedSatellite && (
            <div className="info-panel">
                <h2>Satellite Info</h2>
                <p>Name: {selectedSatellite.name}</p>
                <p>Position:</p>
                <p>X: {selectedSatellite.x}</p>
                <p>Y: {selectedSatellite.y}</p>
                <p>Z: {selectedSatellite.z}</p>
                <div className="model-viewer-container" ref={modelViewerRef}>
                </div>
            </div>
        )}
    </div>
);

};

export default EarthViewer;

2

Not sure but the problem may come from the camera not being updated?

Try to update the camera’s projection matrix camera.updateProjectionMatrix() and the orbitControls controls.update() before calling raycaster.setFromCamera.

const raycaster = new THREE.Raycaster();
camera.current.updateProjectionMatrix();
controls.current.update();
raycaster.setFromCamera(mouse, camera.current);

Update: made this quick (and dirty) example, maybe you could reproduce your issue!

1 Like
3

Thank you so much for that example. I have to go run an errand really quick but that looks incredibly promising and I think i can fix my coe based upon that. Seriously thank you.

1 Like
4

Somewhere over the course of that ordeal i over complicated my code a bit. Seeing your example really helped me get back to the core of what i was trying to do. Thanks again!

1 Like