Loading texture dynamically in gltf

Hi Everyone, I’m trying to load texture to the gltf file dynamically. The gltf already has the normal, diffuse and metallic textures to it. When I first load from the gltf file my model has the correct texture. Now if I even load the same texture from the script and assign it to the mesh everything looks scatered. I tried to play around with repeat, rotations nothing seems to work. Am I missing something?

const model = useLoader(GLTFLoader, "/path/to.gltf");
 <primitive 
                object={model.scene} 
                scale={1} 
                position={[0, -1.2, 0]} 
                castShadow 
                ref={modelRef}
            />

This works fine but now when i add

 const diffuseMap = useLoader(TextureLoader, "/path/to/same texture used in the above gltf.png");
const modelRef = useRef<THREE.Group>(null);
useEffect(() => {
if (model && modelRef.current) {
modelRef.current.traverse((node: any) => {
             if (node.isMesh) {
 const material = node.material as THREE.MeshPhysicalMaterial;
    if (material) {
 material.map = diffuseMap; 
}
}
})
},[model])

This is laying new texture on the model but its all scattered. Any help would be much appreciated.