For the sake of quick debugging - what happens if you use just a plain <mesh> instead of models in your rapier component? Does everything behave correctly?
How are you loading the model?
If you’re using useGLTF to load the model and then pass it down the props - you’re only really using a single model at all times. useGLTF has a bit confusing behaviour of loading and caching just a single instance of the model - so if you use or modify the model in more than one component - you’ll be modifying it everywhere (it has it’s pros and cons - but as above, can be a bit confusing.
If, regardless of how you load the model, you pass it via props to the rapier component - you’d still need to ensure that component is using a unique instance of the model, ex.:
const uniqueModel = useMemo(() => model.clone(), [model]); // NOTE Be sure to dispose cloned models
ps, in threejs you can only have one model in one place.
const mesh = new THREE.Mesh(geo, mat)
sceneA.add(mesh)
sceneB.add(mesh)
three will first mount mesh into sceneA, then remove it, then mount it into sceneB. this is what happened to your primitive, which is just a parent.add(object).