Setting scene scale on webvr has no effect

I have an existing non-vr scene that I want to view in VR.
The scene shows a small city - for viewing in vr, I do not want to walk inside the city, but rather walk around a model version of the city, placed on a table.

For this, I need to scale the city scene down. Setting scene.scale.set(0.05,0.05,0.05) seems to work fine if I do not do xr.enabled, but for has no effect when enabling xr.

What is the suggested way of scaling down a scene or parts of a scene for vr?

@Peter_Strand Why don’t use scale the model down?

the model has all sorts of related object and animations that assumes coordinates.
So just scaling the model would require me to also scale all related things.
Ideally, I would like to be able to say that 1 in the model is not 1 meter in vr, but - say - 1cm. Is this possible?

Did you add the camera as a child of the scene, or as a child of any object in the scene tree ? Because I assume scaling the scene down would have no effect on the relative camera space then.