Instanced Mesh loses color attribute when size changes

I am using React-Three-Fiber to generate an instanced mesh. I have vertex coloring turned on. There is an option to change the radius of the instances but when this happens the mesh turns black. From logging the mesh, I can see that the geometry > attributes > color property no longer exists after changing the radius. Why is this and what can be done about it?

  <instancedMesh ref={unitMeshRef}
                  onPointerDown={(e)=> setActive(e.instanceId)}
                  args={[null, null, numPoints]}
                  onPointerMove={(e) => setHovered(e.instanceId)}
                  onPointerOut={(e) => setHovered(undefined)} >
    <sphereBufferGeometry attach="geometry"
                          args={[options.radius, 20, 20]}>
      <instancedBufferAttribute attachObject={["attributes", "color"]}
                                args={[colorArray, 3]} />
    </sphereBufferGeometry>
    <meshStandardMaterial reflectivity={0.01}
                          metalness={0.01}
                          attach="material"
                          color={"#FF6B00"}
                          vertexColors={true} />
  </instancedMesh>

My guess would be that when the size changes, the parent object is re rendered and the original instanced buffer attribute is lost.

/cc