I don't believe there is a total amount of groups/subs limit. The program may be maxing out memory ressources.
I am not too sure where groups come into play with instancedbuffergeometry though. When instancedbuffergeometry objects is used, Three.js creates 1 object for all instances. Then, each instance can be manipulated via attributes arrays, there exists no underlying objects for each instance you can add a group to.
Assuming there are not too many textures, 8 being the minimum guaranteed, you should be able to set/swap texture for each instance. To achieve such effect, you can create a texture reference attribute and store each texture reference to use in the fragment shader. Or better, you could have 1 texture and pass the proper UV offsets to the fragment shader via attributes.