How can I swap HDR images dynamically in a scene without reloading?

I have a scene that uses an HDRI image. I want to let the user switch between HDR images from within the scene dynamically. How can I do this and what do I need to do to release memory/resources properly so I don’t create a memory leak when I swap HDR images?