If its of any interest to others, I believe I have tracked the issue down, as @Mugen87 suggested, to the GC. I was copying a lot of objects, which could fill it and this caused the GC to fill and need emptying.
I also found that using the memory tab on the chrome dev tools really helped (id somehow never actually used it!) as well as making sure memory is ticked on when recording performance, I could tally up the drop in memory.
thanks to @Mugen87 for pointing me in the right direction