To see this effect, create e.g. a 5x5 array (plt.subplots(5, 5)) and repeatedly open and close the configure subplots menu while paying attention to the position of the subplots.

matplotlib 1.5.1 and master, Qt5 backend.