I always make sure that my refresh rate stays the same otherwise I see similar things.
I was thinking refresh rate also. I run MAME at 800x600 and 85 Hz (PC monitor, obviously), and I think I have the refresh rate set to auto in MAME, but I also use Multires which I think ensures that the refresh rate doesn't change when a Windows program changes resolution. Perhaps if I didn't have that, I would need to specify a refresh in MAME.