Check your mame.ini for:
switchres 1
switchbpp 1
That is a long shot, I don't know much else to suggest. I had one person mention a very similar problem once, and it turned out the video drivers were to blame - perhaps try uninstalling + reinstalling those...