Although I can't recall right off the top of my head the actual message, I do remember something of that sort when I was messing around with the VESA modes, and certain display configs.
What I would do is delete the mame.cfg file, and have mame simply create a new one, see then if the problem still persists.