What mame version are you using. If you're using the command line version, just open mame.ini inside you mame emulators folder and change "resolution" from "Auto" to 1024x768. If you're using Mameui or Mame32 it should be somewhere in the Display options.