Autopan mode
I'm starting to understand how this works. Let's assume we are using 1024x768 as windows res. When you set the max res option under displays options, you're setting the autopan mode. If you set it to 1024x768, then no autopan mode will work, since your desktop will be the same size as the max res supported by the card. If you set it to 688x288, then the resolution displayed will never be higher than that, so you will have low res and a virtual desktop of 1024x768, which is the res set for windows. The problem with MAME is that it is incapable then to display res modes higher than 688x288. When, for instance 640x480 is required, then you see only part of the screen, as if you were using a virtual desktop. The only solution is to disable autopan mode: select 1024x768 as max res under displays options. I'm experiencing a problem at this point: corrupted screen graphics. If you restart windows, it's ok again. Switching res with hotkeys also corrects the display. Do you also have this problem? It only happens when going from low res to high res (any high res mode).
Hotkeys
There's an interesting option called hotkeys. You can define hotkeys for several res modes and switch between them. So, you can normally work at 640x288 (non-interlaced), and switch to 720x480 (the most stable of all non-interlaced modes) when you need more resolution. I've set it with alt-ctrl-1 and alt-ctrl-2. This is also located under the displays tab. What I'd like to know if it is possible to switch between autopan mode and no-autopan mode. It seems this is not possible because these hotkeys only remember desktop resolutions and are not affected by what you set under max monitor resolution.
Colour depth
There's also another interesting issue with 640x288 res mode. If you use this mode with 32 bit colour depth it looks shrink horizontally (it's normal because pixels are much higher than wider). But when you use 16 bits, the proportions look correct. Maybe it's a different res mode which is selected. I'm curious to know.