Hey Frizzle, I'm a GM noob so take these words with a grain of salt
but did you generate your modes with your monitor in Windows Portrait mode by chance?
I had a similar issue because I created & and installed my VMMaker modes with my CRT in Windows Portrait mode. Once I put it back to Windows Landscape, generated the modes,
THEN moved the monitor to a vertical position, I set orientation=vertical in mame.ini, went back to Windows Portrait mode, then GM picked the correct resolutions for all games, it works perfectly (super modes).
As a test to see if you have the correct modes installed, you can set interlace to 0 in mame.ini, if the game shows up in the correct res then you know you have the right modes installed and disregard everything I just said.