In vmmaker.ini, I also changed the following to '1' after pointing to the correct mame executable:
ListFromXML = 1 ; Processes Mame XML and get video mode list from it
GenerateXML = 1 ; Extracts XML from Mame (only needed once)
Then set monitor type to your type of monitor. Mine is a D9400 so I chose that.
; Monitor Type. Valid types: D9800, D9400, D9200, EGA, VGA, MULTI, H9110, PAL, NTSC, GENERIC, CUSTOM
MonitorType = "D9400"
You can run the quickres utility from ultimarc to see if the resolutions were installed correctly (little galaga icon in the task bar). You should see a bunch of low resolutions displayed.
http://ultimarc.com/quickres.exeNext install groovymame to a fresh directory, and create a config:
groovymame.exe -cc
Find the newly generated mame.ini and make sure monitor is set to your type (cga, generic, etc).
Run groovymame.exe yourgame and it should be running at the exact resolution and refresh rate.