I can give an account of my setup path here, maybe it'll be a similar case for you.
I have a Radeon 2400 HD card, Windows XP x64 and and Nanao MS9 arcade monitor to work with.
First thing I did was download and install Calamity's ATI drivers (Catalyst version 9.3 for my card) and GroovyMAME from the GroovyMAME homepage:
http://mame.groovy.org/Then I installed the ATI drivers and restarted the machine.
Next I unzipped GroovyMAME to a folder on my C drive and configured the VMMaker software also available from the GroovyMAME site. This is done by editing the VMMaker.ini with notepad and I chose "GENERIC" as my monitor type since the default "CUSTOM" setting gave some odd resolutions first time I tried it. I also had to set my dotclock to 7.3 as anything below that did not agree with my GFX card. Apparently 2xxxHD cards and up have this issue, though your mileage may vary. Finally I set up the path to my MAME executable and told the ini to create an XML and read that XML after to generate the resolutions I needed. It then generated the most suitable list of resolutions/refresh rates for my setup (limited to 120) and wrote that info to the registry. I then reset the machine again.
At this point you can use arcade_osd to check the available resolutions (those installed and usable via groovymame) to make sure you can actually use them. No problems here.
Lastly I needed to run the games I wanted to play and made sure they all worked as expected. The only games I had an issue with were the Midway Wolf/Y/T unit games (termintor 2, MK 1,2,3, NBA Jam etc) which would be squashed in the middle of the screen. I simply then made an ini for each game (e.g. mk.ini) in a "ini" subdirectory of my groovymame folder for each game and formatted them like this:
modeline 1
resolution 401x256
This probably isn't ideal for whatever reason but it did work for me and I have no stutter, tearing or pixel deformities on these games. You may need to experiment with different resolutions to get it looking correct.
These games also had another issue where they rocketed at about 450% speed which was fixed (albeit not an ideal solution) by enabling tripplebuffering. This also fixed huge Tekken Tag slowdowns while fighters were moving ingame while still keeping the benefits of the setup.
I hope this helps a bit, I can expand upon any points here if needed.