Hi Falk3r,
The cool video features of GroovyMAME (custom modeline generation) are achieved in combination with a special hacked driver for ATI cards called "CRT Emudriver", but:
- CRT Emudriver is for Windows XP only.
- CRT Emudriver doesn't work with ArcadeVGA 3000.
So basically with an ArcadeVGA 3000 it's not worth "upgrading" to Windows XP because anyway you're going to have to use Ultimarc's drivers. GroovyMAME will still work but limited to AVGA's stock video modes.
Switching to D3D is done by editing the -video option in mame.ini, it's trivial.
On the other hand, there are good news for you if you have the chance to try GroovyArcade on an usb pendrive, because this system does support ArcadeVGA 3000 with absolutely no limitations with regard to custom mode switching. So I'd just stick with your Windows 7 installation, use GroovyMAME with it, and try GroovyArcade on a pendrive as an alternative if you want to compare.
Finally, there are some extra options you need to tweak to get GM working with an ArcadeVGA, check the main thread for info, no big deal.