I see what's happening, GroovyMAME is picking a 25 KHz mode and your monitor only admits 31 KHz. On a normal ATI card, GroovyMAME would add extra lines to that modeline to convert it to 31 KHz, but it can't do that with the AVGA 3000 (yet).
So, first download this version of GroovyMAME (groovymame32_0143.013c_test):
http://mario.groovy.org/GroovyMame/WindowsATIDrivers/Then, edit monitor_specs0 option in mame.ini with this values:
monitor_specs0 31499.99-31500.00,50.00-70.00,0.636,3.813,1.906,0.318,0.064,1.048,0,1,576.0,
800That should skip 384 lines modes (25 KHz) and used a 31 KHz mode instead, however I'm not sure how it will work with that mode table, so anyway please post back your logs after that.