If the MAME menus are working, then the resolution is working in GroovyMAME.
On my monitor, vector games are really dark, maybe you just need to adjust the display settings for vector games.
Try creating a vector.ini file with only the following contents as a starting point, then adjust to taste...
#
# CORE SCREEN OPTIONS
#
brightness 1.0
contrast 2.0
gamma 2.0
#
# CORE VECTOR OPTIONS
#
antialias 1
beam 2.0
flicker 0