Launch VMMaker, edit monitor.ini and select your current monitor preset.
Change:
crt_range0 15625-16200, 49.50-65.00, 2.000, 4.700, 8.000, 0.064, 0.192, 1.024, 0, 0, 192, 288, 448, 576
by:
crt_range0 15625-16200, 49.50-65.00, 2.000, 4.700, 8.000, 0.064, 0.192, 1.024, 0, 0, 192, 296, 448, 576
Now you'll be able to create modes higher than 288p, which is not enabled by default. Make sure to mark the option "Export settings to GroovyMAME" so this change is applied to mame.ini too.