You have basically two options:
- Use a vertical resolution of 288 and manually tweak your monitor potenciometers to bring the picture inside the screen area for that specific game, then tweak it back for horizontal games. That's what I do BTW.
- Use a virtualized (interlaced + stretched) resolution for vertical games above a given vertical resolution. You do that by specifying that value in a monitor_specs line:
monitor_specs_0 15625-16200, 49.50-65.00, 2.000, 4.700, 8.000, 0.064, 0.160, 1.056, 0, 0, 256, 448
...here everything above 256 will be calculated as interlaced. You need to add this line both to mame.ini and vmmaker.ini (in vmmaker you need to set MonitorType ="CUSTOM" too and format differs a little bit.)