If I get you right, what you're trying to achieve is this:
- For all games that GroovyMAME picks, say 400x256@60 -> pick this different xres x yres @ refresh
As far as I can see, this is not possible.
I know this can sound a bit awkard but trust me when I say this is the way to go: you should not fight GroovyMAME on the resolutions it picks. Here is where the issue is:
For instance, the resolution that was being picked automatically by mame for Mortal Kombat was off, so I used arcade osd to see what other resolutions my monitor was capable of and chose the closest.
If GroovyMAME is picking that resolution it's because you're telling it that it works: as long as it is available in your system GroovyMAME will want to use it.
However, the answer is not just removing it from the system! If it's been created it's because you told VMMaker that your monitor supports it! This is done by the monitor_specs setup.
If VMMaker is creating video modes that turn out to be unsupported by your monitor, you have to re-adjust it's settings until every video mode created is actually supported, that's the way to go.
Then you port the same monitor_specs to mame.ini and you're done.