The problem is that you've set your limits for 15kHz. Allowing that to go as high as 16.2kHz which is where most arcade monitors cut off, you can get:
(600 visible lines + 25 blanking lines) x 50Hz = 16.2kHz
But nothing higher, as you have set VMM to block them. And even that 600 visible lines assumes you have a monitor preset that allows so many.
The best idea, if you really
need to install a higher res mode, is to adjust your range line. Or even better, add a second range line to your chosen present in the monitor.ini file. (Or make your own preset, duplicating the current range line and adding second custom one, but you get the point).
The problem with doing this is that programs are going to grab and try to use this higher res mode, you can't account for every situation and guaranteed it's going to present problems. That said, it may not happen often, and if it does, you can remote in and fix it. I would recommend putting some sort of 15kHz protection dongle on the video card output port. like a Jpac, if you already have one. Or there are several DIY designs here -
http://forum.arcadecontrols.com/index.php/topic,160869.0.htmlAlternately, get everything set up, save your modelines in VMM, remove that one extra mode, and reinstall the rest and you should be fine - or at least in the same boat as the rest of us
So, your extra range line gets calculated specifically from the video mode you want - 1280x800@60Hz, the mode line for which is going to look something like this:
"1280x800_60 51.6KHz 60.00Hz" 73.48 1280 1298 1314 1424 800 810 820 860 -hsync -vsync
(Number of visible lines each frame + v-blanking) x frames-per-second-in-Hz = scan rate
Scan rate x (visible horizontal width + h-blanking) = pixel clock.
You will then need to go back and work out the timings for the italicised positions below, because it's late and i need to go to bed.
crt_range1 51550-51700, 60.00-60.00,
2.000, 4.700, 8.000, 0.064, 0.192, 1.024, 0, 0, 800, 800, 800, 800
For extra help with the crt_emudriver modelines, there is a monitor presets sticky above that explains a bit more. Pls give that a look, and post up again if you get stuck.