If you followed the guide to the letter, you used the Generic_15 monitor preset. That is for a 15kHz (240p/480i) old SD CRT TV, to protect it, where your multiformat BVM is capable of a lot more, and will go up to 720p (which is 45kHz).
You're going to want to make your own monitor preset. There are a few threads about it, the first one i dragged up in search was in the monitor presets sticky from the top of this forum. From the linked post on down is the info you want, but you should read the OP as well to make sure you understand the terms -
http://forum.arcadecontrols.com/index.php/topic,116023.msg1632407.html#msg1632407What you do then is open VMM, and instead of selecting Generic_15, click to edit the file, then, following the format used in the file, create a new preset called BVM or whatever you like, and add the crt_range lines 0-3 from that image. Make sure to check the box to export to GroovyMAME, and then you have a mame.ini file that looks like the image shown. Then you copy your user_modes.ini file, in the VMM dir, and name it what you want. Or possibly find the one with super resolutions in it already (the resolutions that are 2560 x whatever). Open the new file, remove the video modes (resolutions) you do not want, and add the ones you do, while following the format of that file exactly (it's not complicated). Then you tell VMM to generate the modes, and have a look at them. If they look right, you can install them. The list needs to contain every video mode you want all at once, nothing left out. If it doesn't, take a good look at what's missing or been left out, and see if you can work out why. You can always do this step over and add more modes later if you want. Essentially VMM tests the video modes in the file you point it to, one by one, against the monitor preset - the physical specs you've said your monitor is capable of. Nothing higher than the 768x512i mode passes this test, for the Generic_15 preset.
That should get you started, anyway, though there's always further tweaking down the track. Go slowly, though. PVM's are built like tanks, and BVM's are built like battleships, but if you do something wrong you might have to re-install windows
EDIT: what that linked preset doesn't allow for is 1080i, but get things working first, and see if you can nut out how to get to that.