Try this:
open "Display settings" (right-click on desktop)
click/open "Advanced display settings"
click/open "Display adapter properties for Display 1" Or whatever # your display is.
click/open "List all modes"
choose your video mode from the list!
- "640x480, True Color (32 bit), 60 Hertz"
Click OK a couple of times to exit out!
After the above, you can also use ArcadeOSD (should be in same folder as VMMaker) to select desktop mode (though I'm not sure that you'll need to, at this point).
If problems persist, you could ask about this on the Groovymame branch of this forum.