http://forum.arcadecontrols.com/index.php/topic,153488.msg1608259.html#msg1608259From the monitor presets sticky up above in this forum we find a preset for the D9800, which i assume you have set in VMM and are happy using?
   // Wells Gardner D9800, D9400
   d9800, d9400   
      crt_range0  15250-18000, 40-80, 2.187, 4.688, 6.719, 0.190, 0.191, 1.018, 0, 0, 224, 288, 448, 576
      crt_range1  18001-19000, 40-80, 2.187, 4.688, 6.719, 0.140, 0.191, 0.950, 0, 0, 288, 320, 0, 0
      crt_range2  20501-29000, 40-80, 2.910, 3.000, 4.440, 0.451, 0.164, 1.048, 0, 0, 320, 384, 0, 0
      crt_range3  29001-32000, 40-80, 0.636, 3.813, 1.906, 0.318, 0.064, 1.048, 0, 0, 384, 480, 0, 0
      crt_range4  32001-34000, 40-80, 0.636, 3.813, 1.906, 0.020, 0.106, 0.607, 0, 0, 480, 576, 0, 0
      crt_range5  34001-38000, 40-80, 1.000, 3.200, 2.200, 0.020, 0.106, 0.607, 0, 0, 576, 600, 0, 0
So, let's say we wanted to add 800x600@60.0000.
Scanrate = (total vertical lines) x (frames per second)
and
Total vertical lines = visible lines + blanking interval
With 600 visible lines we'll have a default blanking interval of about 60 more lines.
39,600  = 660 x 60
This is above the highest scanrate your monitor can handle, and you've told VMM to cut off at 38,000 by using the preset. So VMM will either reject the mode, lower the framerate, or turn the mode interlaced (30FPS) and 
then lower the framerate. For cases right on the edge VMM might simply install the mode as it is.
Install the mode if VMM will let you, then go into ArcadeOSD and try to edit the mode. If you can read the screen, find the screen where you can unlock the v-frequency and start playing with the vertical geometry/porches, remembering that you can always hit esc to go back and not save your changes.
If the mode does not display ok to begin with in ArcadeOSD, try it again at 800x600@58.0000 or as low as 57.0000 and go again.
You want 60.0000 if you can get it as that will give you more flexibilty, and stop some older games that are v-sync locked from slowing down under 60Hz.