Taken From Andy's website...
What are the limitations of arcade monitors?
With a 15Khz monitor, there is an important limit. This is vertical resolution. Assuming a vertical sync range of 50-60 Hz, a 15Khz monitor can only display approx 250-300 lines. (this figure is arrived-at by dividing Hfreq by Vfreq, and allowing about 10 "wasted" lines for vertical flyback time) This is perfect for most older games. Any game which originally used a 15Khz monitor can of course be displayed at it's native resolution providing the VGA card can be configured to do so.
Some later games use high-res monitors and have vertical resolutions of 480 or higher. For these, we can enable interlacing and still display at the native resolution. An interlaced picture scans even and odd lines separately, so we can have 480-600 lines interlaced. For the same reason, to display a Windows desktop at 640 X 480 we need to interlace. Unfortunately this does not give a good result on a Windows desktop because interlacing causes horizontal lines which are 1 pixel wide (which may Windows screens have) to flicker at 30Hz. We can run Windows 3D games pretty well though, because games hardly ever contain this kind of regular line type of graphic, and the flicker is not noticeable.
So what about games with resolutions of 301-480 lines? Well we do have a problem here with a 15Khz monitor as we cannot display them with either non-interlaced (not enough lines) or interlaced (top/bottom borders too wide). So for these few games (which originally used 25Khz medium-res monitors), we have to resort to hardware stretching, which re-samples the screen and adjusts the size. Fortunately H/W stretch and interlace degrade a high-res picture less than a low-res one.
What about games with a vertical res of 240 or less?
A game which has a native vertical resolution of 240 will look exactly the same on an arcade monitor whether the monitor is running at a vertical resolution of 240 or less than 240. The reason is this: tweaked modes of less than 240 are configured with top-bottom borders to add to the total number of lines. This is necessary to keep below the 60Hz vertical scan limit. (as number of lines goes down, the time taken to display them goes down, therefore the Vfreq goes up) So either the video mode will have borders (<240 modes) or MAME will have borders and the result is the same, and it will be arcade-real. This is why the ArcadeVGA card does not contain many <240 line modes. They are quite unnecessary. As far as horizontal res goes, there is no issue with this and monitors, they can display any horizontal resolution (note we are talking resolution here not scan rate, which is always 15Khz). The issue here is that the VGA card dot clock needs to be adjusted to provide the correct line scan rate for the required resolution. So with low frequency (15Khz) monitors, we can sometimes run into the VGA card's lower dot-clock limit. (the ArcadeVGA card has no practical dot clock lower limit).
You need to bring your screen res down to something that the monitor can handle.
Link to the monitor FAQ