Try tweaking the Horizontal Sync/Frequency pot on your monitor (the V Sync may need a tweak, too). There should be a 'sweat spot' that provides a stable picture in all modes.
If that doesn't fix it, GroovyArcade may be outputting a frequency/modeline that's out of the range of your monitor. Is the JPAC sync LED lit using the scrambled modes? I don't know how to set up GroovyArcade I'm afraid, as I've never used it.