Hi jlampc,
It wasn't clear to me what your JPAC leds do when your monitor gets out of sync. If the LED keeps light then it's a matter of adjusting on the monitor side (hfreq/vfreq pots). On the other hand, if it's telling you there's no sync, then you need to look at the software side.
If my memory does not fail me I believe Bitbytebit reported the x800 as being the only Radeon not to work with GroovyArcade.
I think you should try editing the dotclock_min value in mame.ini:
dotclock_min 8.0