Could not get it to work better. here is a picture of the monitor chassis connectors:

I'd say it's a sync polarity issue. First I'd try to always use separate sync, although in theory it's possible to output a composite signal natively with an ATI card, I have no experience with this and do not recommend you to try. If yours is a JAMMA cab you probably should have bought a J-PAC instead of the video amplifier, because this one properly combines both sync signals into a composite signal so you could use the same board's connector you're using with the PCB (the one with the -V)
So you need to wire the two separate sync cables either to the the H+ V+ or to the H- V-, or possibly a combination. Just in case use the E connector next the ones you're using.
The last version of VMMaker IN THEORY allows you to change the sync polarities by software, new GM does too. I must remark I don't even know if this works because my monitors seem to accept any polarity or at least I notice nothing when changing. I'm possitive it works for the Linux side.
So get the latest version from here (VMMaker 1.3c):
http://mame.3feetunder.com/windows-ati-crt-emudriver/Add this line to vmmaker.ini, set MonitorType = "CUSTOM", and play with these values:
monitor_specs0 15625-16670, 49.5-65, 2.000, 4.700, 8.000, 0.064, 0.160, 1.056,
0, 0, 288, 448
...which stand for hsync and vsync polarities (0=negative, 1=possitive). Try combinations. Each time you modify the values, run VMMaker and restart.
Good luck.