What you want to do should be very possible, but why not go for 800x600@60 if you can? It might simply be a case of sync polarity, or it might be a case of your range lines being wrong. What Wells-Gardener model do you have? Most tri-sync's typically sync in preset bands, not continuously, though it looks like some of the WG ones are continuous.
The below timings are from the GroovyMAME subforum monitor presets sticky on here -
http://forum.arcadecontrols.com/index.php/topic,116023.0.html and the very top of that thread explains what the values are in purple text, but you may need to do some reading on modelines as well. GM's switchres module likes its sync and porch timings in milliseconds, but a lot of modeline explanations and adjustments elsewhere may use lines - remember these are vertical lines.
Your AMD 7970 card would be able to use crt_emudriver, which is an ideal way to go for a CRT. It gives you good options for 15kHz emulation (240p and 480i), all the way up to the limits of your CRT. The best place to find info on Groovy MAME and crt_emudriver is the first post of subforum sticky -
http://forum.arcadecontrols.com/index.php/topic,151459.0.htmlI'm certainly running Tekken 7 online at 800x600@50i on my 15kHz setup, with the detail down to get more frames from an older PC, so it can be done
// Wells Gardner D9800, D9400
d9800, d9400
crt_range0 15250-18000, 40-80, 2.187, 4.688, 6.719, 0.190, 0.191, 1.018, 0, 0, 224, 288, 448, 576
crt_range1 18001-19000, 40-80, 2.187, 4.688, 6.719, 0.140, 0.191, 0.950, 0, 0, 288, 320, 0, 0
crt_range2 20501-29000, 40-80, 2.910, 3.000, 4.440, 0.451, 0.164, 1.048, 0, 0, 320, 384, 0, 0
crt_range3 29001-32000, 40-80, 0.636, 3.813, 1.906, 0.318, 0.064, 1.048, 0, 0, 384, 480, 0, 0
crt_range4 32001-34000, 40-80, 0.636, 3.813, 1.906, 0.020, 0.106, 0.607, 0, 0, 480, 576, 0, 0
crt_range5 34001-38000, 40-80, 1.000, 3.200, 2.200, 0.020, 0.106, 0.607, 0, 0, 576, 600, 0, 0
// Wells Gardner D9200
d9200
crt_range0 15250-16500, 40-80, 2.187, 4.688, 6.719, 0.190, 0.191, 1.018, 0, 0, 224, 288, 448, 576
crt_range1 23900-24420, 40-80, 2.910, 3.000, 4.440, 0.451, 0.164, 1.148, 0, 0, 384, 400, 0, 0
crt_range2 31000-32000, 40-80, 0.636, 3.813, 1.906, 0.318, 0.064, 1.048, 0, 0, 400, 512, 0, 0
crt_range3 37000-38000, 40-80, 1.000, 3.200, 2.200, 0.020, 0.106, 0.607, 0, 0, 512, 600, 0, 0