Well after spending far too long reminding myself why I moved on from DOS, I have to concur - modern radeons appear to a complete non-starter with Advance mame in Dos or Windows.
I did not try linux (I believe it works quite well - I've got a linux box running but I've already made the very concrete decision not to build my cab based on linux. This just down to software I want to use which is windows only - sucks I know...)
Apparantly you can have better joy with modern Geforce cards (as perhaps you showed above with a GeforceGTS). The only way to run a radeon is to use in the inbuilt sdl library (ie skip straigh*t past advv/advcfg, run amame and it will use directx via sdl I *think* - I was getting a lot of er0rors regarding sdl video drivers in the logging, but the games ran ok.)
This appears to be ruling out advancemame for me, short of buying a trident blade PCI and running dual gfx cards - quite up for trying it, although a search of ebay UK revealed not a sausage...
I experimented with powerstrip a lot too (peice of advice if you are running dual monitors and powerstrip - make the one you want to test the only display) - mainly to try and see what my monitor can do. Seems happy enough in 15-16kHz interlaced (windows looks *horrendous* but amazingly looks really quite good in games I tried. To test I was creating the resolution in powerstrip, than ran normal mame forcing the resolution I'd messed with. If anyone has good info on powerstrip settings I'd love to see some as its quite fiddly..
Lot more experimenting to do though.
quick (dumb) question: games with: X:256 / Y: 224
that seems far less horziontal lines than you would get on a arcade monitor/tv (500-600 lines?) Are they doubled by interlacing/scanlines?
EDIT: Hmmm I was just about to purchase a 9880 PCI card, when I suddenly realised that I'm planning to use windows (doh) and your success was with the vbeline driver in DOS (vbeline and vgaline do not function in windows). I'm beginning to get the feeling I'm fighting a losing a battle (and I know someone's about to suggest the ArcadeVGA to me....)