I've got the same as you but with rgb 6.
I've tried the higher resolutions and they look a lot better, the colours seem deeper and there's less blockiness, however I have two problems.
1. It does'nt run as well as it does at lower res, I get sound breakup and it seems a bit slower.
2. My monitor is tucked up inside my cab so I can't adjust the screen size now!
I think the performance is down to my graphics card which is only a 32mb ati radeon 7200.
If I could get it running smoothly I'd pull monitor out and redo the screen position.
I did find that I could only use it at 1600x1200 mind , any higher and I could'nt see all the picture even though I had black borders around the screen.
Do you think a better graphics card would do the trick or will it make little difference?
p.s. I've tried your settings and at 1024x768 and they look a bit obvious ( chiken wire effect) but at 1600x1200 they look very good indeed, probably the best picture I've seen on mame yet.
I think I'll be pulling my monitor out tonight.