I tried running mame with the ddraw option, which seemed to help a little, but some adjustment on the TV and video card helped a lot. I set the ATI card to output in "Video Mode", and then adjusted the contrast, brightness etc on the TV using a couple of different games. I think I just picked a bad game to start with-Street Fighter looks fine, it was just the sprites in Bombjack that looked pretty crappy. I'll see if I can grab some images to post to see whether more tweaking would help.
I tried setting the resolution on the command line though, and I couldn't really see a big difference between that res and what MAME autoselected. That might just be because it's tricky to compare without being able to see both side by side, so if it's not that dramatic, then I might just leave it as is.
The sitchresolutions doesn't seem to exist in either the mamewah mame ini file or as a command line option to Mame. Could it be called something else?
Either way, I think things are already vastly improved over the first attempt, but there's still a lot of configuration to be done to get this cabinet software set up.