OK.. let me try and help you a bit here.. hope the following makes sense.
The great thing about the ArcadeVGA is that in nativly (i.e. in hardware) supports most of the common arcade PCB resolutions. This means, you don't have to mess around with scan lines, resolutions and all the other mame settings.
just running mame.exe should be good enough!
That said.. you need to instruct mame what resolution the game should run at. Do not assume the default is the correct one. There are lots of tools out there to force MAME into the correct resolution.
Try running the ArcadeVGA MAME resolution tool that's found here:
http://mamewah.mameworld.net/downloads.htmlThis tool will force the games to play in their ArcadeVGA support resolutions and as close to the real thing as possible.
I think the problem you're running into is that MAME runs at the desktop resolution (640x480) and that will always look like crap.
Try playing some of the Neo Geo games. They should run at

x 240 resolution. They should look perfectly clean and sharp! (with no refresh problems or sync problems).
Oh.. and you should see the pixels that make up the game. That's what VGA is all about.. that's the way it was in the arcade. It's the ultimate thing that people strive for.
The thing about MAME is that it tries to be super accurate in terms of reproduction (things like resolution and refresh rate). These things work greeat on something like a PC monitor (that can cope with different refresh rates) but fails terrible on things like an Arcade monitor (since refresh rate is something you tune with the vertical hold and other pots on the monitor!)
This is where things like NoNAME MAME come into play. Check out:
http://www.arcadecontrols.org/yabbse/index.php?board=3;action=display;threadid=10428It forces mame in 60hz (native arcade monitor refresh rate).
Try it out.. you'll find it better than the standard MAME.
As far as the front end.. I strongly recommend MAMEWAH. MAMEWAH will work at arcade resolution, rather than windows resolution, so it'll look better. Also, it's very customizable. So you can make it look really nice to the point that people will never know there's a PC in there.
There's a lot of tweaking that needs to be done to get it all to work together correctly.. and lots of reading to do. Hope this gets you started on the right direction.