you have a few options,
you can get an arcadevga card and use a monitor at 15khz or tv(make a vga to scart if possible 4 that),
I had a pretty smooth picture with arcadevga and arcade monitor - also quite easy to setup,
or you can use soft15khz tool and set up using powerstrip,
or use advancemame.
the most difficult thing is to set up your mame to use the right resolutions(closest to the original).
I use dos and advancemame but there is no magic bullet to fix these problems,
its always a compromise,i still mess about with settings for certain games to run better if needed.
I hardly get tearing but i tend to alter my settings on a game by game basis,im sure it must be possible to do it with a windows based system too.