Those are command line options for windows mame, too.
Actually the handiest vector tool is the vector.ini file, which mame reads instead of mame.ini if the game is a vector game. Just copy, paste, and rename to vector.ini and change the settings.
Though the resolution would only go so high on a monitor, but you may be able to increase the brightness to not notice the scanlines as much.