I've had this idea for a while, but never had the time to play around with it, and was always shocked i never saw anyone else do it.
The basic concept is to disconnect the X/Y yokes and instead of driving them with the circuitry on the chassis, drive them with an amplified stereo audio signal from a PC.
not too terribly different from what's being done here:
If you then feed the display a full-screen solid color in sync with the audio, you essentially get a color vector display. I used to take old TVs and hook them up to the amplifiers in cheap PC speakers to turn them into a visualization for music.
Basically the only work to be done would be a driver that produces the corresponding audio signal for the vector output in mame, and direct it to a secondary audio output.