Do you have the Pclock and Hclock settings for this monitor? Is it a 15khz monitor? I've been there dude, Advance Mame is a great version but a pain to configure if you don't have the settings for your monitor.
The blue picture seems as if your cable hack is not setup right. Also, are you combining the vsync and hsync signals from your hack?
I've had the best luck with Trident T64 cards, might be worth the $30 to get one. I've heard good things about Matrox g400 cards as well but have not had any first hand experience.
Of course you could get the Arcade VGA from Ultimarc and save yourself hours of time and frustration!
Doug