Hi alex,
I'd say the values for the 512x240 mode are better, anyway test both options and check which one works better. You need to set 'monitor custom' in mame.ini, then add either one of these lines:
(Values taken from 640 x 480 modeline)
crt_range0 15625-16500, 49.50-65.00, 2.629, 4.428, 9.271, 0.064, 0.192, 1.024, 0, 0, 192, 288, 448, 576
(Values taken from 512 x 240 modeline)
crt_range0 15625-16500, 49.50-65.00, 4.436, 4.777, 9.896, 0.064, 0.192, 1.024, 0, 0, 192, 288, 448, 576
Notice how the remarked values are just taken from your Powerstrip window. This will create modelines which are similar in size and geometry to the ones used as sample.