I think you need to fiddle with MAMEs settings, this is possibly what you need.
-[no]hwstretch / -[no]hws
MAME uses the hardware stretching abilities of modern graphic cards to
scale the game image to the requested resolution. Depending on the
quality of your graphic card and its drivers, this may be a fractional,
antialiased scaling (nice) or an integer, blocky scaling (not so nice).
In any case, you can disable this stretching altogether and let MAME do
the scaling in software, which is probably slower, though. The default
is ON (-hwstretch). Note: Vector games may actually look better with
"-nohws".