I think you need to fiddle with MAMEs settings, this is possibly what you need.
-[no]hwstretch / -[no]hws
   MAME uses the hardware stretching abilities of modern graphic cards to
   scale the game image to the requested resolution. Depending on the
   quality of your graphic card and its drivers, this may be a fractional,
   antialiased scaling (nice) or an integer, blocky scaling (not so nice).
   In any case, you can disable this stretching altogether and let MAME do
   the scaling in software, which is probably slower, though. The default
   is ON (-hwstretch). Note: Vector games may actually look better with
   "-nohws".