Yep. That's subscaling the 320 px into 256; don't do that. I was wondering if the game was trying to switch from 256 to 320 and your settings were preventing from it, which is the case if you enable integer scaling. As you're on LX and with an old MAME version, I don't know what guide you used to configure the INI, but you normally wouldn't need these:
-nounevenstretch
-nounevenstretchx
Quick per-game solution: force -resolution 320x240 (or whichever the game actually uses).
For the cases where there's a dynamic video mode change at least, ideally you should be using a super wide resolution of, say, 2560 px (for Puyo-Puyo/MD-based games).