I am having a similar problem and it shows up with both AtomicFE and MameWah. I'm using Mame 0.69 and I can call mame from the command line and the resolutions work perfectly, but from within MameWah or AtomicFE, they come out in weird, high resolution modes with very small screens. I have yet to figure out exactly what the problem is.
I'm beginning to wonder if there is some stretching/sizing mode I may have turned off in the cfg, but also, I am not seeing the games pay attention to the ini resolution settings when called from AtomicFE or MameWah. I have been working on this issue for months and I'm completely stumped.