In 0.125, there is no function to control the size or aspect of the snapshots. You have do to this with image resizing software. Effective MAME 0.125u6 - there is.
-SNAPSIZE X,Y
For best results, you want to do it in proportion with the original resolution. Let's say a game you want a large screen shot has a resolution of 288x224... A good size to start would be 3 times that.. so - (288x3=864 and 224x3=672). Issue the command-line -snapsize 864x672 when starting MAME and when you press F12 should produce a much larger picture.