I ran into the same problem when I was using a Geforce3 card -- even with limiting the max resolution to 640x480 and the max refresh rate 60hz (using RefreshForce), some resolutions choked (such as 512x484, I think that's it) and would give me an Out of Range Error.
I'm now running an ArcadeVGA card, and no more out of range situations. Another option is to try and use AdvanceMAME, but it's a challenge to set up correctly.
Kevin