I would, as a test, play around with the two sensitivity variables: In windows, and then in MAME. I'd probably turn it down to where it feels pretty slow in Windows, and then see if the backspin still happens in MAME. Windows, of course, doesn't use the word "sensitivity", it uses the words "cursor speed", so be aware of that. If it still does jump around too much for your taste, turn down the Y sensitivity in MAME. Start with half, and then go from there. By default in MAME, most games are at 100, so try to set it to 50.
Oh, and tabbing back and forth between Windows and MAME can cause bad stuff to happen, too. Make sure to steer clear of that. If you are using MAMEUI and not playing games in full screen, there are many issues that can happen when both Windows and MAMEUI want to use the mouse.