Mouse hacks can work better than what you describe, usually adjusting the analog sensetivity can get things working properly. I've never done a hack like the one you did, I always use a real encoder wheel and optic board wired in to the electronics of the mouse, and never had the trouble you are having. You may have lousy mouse electronics, encoder wheels that are too low-res, or maybe the trouble is with your PC, since you say you have similar trouble with just a regular mouse. You ought to be able to get smooth play in Tempest without stutter or backspin. First, I'd try to achieve that, possibly by tweaking the mouse settings in Windows (assuming that's what you're using). Generally, it is considered best to lower the Windows sensetivity all the way, disable any acceleration or enhanced precision settings, and stick with generic Windows mouse drivers.
Once you've got things playing smoothly with other hardware, you can determine if your spinner hack is having any trouble of it's own. But if you can't get things playing right without the spinner hack, then (at least part of) your trouble is elsewhere.