I'm having problems in XP as well. When I roll upwards very fast, my Happ trackball actually causes the cursor to STOP partway up the screen for a while...
If you're having problems in window desktop, focus fixing that before trying to adjust mame. Desktop problems indicate one of a few things:
a) encoder wheel spins to fast for sensors
b) interface isn't fast enough to count the changes sent from the sensors
c) interface doesn't send the info fast enough to prevent overflows
d) window's mouse accelleration and/or speed is set too high
Since a-c are more toward the hardware side, see if it's d) first. Do something like Chris suggestion step 1, except set the mouse speed very low and turn off enhanced precision. (I could be wrong about the enhance part, see below.)
... accelleration is turned off, and enhance pointer precision is on...
Chris, I thought "Enhance pointer precision" was a renamed mouse accell. (except I guess "enhance" can change the accell curve.) Am I wrong here?
Hmm, with the change accel curve, I guess you might be able to make it a decell curve... [shrug]