Mouse smoothing off, or better yet enable multi-mouse in mame.ini so that MAME uses raw input and bypasses the windows effects.
Jerky movement sounds like there's something worse going on though.
You don't have axis INC or axis DEC mapped to the mouse do you?
What type of encoder are you using?