Your trackball is using a mouse hack: the internal circuit board from a regular PS/2 ball-type mouse, with wires from your trackball soldered in place of the optical sensors from the mouse. The areas circled in blue below control the X and Y axes of the mouse; it looks like all the wires are going there, although it's hard to tell for sure from that (cellphone?) pic. The area circled in red controls the Z-axis; the Z-axis is the scroll wheel from the mouse. Are there any wires going to that area?
Since you get good cursor movement from the Trackball in Windows, you're probably not cross-wired. I'm betting that the scroll wheel optical sensors are still present on the mouse and are sending out a "tick" that MAME sees before you get a chance to send it some movement from the X or Y axes by rolling the ball.
Assuming there are no wires going to the Z-axis area:
A better pic of that area would help, but what you're looking for is a small black component with three legs soldered to the PCB- that's the optical receiver. It'll have a smaller clear component with two legs sitting right next to it- the optical emitter. If the encoder wheel is still present (round spin-able thing with notches cut in the rim), it'll be right between the two. If the wheel is there, remove it, and see if your problem goes with it. Next, try putting a piece of electrical tape around the receiver, to block any light from getting in. If that doesn't work, then remove the receiver, either by de-soldering it, or if that spooks you, you can probably just break it off with pliers, but be really careful not to break anything else.