The trackball works like a
mechanical mouse.
Further details and better pictures
here and there's an explanation of quadrature waveforms
here.
There are 4 wires per axis leading to the encoder:
1. 5v - operating voltage for the LED
2. Data A
3. Data B
4. Ground (Sometimes jumpered inside the trackball case, leaving one axis with only 3 wires)
By swapping Data A for Data B, you are swapping quadrature waveform A for waveform B which the encoder translates as turning the other way.

Scott