I was adapting a Wico spinner (originally in a Mad Planets) to USB today and could not get it to work. Didn't make sense....it was all hooked up correctly and I've done this many times before. Well here's the rub. Certain USB ball mouse boards don't like straight open collector inputs to the encoder circuitry. So the opto outputs will look OK on a scope, but you'll still get no cursor movement onscreen.
Here's the fix: add 10k pulldown resistors on both phototransistor outputs (p-transistor out to gnd). Apparently some mfgr's started integrating the pulldowns into the IR reciever, so if yer hack doesn't work...chances are you need to add these resistors and you'll be up an' runnin'. So for a trackball (2 axis) you need to add 4 resistors. Its easiest to solder them to the opto boards rather than on the mouse pcb.
Note: this mod applies to trackballs/spinners with *just* the optocoupler and no additional circuitry. If yours has more to it (maybe a 74hc14 chip), you likely won't need to worry about it.