I have done this hack several times using a cheap USB ball mouse and Wico home model trackballs. You need to know how these things work before you set out to make this happen.
There's 4 wires to the slotted opto. 5v (thru current limiting resistor matched to the opto type), gnd, speed, and direction. You need to identify all of these connections with a meter and/or a scope. Pwr and gnd should be self explanitory. Speed will show a pulse train proportional to the rate you spin the shaft with the encoder disc on it. The direction is just a high or low pulse. You see it change when you reverse the direction (tells the hardware which way the disc is turning). The trackball type should be identified via its manual (or the manual for the game it was used in) and the connections will be detailed. Its up to you to identify the connections at the mouse optos. So really, you need to do this before taking the opto's out of the mouse board.....