cdbrown,
There are several sites that document mouse hacks. (links on this site) That's how I learned how to attempt it.
The basics are:
1. desoldering the existing mouse optics
2. soldering wires into the place where the optics came out of
3. connect those wire to the proper pins on the trackball optics.
Basically, there are 6 pins (plus buttons if you want them): 5v, ground, 2 signals for the x-axis, 2 signals for the y-axis.
If the trackball works backwards, switch the 2 signal wires for the appropriate axis.
I used a multi-meter (plus looking at the traces) to figure out where to get a 5v source and a ground. The signals come from 2 of the 3 pins of the optic receiver. On mine it was the 2 outer pins.
I learned to de-solder during my hack

Since the mice were only a buck each, I figured, dive in and see what I could do.
Anyway, if you can get dmame to work you should be able to play 2 player trackball games anyway.
Rocky