The bigger board is the Opti-Wiz and the two smaller boards are from the Bestons Imperial trackball. I traced the wires from the trackball to the Opti-Wiz and they were correct. My guess at this point is I may have done something wrong with the Opti-wiz. If you look at the board you will see 6 pairs of solder pads between the chip and the wires. I am thinking I need to bridge some of these.
You have the old version of the Opti-Wiz, which was dual purpose. It was also used as an optics carrier on the original TurboTwist spinner. The pads were for assigning the optics to a particular axis, so they play no part when used for external devices.
Your wiring looks correct, so I would suspect flaky optics on one or more of the trackball boards. The best way to troubleshoot them is connect both boards to the same wire pairs from the encoder, one at a time, and note whether there is an issue with one of them. If both are acting strange on one wire pair, then then check the wiring for that pair. If not the wiring, it could be a problem with the OptiWiz. If the same trackball PCB acts strangely on both wire pairs, it's the problem.
Are you using the original, low-res encoder wheels?
RandyT