Definately sounds like the trouble is with the optic board, trackball wiring, or the endoder wheel alignment. If the mouse hack can move the cursor both up/down and right/left, then that's not the problem. (glad the hack went well, btw) If it was, you'd consistently fail to move the cursor along the same axis, the problem wouldn't switch axes when you change the wiring. Which you already figured out.
Now, you've gotta narrow it down again. First, just look real hard at the encoder wheels- verify that they're both turning, and passing through the optics like they should. If that looks OK, then put the wiring back where it started, verify that the problem persists, and then swap the location of the optic boards. If the problem moves with the boards, then that points to the boards. If not, then the wiring seems most likely.