Have you used a cell phone/digital camera to check if the IR LED part of the optical circuit is working? (2 LEDs per axis)
I didn't realize there were two LEDs per axis, but that makes sense. As you can see from the photos, I have removed the emitters and receivers from the mouse, so I assume you mean the trackball LEDs. I thought I had eliminated trackball error as a possibility by proving that either trackball axis can drive the cursor vertically when wired that way. Am I missing something?
Sorry, mis-read that part of your post.

The trackball part of the circuit does appear to be working fine.
The thing to test is whether the signals from the trackball are making it to the IC on the mouse PCB or if one of the traces are shorted/open.
When you slowly turn the encoder wheel, does the voltage vary between high and low on both H1 and H2?

I'm afraid this is way over my head. Can you tell me how exactly I can use my multimeter to check this? I want to learn but I'm a total noob.
Standard IC layout is that if the "notch" on the IC is up, pin 1 is the upper left pin -- in your photo the notch is down, so pin 1 is the lower just above the "R3" label on the PCB.
Set your multimeter to DC Volts. (5v range)
Connect the black lead to ground, touch the red lead to IC pin 10 or 11 for the X-axis and pin 12 or 13 for the Y-axis -- see pgs. 5/6/17/20 of the datasheet
here for the full pinout/description/electrical characteristics/schematic.
As you slowly move the trackball, you will see the voltage increase/decrease as the teeth of the encoder wheel pass/block IR from the LED striking the photodiode sensor on the trackball PCB.
Looks like the highs should be around 3.2v-3.8v and the lows should be around 1.2v-1.9v for the TP8452AP IC. (pg 17 of the datasheet)
Compare the readings on the good axis to the bad axis.
If one of the lines is stuck high, stuck low, or just not changing voltage like the other lines are, that will tell you where to look.
If all 4 lines (X1, X, Y1, Y2) are good, the problem may be a damaged IC.

Scott