in the Windows desktop the mouse pointer wiggles back and forth very slightly without ever moving its home position
This points toward one of two possibilities:
1. One of the two optos is burned out, causing that data line to be stuck either high or low. (most likely)
2. One of the encoder inputs is stuck either high or low.
When you slowly turn the axis, the data lines should alternate between logic high (near 5v) and logic low. (near 0v)
This quadrature waveform shows what you would see on the data lines (A and B) when you slowly turn the axis clockwise.

For example, if A is stuck low, the encoder will only see phase 1 or 2, never phase 3 or 4.
Phase 3 looks like Phase 2 -- A = Low, B = Hgh.
Phase 4 looks like Phase 1 -- A = Low, B = Low.
Phase 1 to 2 = move mouse 1 step to the right.
Phase 2 to 1 = move mouse 1 step to the left.
Scott