It's possible the mouse has a non standard pinout. I'd expect a 1 pixel movement rapidly back and forth if:
A: you overheated one half of the optical sensor
B: the common pin is swapped with another.
If this happens on both axis, I'd think it's likely the pinout.
+-----+
| O O |
+-----+
| | |
L C R
'normal'
+-----+
| O O |
+-----+
| | |
L R C
'??'