I'd be willing to bet your mini-pac or trackball is failing.  It's hardware related,  you isolated it to the control panel.  If I had to bet on a component,  I'd say it's the Mini-pac going bad.  If the trackball were going bad,  it would likely be mechanical failure and would be consistent.
The Mini-pac OTOH,  could initially appear to be ok,  but as it's used whatever component is failing slowly degrades performance.  I'm not familiar with that kind of board/processor design to comment more specifically on what might be failing in it,  on a PC that's behavior I'd expect to see with failing memory/cache memory.  Anyways,  each time you stopped using it as input,  or switched it to another system,  it initially works/improves but then degrades.  Replace the Mini-pac IMO.
Edit:  Actuallly,  double check me,  play marble madness for a bit.  If you see the same behavior,  replace the mini-pac.  Apologies...sleepy!