I'd be willing to bet your mini-pac or trackball is failing. It's hardware related, you isolated it to the control panel. If I had to bet on a component, I'd say it's the Mini-pac going bad. If the trackball were going bad, it would likely be mechanical failure and would be consistent.
The Mini-pac OTOH, could initially appear to be ok, but as it's used whatever component is failing slowly degrades performance. I'm not familiar with that kind of board/processor design to comment more specifically on what might be failing in it, on a PC that's behavior I'd expect to see with failing memory/cache memory. Anyways, each time you stopped using it as input, or switched it to another system, it initially works/improves but then degrades. Replace the Mini-pac IMO.
Edit: Actuallly, double check me, play marble madness for a bit. If you see the same behavior, replace the mini-pac. Apologies...sleepy!