That may be the case, but they still roll farther, and straight. Mine's like a mule with a bad leg.
There are a few things to look at. Try loosening the bolts on the case a bit and see what happens. If the case is warped, it could be causing something to bind. Try rolling it without the top cover and see if there is any improvement, and look for any stickage. If there is anything in the bearing wells, get rid of it. Anything that could compress the exterior of the bearing even slightly, will induce drag.
If all looks good, then work the hell out of the trackball, and do it as often as you can, while playing a game or not. All good quality bearings are tight in the races at first, lube or not. If they weren't, they would not have any precision. To rapidly accelerate wear, dissolve all lubrication out of the bearing with solvent, and work them a few hours "dry". Use the solvent again to clean them out and replace the light oil so they don't continue to degrade. There's not much else to do, or which can be done, to get it spinning freely.