One of your bearings is probably not as good as the other.
If you go to radioshack , pick up this stuff called component cleaner (me thinks) Its in a small white metal can - about 6 inches tall.
Take a bearing and hold it, while spraying the stuff on the top of one side. Let it sit for 30 seconds... so that it can seep into the bearings. Then, spin in several times - for like 15 seconds. Repeat this one more time on the same side. The first application will eat away the gunk, and the second will add clean lubrication. Now, repeat on the opposite side 2X.
After all bearings are finished, they shoula spin like Mad!
If however you notice that after all the bearings have been cleaned... and yet one of them dosnt seem to maintain a long spin-time, you probably have a bad bearing and need to replace it. Try to clean it a few more times before you toss it.
Clean the ball and housing so that nothing can contaminate the fresh bearings. Lube and dry the metal bars the ball sits on too. Close it up and rock and 'roll'

Lastly, I have the 2.5 trackball. The 3 inch tball uses a wider spacing of the bearings. (bearings are placed further twords the sides of the ball rather than under it) Not sure about this... but I think this creates more weight/pressure on the bearings... thus slowing it down with more friction. Not sure if this also may be the problem.