Alright, I purchased some "electronics cleaner" from Radio Shack, and used this to clean out the bearings.  This was what someone suggested on a different thread.   After using this on them, they all seemed to spin much better than ever before.    I then used some 3-in-1 oil in each bearing.   This slowed them down a bit, from when there was nothing in them, but they still spinned pretty well.   And in the long run, I am sure having oil in there is best.    I spun them on a drill for a few minutes each as well.   Basically I just put a pencil and tightened it in my drill.  Then I put the bearing on the tip of the pencil.    I held onto the bearing, and turned the drill on.   So anyway, I did this for a few minutes each.        I put my newly cleaned, newly oiled, freshly spun bearings back into my trackball.   It still does not spin correctly.  When I give it a good push upwards, the cursor still goes up straight for a moment, and then drifts to the upper left.    If I spin it downward, it drifts to the lower right.   As shown in my picture that I originally attached.  However the drift is much much much more severe than what I've portrayed in my picture.
I feel like throwing this trackball out the window...   

So, I now have two entire trackballs, complete with bearings and encoder wheels and everything.    I also have two more additional encoder wheels, and 6 extra bearings.   With all of these items, I cannot get one single working trackball.   No matter what combination of bearings, wheels etc, I use, I can't get it to roll right.   The drift ruins everything.  It makes games like Golden Tee and any bowling game virtually unplayable.   I rarely, if ever, can get a straight drive/roll.   I've tried using both trackball cases (with their included optical sensor things).   Both do the same thing, so I know it's not the sensors that are the problem.
What am I doing wrong here guys?   Do your trackballs go completely straight up and stay straight when you give them a good push?   Should I just give up here?    I am at a complete loss.