Basic physics says that YES, it does make a difference. First you have your hand velocity spinning the trackball. Then you have the added speed of your body.
Unfortunately, the difference is so minute, you'll never realize it. Plus the other reasons stated above about control and such.