You might also have a defective or worn pot... or the pot is the wrong type.
There are linear and audio taper at radio shack. I believe you want linear.
Where it goes like : 1,2,3,4,5,6,7,8,9... instead of the other type which
is more like: 1,2,5,8,12,30,100...