Last setup I fixed with this issue had a volume bad pot (of all things). Other problems I've had causing this kind of behavior are bad AC coupling caps, lifted/poorly connected ground traces, and bad connectors.
I presume you've tested this with various sources and all exhibit the same problem?
Do you have a scope? A very effective method for finding the problem is to put a nice pure sine wave in at a reasonable amplitude and probe around until you see something that isn't a sine wave anymore (clipped, tons of added harmonics, etc.), following the signal path. You can accomplish something similar by probing around with a speaker attached to a high-impedance amplifier, listening for when it starts to sound bad.