It could be a loose connection from the speaker wire to the jamma connector - check the solder joint to make sure its good (on both the + and - sides) since if thats what you're wiggling around its probably that. If the solder is good, or replacing it doesnt help, then it might be on the board in which case thats beyond my field of expertise, it could also possibly be a cold solder joint on the speaker (or loose wire if its just wrapped around like in my case :/), but it doesnt sound like it from what you've said