This sounds like something is going wrong PEBKAC. It's not really possible for this to happen in MAME or GM, so..
Setup a test copy of MAME or GM with nothing else running, run a game and hit "5" which is default for coin 1. Now rebind it and try again, remembering that you have to go through both the default controls and the player 1 controls. You can set "select" and coin 1 to the same key, but nothing else should be bound to that key.
If this fails, download a new, different version of MAME/GM and repeat.
Somewhere along the line, hopefully before you get to rebuilding your whole setup, you'll figure out where the problem is
Other programs can interfere, and we all make mistakes and have to backtrack.