Did you actually CUT the wire to the Coin 1 pin, or just disconnect it from the button?
From the sounds of it, you say you spliced into the wire, which would leave the existing coin mech in parallel with the coin button.
I would try cutting the wire that leads to the coin mechanism, and see if that cures it.
If it works properly with just the button as Coin 1, your problem is with the actual mechanism itself.
If you want to check the mech, it's not really complicated.
Just make sure that you hear the switch at the bottom of the mech clicking when the wire that actuates it is hit (as if by a coin).
As far as the settings that were done on Battletoads, if the 4 coin mechs are physically wired individually to the JAMMA harness, the "single coin slot" setting was probably a dipswitch thing.
Most of the "four" player games had dipswitch settings that would allow them to be used as a conversion in 2 player cabs (with only two players available obviously).
Because the game designers didn't know the exact layout of the conversion cab, they also usually included settings for "one coin mech only", "two coin mechs that take the same coin", "two coin mechs that take different coins", etc...
Another benefit of this was that the ops could quickly bypass a non-working coin mech using the dipswitches, and get the game back online.
That allowed them to actually FIX the game after hours, when they wouldn't be losing money doing it.