Personally, I'd only have 3 buttons for player 3 and 4.
That would give you 1 leftover input for an "exit" button.
Keep in mind that two of the buttons you are using will be seen as mouse buttons.
Don't hook them up to a button that's used while in the frontend. It will work fine while in MAME. Might not work in other emulators.
EDIT: If you don't need an exit button, make the one unused input the right mouse button. 
Couple other ideas, less than optimal, but they'd work:
Map the start button to both coin and start. or...
Require the player to hold a button and press start to insert a coin.
....or just get another encoder.