To keep the extra joystick from messing up player 1 or whatever as rampy describes, you could put an A/B switch on their grounds, so that only one can be engaged at a time.
I'd suggest either a slider/toggle switch or a light up button that'll indicate when the 4 way is engaged, so you don't get your first guy killed using a dead stick.
You can still wire all the directional switches in parallel, but the commons/grounds would go to the switch, which would then go on to the iPAC.