The "player number" is arbitrary when designing the control panel because you can make any controller Player One, Two, Three, Four, etc. in each game's individual configuration. (Which is why I kind of think it's NOT a good idea when people put the player number markings on their artwork. It's not hard to tell where you need to stand based on the game.)
For example: In Street Fighter Two, the middle controls would be P1 and P2 with the outer controls unmapped (unless you want to map them 1 and 2 for redundancy.) And in Gauntlet, you'll map them from left to right, P1, P2, P3 and P4. Etc.
The only thing you have to worry about is if you use a specific input as a SHIFT button for certain encoders but I don't imagine it's a big deal.