I know you probably wanted to cover the most games possible with your setup, but here's what I would do. Make a list of all the games you want to play on your cabinet, with the required controls next to the name (ex: Pac-Man, 1-player, 4-way joystick with no buttons. Samurai Shodown, 2-players, 8-way with four horizontal/curved buttons. Black Tiger, 1-player, 8-way joystick with two horizontal buttons).
Then you'll see what type of control panel would cover most of your games. That should be your primary control panel, if you go with the swapping control panels idea.