When figuring out how to make my control panel, I really want to avoid certain compromises. I think I've got it all designed, and I'm partway through building it.
The following must be accomodated:
2 player karate champ layout
4 player joystick games
2 Player Street Fighter games
4 spinners (Warlords)
2 (or more) trackballs (marble madness)
Can't be too tight of a squeeze
After scrapping plan after plan, I finally came up with this(sorry for the nasty pictures):

For each of the 4 players, I can remove the controller (notice the ~5-inch square around the joystick) and drop in whatever controller I want. One really nice thing about doing it this way is that I can have all the different combinations that I think I'll ever want without having a whole roomful of 25-inch-wide control panels. I can do it with one boxful of joysticks/trackballs/spinners. Also, this setup will accomodate some of the really absurd two player games like games with dual joysticks and trigger/thumb buttons.
I kinda figured that most of the time there won't be 4 players around, so I'll also have a second more sober-sized panel for only 2 players. I figure that for 2 player games, the 4-player setup that I have won't be ideal because you wouldn't be facing the monitor straight-on (I'm thinking that in marble madness with the trackballs it could pose a particular problem).

With the 25-inch-wide panels being removable, I won't need to use as many buttons because the two large control panels will mostly be just a frame for the 25-inch panels. If need be, I could drop in a whole nother strange control panel that doesn't fit the rest. I can rotate the panel upside-down if I want to switch between right/left layout. A last nice feature is that a 25-inch panel fits perfectly in my keyboard platform for my computer desk:

The two biggest drawbacks are that I probably won't manage to make it look very pretty, and that it is taking me a lot of time to build/design. Between planning the wiring and the woodworking, it sure it a lot of work.
I'll have a full writeup in a few years when it's all completed... It seems that I spend an awful amount of time playing the games instead of just building the darned thing.