Start by making a list of games that you 
most want to play and focus on them. (Step #2 in the design process 
here)
If your list is "all of them", you will almost certainly 
not be happy with the resulting Frankenpanel.
Try to limit the list to less than 200 titles.
Sort the list into groups of games that use the same types of controls.
For example, if Ikari Warrors and Star Wars are on your list, you can make one panel/standalone controller with mechanical rotary joysticks and another with a SW yoke/analog joystick.
The rotary joystick panel would need a rotary encoder and should have 3 buttons if you want to play Ikari Warriors 3 in addition to all the other 
rotary joystick games.
The SW yoke/analog joystick needs an analog encoder for the potentiometers used in these 
analog joystick games.
This wiki page as links to the five major types of encoders: 
    Keyboard Encoders
    Gamepad Encoders
    Rotary Encoders
    Optical (Mouse) Encoders
    Analog Encoders 
Scott