There are about as many different opinions on this topic as there are members of this community. There really is no "best way" at all and its one of those things that you just have to find out what suits you best. With that said, I won't stop there and leave you with the typical "do what suits you best" message, but offer to you what I think are the best ideas.
Frankly, if you are going to make swappable cp's you MUST make them so that it is done extremely easy. The reason I say that is if it is a hassle to swap the panels then you aren't going to do so, and you'll just wish you had a more "all-in-one" panel. If you are going to have swappable panels, have one connector to plug, unpug, lift panel, replace panel. The velcro idea is very good.
On that same note, I'm not a big fan of "all-in-one" panels. They are too cluttered, big, and things just get in the way. Again, personal preference. However, I also wouldn't like to have a panel for virtually every different type of control, leaving you with 4 or 5 different panels that you have to swap every time you want to switch games. These also take a long time to build and you also have to store them somewhere when not in use.
The key is to find a happy medium between number of panels (or modular inserts) and controls on each. IE: The fewer panels with the most controls without getting "cluttered," the best.
This is vital: Combine your buttons with as many controls as possible. Too many people use too many unnecessary buttons. If you have a panel with 2 joys and 6 buttons each and you have a trackball and a spinner on that panel too, there is no need for any extra buttons. Just use the same buttons for each control!
I hope that this gives you a good guideline, and if you need any further help, just ask away!
AD