That's a really nice cp Telengard...it has gotten me thinking about modular ideas. It seems like changing around components in yours might be a bit slow, I would rather have them just pull out and plop in the replacement.
What do you think about this for an idea. Have the top of the cp be a hinged frame that lays over the control panel modules and clamps down holding them in place? To change say a regular joy for an ikari joy you would un-clamp the cp hinge, raise it up, pull out the regular joy, unplug the ethernet plugs, plug in the new joy's plugs and clamp the top back down...shouldn't take more than about 20 seconds or so.
It's actually not as bad as you would think. I use very short screws and have a hand held cordless screwdriver. Very rarely do I have to swap them all so it usually takes under a minute. A hinge would be a good idea for sure. I had a few things I had to keep in mind when building mine:
I have a metal control panel so I had to use something thin, so the rack mount blanker panels were perfect.
They are also standard sizes so I don't have to worry about gaps between modules. I bought aluminum ones since they are easy to drill through. I used to have steel swappable panels (entire panels) and it would take quite a long time to cut through.
I seem to be very rough on controls without realizing it (seems somewhat common). I wanted the modules to be rock solid. The second it feels like there is something loose I know I would start to play more gingerly which when playing a game ruins it for me. The screws and panels *never* feel like they are going to move, even during a very rough game of Robotron.

Someone implemented something similar to what you mentioned but it involved something other than a hinge I think going over the top of the modules. Do a search on modular and I'm sure you'll find it. They look great but they are wood and I don't think I could pull off cutting wood accurately enough.
~telengard