Hey Kaytrim,
Was there any more progress on this panel? I have recently started construction on my own semi-modular panel (
thread here), and I had no idea that this thread existed. My idea was very similar to yours, in that not everything would be modular. In mine there will be only three module bays, and a module will contain the entire joystick/button assembly (or trackball, or wheel, etc.).
Anyway, I was struggling with some of the same quick-connect and panel mounting issues that you were considering. While I haven't yet finalized my panel interlock design, I decided to abandon the self-plugging connectors in favor of CAT5 cables manually plugged in inside the CP box. It just seemed that the complexity and real-estate needed for the self-plugging connectors was going to mean my panel may never be finished.
So did you make any progress?