Count me in on a "Go for it!" vote. You could have 2 "normal" panels with the normal controls (2 player setup, trackball, spinner, 4 way stick, etc) and then the 3rd modular for all the "non-normal" controls like you say (yoke, steering wheel, etc.). I have to admit that if I thought of doing it that way, making one of my three panels modular, I probably would have. It is a bit much, as you say, but to me it is worth it in the end when you can quickly get just about any CP setup you want by spinning the thing and/or swapping a control or two.