I started out years ago with swappable panels, held in place by cabinet locks.
That worked pretty well for the full panels, but there was a bit of slop to them.
You could bend the catches to minimize it, but it would return after awhile.
After seeing how many duplicate controls I was going to have to buy, I swapped over to a modular system.
I tried the cabinet catches, but didn't like them because they were more prone to coming loose when the remainder of the CP wasn't being held down by your button hand.
I experimented with Velcro, but didn't like the way it wiggled around.
My cab was fully built, so I didn't have the ability to retrofit a tilt-in system.
I couldn't get the router where it need to go to do it, so I opted for a bolt-in system.
The bolt-in system is slower than I'd really like, but very secure.
I tend to fixate on a single game for awhile, so I don't do swapovers after every game.
That is the biggest complaint I've heard from others about modular.
They want a one-panel fits all solution, to avoid having to swap.
I prefer having exactly the right controls for the game I want to play.
As far as control swaps go, RJ45 is definitely the way to go for that.
You can connect/disconnect the modules with one hand, which leaves the other free to hold the module.
The analog swapovers are a bit more problematic.
I put an RJ45 jack between my mouse hack and my controls, but you need to disconnect the USB cable before disconnecting the RJ45, or you risk causing a surge on the power/ground lines.
The other problem is that you have to relaunch MAME if you swap controls, because it only initializes what's there when it launches.
All of the problems are far outweighed by having the proper controls, and ONLY the proper controls, for the game you want to play, IMO.