I've run into this on my config; I have 5 hot swap panels and I had issues when trying to switch in between games while in the Front End. (I use AtomicFE) At first I had to exit out of the FE, swap the panel and get back in. Not a big deal, but I wanted to make it more seamless.
What I ended up doing was mounting all the interfaces in the cab, instead of on each control panel. (Also reduced the number of interfaces I needed!) Then, I just put two female DB25 cables on them. Each panel now just has male DB25 cables. Just make sure to map out what is player 1, 2, etc controls and make sure to connect them properly to where they will match up on the interface.
Just making the mappings consistent will most likely not work for you, as when you remove and add USB devices while in the program, it doesn't update, even though Windows can handle it. However, since I don't use HS, it might work for you.