Yeah, it's definitely a pain, though not really for most people with relatively normal Mame cabinets.
But if you're planning swappable control panels, or if you want to have a 2 player cabinet, but be able to plug a couple of handheld controllers in for 4p games when friends are over, things can go south quick.