It's ok to hot swap the switch inputs. Technically you're not really hot plugging anything anyway, when you are sitting stationary with no switch pressed, it's just a dead wire coming from Ipac to a dead end on your switch. By unscrewing it, you're just removing an antenna wire with no electrical impact on what you're doing (unless you end up with an electro static discharge but that's another question, ground yourself first!)
When the switch inputs are left floating after you unplug the switch (or while it's plugged in but not pressed) the inputs are pulled to a default high level on the board so you don't leave anything hanging inappropriately either.