I don't know why/how it is causing problems, so I won't try to help make it stop, but as far a switching control from the computer to Baby Pac and back again you could use relays. The joystick would work with 1 relay and you could use another relay for up to 4 buttons. The ground staying connected all the time shouldn't be a problem since it should need a complete circuit to have any effect at all on the system. If you found a way to trigger the VGA switch with a relay you could control it all from one switch and never know the difference, when you switched to the computer the monitor and buttons/joystick would all switch over and when you switch back they would all switch back.
That's a very generalized idea, but I could be more detailed provided there isn't a simpler solution. I wonder if putting diodes inline with the ipac button connections would help. Maybe somebody more savy can chime in on that idea.