Nope you can get away with a 4 way 9pin serial switch box and a 2 way Vga Box.
I the the same setup but I have the PSX, Xbox ,DC and PC with Laserdisc. Just make sure you switch to the correct game on the 9 pin switch or you can good up your X Arcade.
I have the Magic VGA which looks pretty cool. Works a treat, now i need to find the time to play with them all.
KVM would not be the best solution IMO.