Actually, many IC designs are NOT passive when unpowered. They'll hold the bus in a "low" (grounded) state. Even if you remove both power and the ground reference, you may have problems with it trying to hold all the bus lines at the same state.
Also, even at the relatively slow bus speeds used by old console gaming systems, attempting to run several carts to the console without any buffering, especially over long-ish wires, is quite likely to result in signal integrity issues.
What you'd probably have to do in this case in order to make it reliable is buffer each cart individually logically ANDing the buffer enables with "cart slot enable". This isn't *too* hard. IIRC, it's roughly what's done on the multi-slot Neo-Geos. It does require knowing a little bit about the cart interface on your target systems, but most of them are so old at this point that somebody's probably worked it out. You'd probably want to make a PCB to do it, just because the number of connections is high. That would also give you a structure to mount the card slots themselves on, which could be handy.
If you're really interested in it, this is something I could design for you. Send me a PM if so. It may be cheaper to just chuck multiple consoles in the system and switch the video/controls, though.