Depending on how well it's made, you could just buy a cheap Composite + Stereo audio switcher. Use the composite i/o ports for the Y cable, and the Left/Right ports for the Cb and Cr cables.
So long as they have it adequately shielded and use short wires inside to do the join, it shouldn't affect video quality too much. You will have to then use a separate switch to do the audio switching.
Or, the other thing you could do is get a SVideo switch that also switches a composite signal at the same time. However this one would require you to convert the Cr and Cb cables into a Svideo plug and then again on the other side. This means your Cr/Cb is switched on a Y/C pins of the svideo port, and the Y switched on your composite port, while your stereo audio could still be switched by the audio ports.
LMK if you need a diagram for this.
Alternately, you could just buy a low end Pioneer receiver that offers Component A/V switching (and probably upgrade your sound system at the same time..unless you have a really good receiver as it is)