I had a similar problem with analog audio (via the proprietary connector) and digital optical. AFAIK, the PS3 will not output sound to both simultaneously. You have to change the option in the menu depending on your mood. It's a major shortfall, in my opinion.
Fortunately, I upgraded to an HDMI switching receiver so I don't have that problem anymore.