The running process doing the final php processing (whether that be the web server or in our case a fastCGI process) requires write access to the entire SMF installation. So at a minimum do owner with write and add any more you feel necessary for your needs. Patching, uploading of avatars/attachments and package installation require write access. If you wanted to be paranoid, you could change permission on the files after these events, and I'd be happy to explain in detail where that could go. But, I feel it more important to lock the front door and leave the windows open if that's what is required to operate.
The SMF docs and guys suggest just putting everything at 777 and 666 which is just poor in so many ways. Then the real battle is keeping your software current.
I'm always happy to discuss what we do behind the curtain if anybody else is interested.