Heya kspiff!
1) I've never used a Radeon 8500 with MAME, but I haven't heard of any problems with it.
2) I'm using a SB Live, and it works fine. There are DOS drivers available for it, so you're good to go.
3) Never used an optical mouse, so dunno about this one. You're wanting to use it as a mouse, or are you hacking a spinner into it?