I'd be surprised if it hasn't already been done. I think the only thing that might be necessary is a quick rewrite of the input code. That might not even be required but I'm sure it'd be trivial if it is. The graphics limitations of PS3 linux shouldn't have any effect and even though it'll only use a single core, I think the cores are pretty comparable to a recent (but not blazing) PC.
Heck, if you use a keyboard or USB keyboard encoder, you don't even need to worry about the controller issue. Just install linux and mame and you're done.