I'd go with MacOS 9.0.4 or 9.1 so you can run ancient iTunes and play MacMAME too.
Don't do this!!
You don't have enough RAM to run OS9 by itself, much less putting anything over the top of it.
I put OS 8 on a co-workers 5300 with 24 Meg of RAM, and it will barely run the OS without using virtual memory.
NOTE: You want to turn virtual memory OFF in the memory control panel, to FORCE it to use the RAM, instead of the hard drive.
You also want to turn off all unnecessary extensions, to free up as much processing time as possible.
If you are unfamiliar with Macs, imagine running WinXP on a Pentium 100 with 8 Meg of RAM--vs running Win 3.1.
That's about the difference is OS size we are talking about here.
I wouldn't go any newer than OS7.6, or you are going to have RAM problems.
I'd also stay back at MacMAME 33.
It has the least strenuous requirements of any of the early versions.
As MinerAl mentioned, the games supported by 36 probably won't run anyway, and the newer a version you go, the bigger the code gets.
MacMAME 33 Final REQUIRES 6,082 K to launch. (3,000 if you enable virtual memory, and take the performance hit).
MacMAME 36 REQUIRES 16,540. (10,000 with virtual memory).