Part of it is what the compiler can do. Actually, I think mame compiles better than pentium pro. It's compiled with the I686 flag set to on. That's 686, pentium III class I believe.
Anyway, it's also in part what the compiler supports too.
IIRC:
486 = i486

pentium = i586
pentium MMX = i585 + MMX code
pentium pro = i686
pentium II = pentium pro w/ different cache + MMX
pentium III = pentium II + SSE
pentium IV = "i786" (includes MMX, SSE, and SSEII)
If you notice, pentium pro, II, and III are all i686 chips, with the only changes are the cache and new math stuff (+ Mhz, but we're talking about compiling). The rest of the chips' timings, strengths and weaknesses are about the same; so optimizingfor them are about the same.
Pentium IV is the first chip with changes other than adding math stuff.