As the person above says.
Mame gets slower because (As I understand the project plans) as time goes on it becomes optimized for the newer hardware at the expense of the older hardware. Hardware as in the arcade machines, not your computer.
Recompiling mame by stripping out all but one game won't change things, because you won't have changed what the targets are, just removed the targets.
If you want to improve speed of some older game, your best bet is to find and use earlier versions. I wouldn't use the earliest version with support, my understanding is that the earliest version is likely to be the least accurate, most likely to have some bug, or least complete.
I believe the mameinfo.dat will give clues as to which version might be best for the game you want, it will list the revisions in which any given game had some change occur. That'll give you a good idea of which build should be optimal.
Haze reads these boards, and i think a few other mamedev's do too, if I'm off on anything hopefully one of them can correct me.