If you won't be running newer games on this system EVER.. and you know that it'll only be for MAME, go with AGP. You can then use an AVGA if you ever wanted to, and you will never really need a brand new video card for MAME, even if you want to try to run newer CHD games. AGP will still be around for a number of years, there will be cards, and I'm betting there will also be AVGA cards.
PCI-E is the new format, get it if you want to futureproof your system a little bit, or if you're fairly certain that you'll be upgrading this PC as opposed to just getting a newer one down the line.
MikeQ's explanation is true, but is mostly irrelevant for a MAME PC. You won't see any performance gains on a PCI-E card over an AGP card using MAME.