I've been testing an old PIII-800MHz for my MAME cab and call tell you a few things with confidence:
1. For most 3D games, especially the CHD variety, you're screwed anyway so don't even bother. Hopefully you, like me, don't like any of them. If you do, get the fastest Core 2 Duo you can afford an overclockable motherboard (see
this thread2. For 2D games, any video card will do fine, even integrated as long as you have a minimum of memory. The only disadvantage of most on-board video cards is that they use your system memory. I tried an old AGP video card on my system and it did not improve performance in MAME by the slightest bit. Besides, if performance is at a premium and/or you plan to use a real arcade monitor, you should be using directdraw to render the video in which case the video card won't matter anyway. Oh, an if you want it for 3D games, remember you're screwed anyway (see #1). Caveat: your hardware might react differently, please borrow a video card and try this out for yourself.
3. As for external sound card, I have not tried one myself so I can't comment but I don't expect that to make any difference. The memory used for sound is a fraction of that used for video and, again, an external video card does not make much difference. Again, same caveat. Borrow one and try it out.
4. Tiny XP. This is an urban legend that people hear and pass along to others needing to scrape out performance without actually trying it on their own systems because they don't need it having modern computers. I built a super minimalistic install of Windows XP using nLite and removing everything except the motherboard, CPU and sound drivers and Direct X. As a result I gained about 2% improvement in MAME performance. I then went back and made a much bigger installation and cut a lot of fat but kept networking capabilities and I still got the 2%. My point here is that the performance gains are minimal. By the way, avoiding MAMEUI thinking it takes up resources and slow MAME down compared to the command line version is another urban legend. Try it for yourself. Then again I plan to use MALA as front-end.
5. Windows 98SE or DOS. I've never run on DOS but I used to run Windows 98SE back in 2001 and when I switched to XP actually gained a few percentage points improvements. However, it can depend on how my system was configured back then and I am not going to dual Win98SE now just to try it out. Besides it is defunct OS as someone else mentioned and I don't think it is worth the hassle.
6. Memory. The trick is, where the hell is the point of diminishing returns? As I understand it, MAME loads the whole ROM into memory. However, the largest ROM games around tip the scales around 80MB (at least the ones I have tried). I tested Garou: Mark of the Wolves (80GB ROM) with 384 and 512MB of memory and did not see any difference whatsoever so I think that sweet spot is probably somewhere around 192 to 384. Now this is something you should test for yourself. Play with difference combinations of the memory you have now starting with something really low like 128MG. If you don't see any difference after say, 256MB with the most massive games you want to play, then what's the point of getting 512MB? And if you think you might need it for the largest CHD games, don't worry. You're screwed for those, remember?
7. Hard drive: don't bother. Just get one that will hold all your games and have room for the OS, pagefile, etc. The only difference hard drive speed makes is while loading the ROM to memory. After that, it will be just sitting there for the most part.
8. Wireless networking. Try to avoid having this active during games if performance is at a premium. It is the one thing in all of my experiments that gave me a significant performance hit. Of course, you should not be running other applications along with MAME. Shutting down the antivirus did not make any difference for me so you might just want to stay protected.
9. Processor speed: the whole grail of MAME performance. Period. But like me, you want to make what you have work so upgrading here is not an option (and probably unnecessary in your case, keep reading).
10. Earlier MAME versions. Now we're talking

. This is, by very far, the best way to increase performance in MAME.
On my PIII-800MHz, PC-133 512MB, on-board video and sound, I can:
- Play all NeoGeo, CPS-2 games at full speed with MAMEplus v0.104
- Play the Mortal Kombat series with fastMAME v0.56 (uses the same current ROMS)
- Play CPS-3 games with autoframeskip using VC compiled MAMEplus but not very smoothly at all. I am working on this front and I think have a solution (and it is not CPS3emulator). However, I think you will not have a problem with you r1.7GHz on this game.
- Can't play Donkey Kong with the new discrete sound improvements at full speed but you probably can.
11. MAME optimizations. I have tested the different optimization options with the GCC compiler and the gains don't surpass 2% in my system. However, using the VC compiled of MAMEplus I get about a 10% gain in performance so I recommend using this. By the way, I'm not trying to plug for MAMEplus or anything. I have nothing to do with them but I love it because of the performance boost I get from the VC compilation and the ASM core for CPS-2 and NeoGeo games (you don't need this as your computer is more powerful than mine), high score and command list support and confirm quit feature. Even if you don't want MAMEplus, you could compile the regular version of MAME with Visual Studio (free) to get the performance boost.
Then again, there will be some games you won't be able to play but that goes for everybody anyway and you will be able to play most.
I recommend you use the latest VC compiled MAMEplus for the vast majority of your games and try older versions for whatever games you might not run full speed.
Please try playing with some of these suggestions before you waste good money on PC hardware and report back here with any findings you make to help guide others.
Contact me if you have questions.
Regards.