It definitely depends on what you are wanting to run, and what form factor you are going for. For example, I like early 80's games, and I found that by tweaking MAME command line parms I can most if not all of them quite well (60fps) on a lowly Pentium 233 with a TNT1 video card! On the other hand, more recent 90's titles would need much more power. For most of these, a 1Ghz Athlon would be both affordable and flexible (though as was pointed out, there might always be one or two titles out there that will have problems no matter what your config).
Also, if you are trying to build a form factor that doesn't leave alot of room, you might want to consider all-in-one motherboard combos that integrate sound and video onto the board, to eliminate the 4-7" height needs for PCI & AGP cards. A GeForce 3 does you no good -- focus on strong 2D performance.
The first thing I would recommend is spending a weekend writing down your goals, including a "must have" list of games. Once you know this list, you may be surprised at what you may or may not need from your hardware, and will also help nail down exactly what the control/button requirements are going to be for your panel.
Rich