MAME Benchmarks

[Chris]

The question often comes up "What processor is good to run x?" and the response is usually ends up pointing to the MAME benchmarks page at http://benchmark.mameworld.net/ .  However, there really aren't that many benchmarks in that database, and lots of the ones that are there are for games that almost no one has any trouble running.

Yes, this request is selfish, 'cause I'm about to upgrade my cabinet... but if everyone could take a moment and add a few benchmarks to the database, I think it would be a great help for all of us.  I'm adding in data for an Athlon 64 at 1800 and 2000 MHz right now, and I'm going to go into my office later where I have access to my test lab and get some on some other machines.  Let's fill that database up! 

--Chris

[Chris]

Here's my additions for the evening:

ROM	MAME Build	CPU	MHz	FPS
area51	0.111	AMD Athlon 64	1800	110
area51	0.111	AMD Athlon 64	2000	128
blazstar	0.111	AMD Athlon 64	2000	178
calspeed	0.111	AMD Athlon 64	1800	14
calspeed	0.111	AMD Athlon 64	2000	18
carnevil	0.111	AMD Athlon 64	2000	34
carnevil	0.111	AMD Athlon 64	1800	31
cbaj	0.111	AMD Athlon 64	2000	61
cottonbm	0.111	AMD Athlon 64	1800	41
crusnusa	0.111	AMD Athlon 64	2000	42
crusnusa	0.111	AMD Athlon 64	1800	39
cryptklr	0.111	AMD Athlon 64	1800	67
cryptklr	0.111	AMD Athlon 64	2000	75
daytona2	0.111	AMD Athlon 64	1800	10
ddra	0.111	AMD Athlon 64	1800	79
ddra	0.111	AMD Athlon 64	2000	88
gforce2	0.111	AMD Athlon 64	2000	97
harddriv	0.111	AMD Athlon 64	2000	63
hthero	0.111	AMD Athlon 64	2000	189
hyperath	0.111	AMD Athlon 64	2000	70
hyperath	0.111	AMD Athlon 64	1800	62
irobot	0.111	AMD Athlon 64	2000	221
kof2002	0.111	AMD Athlon 64	2000	162
openice	0.111	AMD Athlon 64	2000	84
openice	0.111	AMD Athlon 64	1800	74
p47aces	0.111	AMD Athlon 64	2000	87
pacman	0.111	AMD Athlon 64	1800	648
pblbeach	0.111	AMD Athlon 64	1800	60
pblbeach	0.111	AMD Athlon 64	2000	67
propcycl	0.111	AMD Athlon 64	1800	14
propcycl	0.111	AMD Athlon 64	2000	15
ridgerac	0.111	AMD Athlon 64	2000	12
roadblst	0.111	AMD Athlon 64	2000	180
rvschool	0.111	AMD Athlon 64	1800	32
rvschool	0.111	AMD Athlon 64	2000	39
sfex	0.111	AMD Athlon 64	2000	51
simpsons	0.111	AMD Athlon 64	1800	250
sonic	0.111	AMD Athlon 64	1800	89
souledge	0.111	AMD Athlon 64	2000	55
souledge	0.111	AMD Athlon 64	1800	49
starblad	0.111	AMD Athlon 64	1800	44
starblad	0.111	AMD Athlon 64	2000	50
stunrun	0.111	AMD Athlon 64	2000	91
stunrun	0.111	AMD Athlon 64	1800	83
surfplnt	0.111	AMD Athlon 64	2000	20
swa	0.111	AMD Athlon 64	2000	102
sws95	0.111	AMD Athlon 64	2000	127
sws99	0.111	AMD Athlon 64	1800	35
sws99	0.111	AMD Athlon 64	2000	36
swtrilgy	0.111	AMD Athlon 64	1800	9
tekken	0.111	AMD Athlon 64	2000	55
tekken	0.111	AMD Athlon 64	1800	50
tekken2	0.111	AMD Athlon 64	1800	50
tekken2	0.111	AMD Athlon 64	2000	57
tetrisp	0.111	AMD Athlon 64	1800	107
tetrisp	0.111	AMD Athlon 64	2000	122
timecris	0.111	AMD Athlon 64	2000	15
timecris	0.111	AMD Athlon 64	1800	13
toutrun	0.111	AMD Athlon 64	2000	121
vbowl	0.111	AMD Athlon 64	2000	74
vbowl	0.111	AMD Athlon 64	1800	65
vf	0.111	AMD Athlon 64	2000	71
vf	0.111	AMD Athlon 64	1800	64
vfkids	0.111	AMD Athlon 64	1800	28
vformula	0.111	AMD Athlon 64	2000	75
vfremix	0.111	AMD Athlon 64	1800	37

[Chris]

I just finished running some benchmarks using my P4 3.2GHz at work... I was surprised by the results.  On the newer 3D systems the P4 outperformed the Athlon 64 2800+ (1.8GHz, some tests overclocked to 2GHz), as expected.  But on older 2D systems, the Athlon was usually faster, sometimes by a wide margin!

Example: Soul Edge
Athlon 64 at 1800 MHz: 49 fps
Athlon 64 at 2000 MHz: 55 fps
Pentium 4 at 3200 MHz: 68 fps

Example: RoadBlasters
Athlon 64 at 2000 MHz: 180 fps
Pentium 4 at 3200 MHz: 96 fps

And here's the real surprise...

Example: Pac-Man
Athlon 64 at 1800 MHz: 648 fps
Pentium 4 at 3200 MHz: 87 fps

 

Both tests run using the same version of MAME .111b , the default download.  I'm at a loss to explain the P4's aversion to Pac-Man.

--Chris

[releasedtruth]

Very interesting results I would say. I should do some benchmarking of my own. I have 3 machines to try things out on, but my P4 2.2 machine that runs the cab isn't exactly modern by any stretch.

GT

[Chris]

my P4 2.2 machine that runs the cab isn't exactly modern by any stretch.

This isn't necessarily a bad thing, as many of us use "surplus" hardware for building cabs...

[Taborious]

What are you doing the benchmarks with? and how easy is it to use. I haven't done a lot of benchmarking but would like to start...

[Chris]

There's a batch file on that site to run the test.  I had to tinker with it a tad to take out options that aren't in MAME anymore.  The one I used is:

mame %1 -noafs -nothrottle -ftr 10000 -r 1280x1024x32 -nowaitvsync -norc -nosleep -effect none -nobezel -nooverlay

I saved that as bench1280.bat, so to test, say, Pac-Man, I just do:

bench1280 pacman

on the command line.  It'll run through 10,000 frames as fast as possible and exit back to the command line showing the FPS.  For games like Pac-Man, 10,000 frames will be over in seconds; for a 3D game that's taxing your system, it could take 15 minutes for the benchmark to run, although theoretically once it's looped through the attract sequence at least once you should be able to exit and still have a reasonable result.

For a couple of the games that have to be calibrated or have a flash RAM prepared, I ignored the first result and ran it again after the calibration or preparation was complete.

[Taborious]

You're running the test at 1280 resolution? I'm only running 800x600 I think, so what, if any, is the logic behind the resolution your test at; just whatever one you want?

[Chris]

You're running the test at 1280 resolution? I'm only running 800x600 I think, so what, if any, is the logic behind the resolution your test at; just whatever one you want?

That's to match the other entries in the chart.  The site says to use 1280 for PC monitor output and 800 for a TV or arcade monitor setup.  I'll bet the resolution only makes a big FPS difference on vector games.

[Chris]

Looking at NewEgg, I'm strongly thinking of getting an Athlon 64 4000+ (San Diego core at 2.4GHz), maybe mildly overclocking it to 2.6.  I can get the processor and motherboard for $137 total at NewEgg.  The potential competition is the Pentium D 805 (Smithfield core at 2.66GHz); motherboard and processor total $127 at NewEgg.  Decisions decisions.  The Pentium D is a dual core, but MAME won't use the extra CPU... maybe Windows will?  Neither one of these CPUs is on the benchmark table yet...

[Taborious]

I have always wondered if turning of hyperthreading helped any of the benchmarks? Anyone run benchmarks with hypertherading on and then off?

[Chris]

Now that you mention it, I have seen reports that turning off hyperthreading can help, but I haven't tested it.  Maybe that would explain some of the poor results in Pac-Man.  I'll try it without the hyperthreading when I get a chance and see if that improves things.

--Chris

[Chris]

Nope... I tried Pac-Man and Soul Edge really quicky and got exactly the same frame rates with HyperThreading turned off.

[Lilwolf]

I've always thought it would be better to rank all games on one piece of hardware (ok maybe a few, AMD, Intel, ect).  And not emphasize the frame rate, but how they rate compared to each other.

Why?  So when you buy a new piece of hardware, you can run down the lost and move up / down the list until you find the slowest games you can run at near / full fps... And only add those to your cab.

This way you don't have to look around and try them all... or worse yet... run a game for the first time when others are over... and make your cab seem broken.

[u_rebelscum]

I've always thought it would be better to rank all games on one piece of hardware (ok maybe a few, AMD, Intel, ect).  And not emphasize the frame rate, but how they rate compared to each other.

Why?  So when you buy a new piece of hardware, you can run down the lost and move up / down the list until you find the slowest games you can run at near / full fps... And only add those to your cab.

This way you don't have to look around and try them all... or worse yet... run a game for the first time when others are over... and make your cab seem broken.

Not bad idea for comparing general game play. 

People won't be satisfied though, since different games like different hardware err... differently. 
You'd need a list for at least a64, p4, & core2 CPUs, and possibly for core, pm, p3, semperon, & athlon(32); game P(sx) might run slower than game Q on cpu A(md), but faster on cpu I(ntel). 
Add that game X might be okay at 90% fps while game Y sucks at steady 95% fps. 
Or game Z averages 97% fps but goes to 50% at some key point vs that game W that never goes below 90% averaging 95%. 
And newer versions of mame effect different games more than others (such as adding dynamic recompilation, or better emulation) possibly moving games up or down the list. 
And there are 6000+ games (3000+ parents); by the time you run them all, the next major version will be out.

Not that that game relativity lists are a bad idea; they would make it easier for people with slightly different hardware than tested to dial in on which boarderline games to test, and which are hopeless.  I just don't think they'll be any easier to keep up to date.

Nope... I tried Pac-Man and Soul Edge really quicky and got exactly the same frame rates with HyperThreading turned off.

Chris, did you enable mame's multithread option (-mt) with hyperthread on?  Looking at the mame32 benches, John noticed a difference.  Compare the d3d@3.6, d3d/MT@3.6, and nv/ns@3.6 columns.  (-mt = multithread, nv/ns = no video, no sound).  He used a core 2 duo, though.

[Chris]

Chris, did you enable mame's multithread option (-mt) with hyperthread on? 

No, I hadn't.  It didn't really make a difference with Soul Edge (70 with -mt vs 68 without), but it made a HUGE difference with Pac-Man (156 with -mt vs 88 without).  Still way, way behind the Athlon, though.

[Chris]

I added benchmarks for an Athlon 64 3700+ San Diego (2200 MHz) last night and I'm doing an Athlon 64 4000+ San Diego (2400 MHz) right now.

[Buddabing]

Aaron Giles did a bunch of benchmarking recently. Look on his page.