MAME(tm) Benchmark project

[GGKoul]

Hello

Since people are always wondering what speed of processor is required to run most of the games in MAME at a decent framerate.

[Gunstar Hero]

I think you mean Metal Slug 4. 

[GGKoul]

I think you mean Metal Slug 4.

[menace]

How are you going to tackle the mame version conundrum?--i.e. older versions are faster...

For instance i can run .37b12 on a p2 266 with 64Mb ram no problems but not .62.  But I can run .62 on a p3 450 with 128mb ram.

[Gunstar Hero]

Killer Instinct 2 would be a good one... it's a bit more "playable" than California Speed since you don't have to deal with that bizarre steering initialization on "power up."

[2600]

Aren't there already a couple of projects like this?  Don't think they are super actively maintained, but if people aren't using them now they aren't going to in the future.

People ask because they are hoping for a quick answer and are too lazy to search or do any kind of research.


Now if you want to do this from a documentation standpoint and want to document speeds on all versions of mame and changes between them that may be interesting, but you aren't going to get proper results with 5 games.

[lokki]

There are a few bench marks out there already... but like 2.6k said they are not updated frequently.


http://www.classicgaming.com/mame32qa/bench.htm

http://benchmark.mameworld.net/

Regarding your list.

KOF and MSlug both run on the neogeo. So if one runs OK the other should be ok as well. (minor differences).


Some good test games (to see how well your computer fares)

Include

CrusnUsa series (midv)
Cotton Boomerang (STV)
Tempest (Test Vector games)
MK II
Carnevil (Seattle)
Killer Instict
Tekken (psx)

ETC

[GGKoul]

I'm just trying to get a list out there that outlines if I have X processor,

[SirPoonga]

Neogoe is such a good system to test with.  Definately look at super dodge ball's attract scene.

[GGKoul]

There are a few bench marks out there already... but like 2.6k said they are not updated frequently.

http://www.classicgaming.com/mame32qa/bench.htm

[Lilwolf]

I would personally love to see someone benchmark all games on ONE processor and put them in order...

So you say... I can play game X perfectly... so I should be able to play up to game Y... Wow, game Z runs too slow... so all games from that point on shouldn't work.

I don't really care about 4 games running on different systems... because I can just run all 4 of them.... What I really want to know is what games I shouldn't even try to run on my system...

[tommy]

How would mame run on this system i never used this one before,dell.celeron 1.70 ,radion 9200, 640 ram

[SirPoonga]

I would personally love to see someone benchmark all games on ONE processor and put them in order...

So you say... I can play game X perfectly... so I should be able to play up to game Y... Wow, game Z runs too slow... so all games from that point on shouldn't work.

I don't really care about 4 games running on different systems... because I can just run all 4 of them.... What I really want to know is what games I shouldn't even try to run on my system...

Actually, this shouldn't be too hard to script.  We know you can start and stop mame with any game (look at HowardC's screensaver), just add to that reading the stats that mame spits out when you put it in verbose.

[GGKoul]

I would personally love to see someone benchmark all games on ONE processor and put them in order...

So you say... I can play game X perfectly... so I should be able to play up to game Y... Wow, game Z runs too slow... so all games from that point on shouldn't work.

I don't really care about 4 games running on different systems... because I can just run all 4 of them.... What I really want to know is what games I shouldn't even try to run on my system...

Could be done.

[SirPoonga]

If you are going to benchmark you should include

1) a vector game
2) a distrete sound game (asteroids)
3) Neogeo
4) CHD game
5) The largest rom game, garou or whatever it is
6) pacman, it's how mame started!
7) game with heavy sound processing (mk series)
duel monitor games (pc10?)
9) Average 80s vertical monitor game
10average 80 Horizontal monitor game
12) same with 90s
13) a zinc game
14) gtg?  well, shuffleshot would work just as fine, same hardware.
15) super dodge ball's attract screen  I forgot which version, I think the neogeo version, but would have to try all three.

[SirPoonga]

When benchmarking I think the game needs to run at idle AND while in use. 

So the best thing to do would be come up with a list of games.  Create a playback script for each in idle and in use.  Then get stats for each.

I think this all could be done with a script too.  You just load mame and the roms on a machine and runt he script.  Come back in about an hour and you will have your results

[GGKoul]

When benchmarking I think the game needs to run at idle AND while in use.

[lokki]

One of the hard parts about bench marking MAME is it is a moving target.
Once you finished benchmarking every game a new version (or u update) will come out and you will need to start again :-(

For example Aaron is currently revamping the MAME sound system. And this will likely mean a performance hit for every game in MAME

http://aarongiles.com/weblog/archives/2005/02/speed_and_the_n.html

[lokki]

If you are going to benchmark you should include

1) a vector game
2) a distrete sound game (asteroids)
3) Neogeo
4) CHD game
5) The largest rom game, garou or whatever it is
6) pacman, it's how mame started!
7) game with heavy sound processing (mk series)
duel monitor games (pc10?)
9) Average 80s vertical monitor game
10average 80 Horizontal monitor game
12) same with 90s
13) a zinc game
14) gtg?

[GGKoul]

One of the hard parts about bench marking MAME is it is a moving target.
Once you finished benchmarking every game a new version (or u update) will come out and you will need to start again :-(

For example Aaron is currently revamping the MAME sound system. And this will likely mean a performance hit for every game in MAME

http://aarongiles.com/weblog/archives/2005/02/speed_and_the_n.html

Yes, but if there is a script out there that would get the benchmark info automatically.

[SirPoonga]

right, you need to store stats for rom and version tested.

[elvis]

There are a few bench marks out there already... but like 2.6k said they are not updated frequently.


http://www.classicgaming.com/mame32qa/bench.htm

http://benchmark.mameworld.net/

Yeah, that second one is mine.  I must admit I haven't had the time lately to keep up on top of things, but hopefully that will change in a month or so.

I'm in the processess of developing a more automated way of benchmarking.  I've written a few scripts in linux that will benchmark every single game on a particular processor.  That's all fine and dandy, but pretty useless.  As mentioned above, a few examples of particular processors is usually enough for a broad overview.

Generally speaking you only need to benchmark games up to double FPS.  Once you can prove a game gets 120FPS on particular processor, it's safe to assume any processor in that family that is faster will run the game just fine.

I've also had a number of people submit high-end P4EE and Xeon benchmarks to me which I haven't added yet.  The idea was to show what high amounts of L2 cache does to MAME benchmarking, but again I've been busy and haven't had the chance to update.

Also when I do update, hopefully I can arrange the benchmarks in a more intelligent format.  Perhaps some database access with PHP will make life easier, and more searchable.  Maybe even something that users can submit to would be smart?

[GGKoul]

Elvis, I'll help you out with this project.

[SirPoonga]

Also when I do update, hopefully I can arrange the benchmarks in a more intelligent format.

[Chris]

S.T.U.N. Runner and either NBA Jam TE or 2on2 Open Ice Challenge are the games I usually use for benchmarks on systems between 600 MHz and 1 GHz.

[elvis]

Elvis, I'll help you out with this project.

That'd be neat.  I'll contact the mameworld kids and see if they've got mysql or other database we can use, and then get stuck into it from there.

[elvis]

Also when I do update, hopefully I can arrange the benchmarks in a more intelligent format.  Perhaps some database access with PHP will make life easier, and more searchable.  Maybe even something that users can submit to would be smart?

PHP and MySQL.  If you have a webspace that has MySQL and PHP I might help you out on that.
Or if you are on a windows machine it would be best done with asp.net and access.

To add.  You also need to have a standard set of settings.  Now this will changes are features are added or changed to mame.  At what version of mame the standards changed need to be noted.

There are optimization settings for mame that act differently on different machines.

If I was tesitng I'd just test with mame defaults for that version.

Maybe keep track of some optimized runs too.  But main goal being vanilla mame setup.

Woo!  Help coming in from all directions!

What I do is post a simple .bat file that folks can use to benchmark games.  Yes, this will need to be updated when feature flags are added to/removed from MAME.

I would really like for folks to be able to add their own entries directly.  Like a big public database.  That should really get the entries flowing, and mean there's plenty of data to search by for a particular game on particular hardware under a particular version of mame.

[SirPoonga]

I would really like for folks to be able to add their own entries directly.

[Fat_Trucker]

Great idea.

If you are going to present results in some kind of database it might be worth adding an amendment column at the end of the row, so that if benchmarks report problems with speeds in certain games in the current version of MAME you can add a comment at the end of the row stating which version of MAME handles it best (perhaps via feedback from the community).

That way there is no need to benchmark for every version of mame. You can simply benchmark the most recent version and update the comment in the amendment column to report any improvement/detriment with an older or newer version of MAME.

[SirPeale]

I don't think that benchmarking *every* version of Mame is worthwhile.  Perhaps better off to start @ .01 and work forward five versions at a time.

I recently tried Pac-Man on a P100, and it actually ran fairly well, about 45FPS.  I tried it both on an arcade monitor and a PC monitor, and there was negligible difference.  This was using Mame(tm) .90.  That's not too bad.  Galaga gave similar results.  Pacmania, OTOH, was 4 FPS.

[ray_slup]

maybe somehow have the program run as a screensaver
you know just run the game for awhile then switch to the next game
(that way people could use it with there frontends
or incorporated into frontends so you would get more users)
so then they wouldn't have to take the time to run the whole program(all the games) which might take awhile on older machines
  or maybe I don't know what the hell I'm talking about