Software Support > GroovyMAME
Linux-command-line-tools for mame-gamelist-processing
Lomaxx:
I've been playing around with simple command-line-tools in Linux in order to get a list of games using a certain video-resolution. There are probably more efficient ways of doing this, but I'm not very experienced with other solutions (like gawk, pearl/python-scripting) mentioned on the net for this issue, so i stuck mostly to "(e)grep", "sed" and my favourite editor "vi".
I could simply provide an already processed list, but since the gamelist of mame changes as time goes by, I decided to write down the things I did for those of you who might do something similar. This seems to be rather long, but that's only due to the fact that I try to explain everything in detail and noticed that not everything worked out as I planned it in the beginning. So in the end this became sort of a diary.
First of all i wrote the output of the full xml-list of groovymame to a file:
--- Code: ---groovymame -listxml> groovymame_output.xml
--- End code ---
The resulting file is rather big and I didn't want all information provided, so i used "egrep" to only pick out some of the lines:
--- Code: ---egrep -o "<game\ .*|<description>.*|<year>.*|<manufacturer>.*|<display .*|<input.*|<control.*|<driver\ .*|</game>" groovymame_output.xml > groovymame_output_partial.txt
--- End code ---
The "-o"-switch tells egrep to only output the exact pattern that was found and not the whole line that contained the pattern. In this example the difference is that the leading indention is removed.
In order to verify if the results a little, I tested how often these sections are found, which can be done with "egrep -c" ("-c" just counts the number of hits)
--- Code: ---egrep -c "<game name=\"" groovymame_output_partial.txt
10630
egrep -c "<description" groovymame_output_partial.txt
10630
egrep -c "width=\"[[:digit:]]*\" height=\"[[:digit:]]*\"" groovymame_output_partial.txt
9479
--- End code ---
While the number of hits for the short name and the full description was exactly the same, the number of hits for the resolution differed. I looked at the file again and noticed that some games do not list the resolution. Seems like these games are not yet supported but were already added for future releases of mame since their status is set to "preliminary".
More surprisingly some game-entries contain two <display>-tags. Some of them seem to consist of some dual-monitor setup, but as far as I can make out not all of them do. For example try
--- Code: ---grep -B 1 -A 50 "Ghouls'n Ghosts" groovymame_output.xml
--- End code ---
Sidenote: After creating the file "groovymame_output_re-lined.txt" (Read on a few lines to see how this is done) you can use the following command to get a full list of the 214 games containing two "<display"-lines:
--- Code: ---grep "<display.*<display.*" groovymame_output_re-lined.txt | grep -o "<description>.*</description>"
--- End code ---
As far as I noticed there are three types of statuses: "good", "imperfect", "preliminary". Additionally there are also clones listed which you might want to sort out as well.
But before I continue I will reformat the text-file in such a way, that all data-fields of each game are listed in one line (one line per game).
--- Code: ---tr -d '\n' < groovymame_output_partial.txt | sed -e 's/<\/game>/<\/game>\n/g' > groovymame_output_re-lined.txt
--- End code ---
This command-line consists of two commands: The "tr"-part removes all line-breaks and the "sed"-part adds one single line-break after each "</game>".
Now we can search for lines that for example contain the status "good" or "imperfect" and are no clones and redirect them into a new file
--- Code: ---egrep "<driver status=\"(good|imperfect)\"" groovymame_output_re-lined.txt | grep -v "cloneof=" > groovymame_output_selection.txt
--- End code ---
For me this resulted in a list of 3493 games. I'm a bit surprised that there are so many clones, which makes me a bit unsure that it's recommended to sort them all out. Can somebody maybe comment on this? Do clones sometimes yield advantages over the original versions? Also: Is it not too strict to sort out all preliminary games? Here are some statistics from my grep-results:
--- Code: ---cat groovymame_output_re-lined.txt | wc -l
10630
egrep "<driver status=\"good\"" groovymame_output_re-lined.txt | wc -l
6079
egrep "<driver status=\"imperfect\"" groovymame_output_re-lined.txt | wc -l
1478
egrep "<driver status=\"preliminary\"" groovymame_output_re-lined.txt | wc -l
3073
egrep -v "cloneof=" groovymame_output_re-lined.txt | wc -l
5382
--- End code ---
Unless I'm doing something wrong here, this means that half of the games are clones. Anyway, I will continue with the above specified selection of "groovymame_output_selection.txt".
We can now use this selection as a good source to do faster, further processing of our choice. For example I wanted to get a list of all resolutions that games use. Here is the command-line i used:
--- Code: ---egrep -o "width=\"[[:digit:]]*\" height=\"[[:digit:]]*\"" groovymame_output_selection.txt | sort -ru
--- End code ---
This line searches for lines containing the pattern
width="" height=""
with any amount of digits between the quotation-mark. The output of the egrep-command is then handed on to the "sort"-command, which because of the "-ru"-switch sorts the lines in reverse order and removes any duplicate lines. The sorting is not yet perfect, but forget about that for now.
You will get a list of ~206 resolutions. Browse the list a little and notice that there are some very low and rather high resolutions: 22, 64, 1024, 1280
I was curious what these games are about, so I built up a line that finds the description of these games:
--- Code: ---egrep "(width|height)=\"(1280|1024|22|64)\"" groovymame_output_selection.txt | egrep -o "<description>.*</description>"
--- End code ---
This line searches for lines containing "width"- or "height"-values of the above mentioned, suspicious resolutions.
The whole egrep-output is then handed on to another egrep-command that searches for "<description>.*</description>". The result is a compact list of the long name of all the games with the suspicious resolutions. I looked these games up and as far as I can tell , I can safely forget about these and concentrate on games with width- and height-values with exactly three digits.
Now lets list all resolutions again with this 3-digit-limitation:
--- Code: ---egrep -o "width=\"[[:digit:]]{3}\" height=\"[[:digit:]]{3}\"" groovymame_output_selection.txt | sort -ru
--- End code ---
This time the remaining resolutions are correctly sorted. You can as well sort for height if you want to:
--- Code: ---egrep -o "width=\"[[:digit:]]{3}\" height=\"[[:digit:]]{3}\"" groovymame_output_selection.txt | sort -ru -k 2
--- End code ---
Another example of what you can do is to search for games up to a width or height of "288", which is of interest to me because that's the resolution I can use in progressive modes on my scart-tv. This time I will go for the short name and not the full description and redirect the output into some file:
--- Code: ---egrep "height=\"[12][0-8][0-8]\"" groovymame_output_selection.txt | egrep -o "name=\"[^\"]*\"" > up_to_288_games.txt
--- End code ---
--- Code: ---egrep "width=\"[12][0-8][0-8]\"" groovymame_output_selection.txt | egrep -o "name=\"[^\"]*\"" >> up_to_288_games.txt
--- End code ---
That list will contain duplicate entries, which can be easily removed. Also I wanted to have nothing but the name itself (no name= and no quotes):
--- Code: ---cat up_to_288_games.txt | sort -u | cut -b 7- | tr -d \" > up_to_288_games_cleaned.txt
--- End code ---
And to see the number of games:
--- Code: ---cat up_to_288_games_cleaned.txt | wc -l
3121
--- End code ---
I didn't fully verify the result yet, but it should be correct to the most part of it.
Another list I wanted to have was one of only vertical games that fit the screen of my TV better when it's rotated by 90°. First I thought I can break it down to the resolution: a vertical game should be one, that has a bigger "height" than "width". But that's not right. I started to realize this when looking at games with the same height and width. For example for 480x480 these are "Crate Raider", "Spy Hunter" and "Turbo Tag". When looking at these games at mamedb.com, then you will see that "Spy Hunter" and "Turbo Tag" are vertical games, while "Crate Raider" is a horizontal game. I looked at the xml-data of these games and noticed that "Crate Raider" has a rotate value of "0", while the other two have a value of "90". So is that the key to find vertical games? To be honest I have been in doubt that this is correct for all games.
I took a look at the game "Raiden". I know that this is a vertical game. The resolution of the game is 256x224. Again a proof that vertical games do not necessarily have a greater height-value than width-value. The rotate-value however is "270".
I decided to create a list of all games that have a rotate-value of "90" or "270"
--- Code: ---egrep "rotate=\"(90|270)\"" groovymame_output_selection.txt | wc -l
829
egrep "rotate=\"(90|270)\"" groovymame_output_selection.txt | grep -o "<description>.*</description>" > groovymame_output_rotated.txt
--- End code ---
I compared a couple of the games with mamedb.com and the results seemed to be rather satisfying. I couldn't make out any horizontal game. However I am not sure if some vertical games are missing.
Remember that my file "groovymame_output_selection.txt" already sorted out all games with preliminary status and that were no clones of some other game. So the last file we created should contain only "good" and "imperfect" emulated vertical games without clones. If you want a purified list without the start- and end-flag of <description>, then use sed to cut these tags of:
--- Code: ---sed -i -e 's/<description>//' -e 's/<\/description>//' groovymame_output_rotated.txt
--- End code ---
Be careful with that "-i"-switch of sed. It stands for "inplace" and makes sed directly and permanently modify the specified file. Leave "-i" away and you can test the result as it will be shown on standard output. Or redirect the output to a new file like we did before.
Here is a slight variation in case you want to have the same list of games as zip-archives:
--- Code: ---egrep "rotate=\"(90|270)\"" groovymame_output_selection.txt | egrep -o "name=\"[^\"]*\"" | cut -b 7- | sed -e 's/\"/\.zip/' > vertical-games-ziplist.txt
--- End code ---
I tried more or less the same search with "rotate"-values of "0" and "180" and received 2653 games. Together with the above 829 games this gives a total of 3482 games. However "wc -l groovymame_output_selection.txt" shows a total of 3493. What's up with the 11 missing entries?
--- Code: ---egrep -v "rotate=\"(0|90|180|270)\"" groovymame_output_selection.txt | egrep -o "<description>.*</description>"
--- End code ---
Except for "Gamball" and "Dr.Who The Timelord" all these entries seem to be system-roms. I tried to run both games (m_gmball, m_bdrwho) which resulted in a segmentation fault of groovymame. So my decision was to forget about both games.
Edit: Skip the rest of this post. I made a dumb mistake which made me thought that the results are incorrect, but as far as I can tell by now they are pretty much fine.
Now back to the results of games with a rotate-value of "0" and "180": Already the very first games I looked up in the mamedb, showed that there are also vertical games among them. So my doubt was justified. Unfortunately I couldn't make out any more complex filtering that would help. For example a rotate-value of "0" and "180" AND higher/lower width-value than height-value didn't work as well.
Some games list a "htotal"- and "vtotal"-value, maybe that would help, but unfortunately only 1385 (of 3493) games list these values. So even in case they would help, this wouldn't bring up a complete list.
At the moment I am clueless. And since all this is becoming a quite long text, I decided to release it for now. Maybe someone got an idea. Feel free to ask any questions - as you might have noticed I somewhat enjoy stuff like this.
Lomaxx:
I made a very stupid mistake while verifying the results of the gameslists for vertical/horizontal-games: While browsing the mamedb.com-pictures I only looked at the thumbnails, which always seem to be vertical oriented. Once you click into the actual game-info there, you will see the proper screenshot. Just realized this. Doh!
So there is still hope that the rotate-value is a proper way of sorting them out. I will make a more intense verification of that tomorrow and report back.
Paradroid:
Fascinating! Thanks for documenting this and posting your findings. :)
I've been thinking along very similar lines although I would use different tools to reach the same outcome (sorry, still a Windows guy for now... ;) ).
My idea is that if you know what the most common (or important... although this is subjective, I guess) resolutions used by MAME are, then you can make sure your GroovyMAME monitor specs and monitor setup are focused on these.
For example, what's the most common arcade game resolution? I have know idea. If you're able to pull this information from MAME and do some processing then that should be simple to work out.
There are lots of possible uses for this information. I'm definitely keen to see where you head with this...
Lomaxx:
Thanks for the feedback, Paradroid.
Some updates:
- First of all i changed the thread-topic a little, since i thought it hasn't been very accurate.
- Then I forgot to mention that the long game-names (the one in between the <description>-flag) show the "&"-character as "&". If you want to replace them with "&" again, then you can do it with "sed". An example-line which would remove "<description>","</description>" and exchange "&" back to "&" is:
--- Code: ---sed -e 's/<description>//' -e 's/<\/description>//' -e 's/\&\;/\&/g'
--- End code ---
You could add this line behind some egrep-command of your choice seperated by a "|" (that's called a "pipe"). Also see the examples in the first post to understand how to pipe output into "sed". Alternatively you can specify a file at the end and/or redirect the output into a new file.
- Most importantly of all: I again compared the first ~100 entries of the vertical- and horizontal-game-list with mamedb.com - this time correctly I hope. :P The result was pretty satisfying, although it's no 100% proof of correctness. For quite some games no screenshot was available, others were not listed at all and sometimes only a ingame-/title-screenshot was shown but no shot of the cabinet with the monitor (which is after all the most reliable proof). But at least i couldn't find any clear examples of mistakes.
I will edit the above post in order not to confuse anyone.
@Paradroid: A sort of statistic of which resolution is used how often shouldn't be too hard. I will look into that.
Lomaxx:
With a little help from the net I found a way to get a statistic of how often each resolution is used rather fast. First I tried to do it with the "-c"-switch (stands for "count hits") of grep. But that only counts all hits. I could have written a bash-script that calls "grep -c" for each search-pattern, but gladly I stumbled upon a far easier solution.
Remember the egrep-line that I used before to get a sorted list of all used resolutions? Well here it is again:
--- Code: ---egrep -o "width=\"[[:digit:]]*\" height=\"[[:digit:]]*\"" groovymame_output_selection.txt | sort -ru
--- End code ---
All we need to do is to remove the "-ru"-switch from sort ("r" could be left as it simply does a reverse sorting, but "u" needs to be removed as it does the same as "uniq", i.e remove double entries) and pipe the output into the "uniq"-command which has an option to count how often each duplicate line appeared. After that I sort the result once again.
--- Code: ---egrep -o "width=\"[[:digit:]]*\" height=\"[[:digit:]]*\"" groovymame_output_selection.txt | sort | uniq -c | sort
--- End code ---
Here is the result for quick reading:
--- Code: --- 592 width="256" height="224"
438 width="320" height="224"
322 width="256" height="240"
304 width="320" height="240"
193 width="288" height="224"
166 width="640" height="480"
130 width="384" height="224"
77 width="384" height="240"
74 width="512" height="224"
62 width="256" height="256"
61 width="336" height="240"
48 width="384" height="256"
47 width="640" height="240"
46 width="512" height="240"
42 width="416" height="224"
42 width="320" height="200"
41 width="240" height="240"
38 width="260" height="224"
36 width="512" height="256"
35 width="352" height="240"
34 width="512" height="288"
34 width="256" height="192"
30 width="240" height="224"
26 width="512" height="480"
26 width="256" height="232"
21 width="320" height="232"
20 width="304" height="224"
19 width="399" height="253"
17 width="288" height="216"
17 width="240" height="248"
16 width="671" height="216"
16 width="448" height="224"
15 width="496" height="384"
15 width="256" height="248"
14 width="488" height="384"
14 width="384" height="232"
14 width="304" height="216"
13 width="544" height="480"
13 width="512" height="384"
13 width="432" height="224"
13 width="240" height="192"
12 width="272" height="224"
11 width="400" height="256"
11 width="336" height="239"
10 width="480" height="64"
10 width="464" height="248"
10 width="360" height="240"
10 width="292" height="240"
10 width="256" height="208"
9 width="400" height="300"
9 width="400" height="280"
8 width="508" height="240"
8 width="280" height="210"
7 width="703" height="272"
7 width="512" height="400"
7 width="400" height="240"
6 width="400" height="224"
6 width="320" height="256"
6 width="304" height="240"
5 width="704" height="480"
5 width="576" height="224"
5 width="512" height="236"
5 width="400" height="248"
5 width="338" height="240"
5 width="336" height="225"
5 width="240" height="256"
4 width="384" height="384"
4 width="380" height="224"
4 width="368" height="240"
4 width="360" height="224"
4 width="280" height="224"
4 width="276" height="240"
4 width="248" height="256"
3 width="672" height="240"
3 width="576" height="432"
3 width="576" height="400"
3 width="512" height="448"
3 width="496" height="480"
3 width="480" height="480"
3 width="432" height="256"
3 width="384" height="280"
3 width="352" height="296"
3 width="352" height="256"
3 width="294" height="294"
3 width="294" height="238"
3 width="288" height="240"
3 width="280" height="240"
3 width="256" height="216"
3 width="248" height="240"
2 width="720" height="768"
2 width="508" height="384"
2 width="508" height="254"
2 width="464" height="224"
2 width="394" height="240"
2 width="376" height="240"
2 width="368" height="224"
2 width="336" height="238"
2 width="320" height="190"
2 width="318" height="239"
2 width="272" height="236"
2 width="256" height="239"
2 width="256" height="230"
2 width="236" height="224"
2 width="224" height="224"
2 width="1280" height="1024"
2 width="1024" height="1024"
1 width="800" height="600"
1 width="544" height="242"
1 width="512" height="232"
1 width="512" height="228"
1 width="512" height="192"
1 width="511" height="399"
1 width="508" height="224"
1 width="496" height="232"
1 width="496" height="224"
1 width="480" height="464"
1 width="480" height="224"
1 width="448" height="240"
1 width="392" height="224"
1 width="376" height="250"
1 width="376" height="224"
1 width="368" height="232"
1 width="366" height="240"
1 width="360" height="245"
1 width="352" height="224"
1 width="342" height="240"
1 width="338" height="236"
1 width="335" height="240"
1 width="322" height="241"
1 width="321" height="224"
1 width="320" height="248"
1 width="320" height="236"
1 width="319" height="255"
1 width="319" height="240"
1 width="319" height="223"
1 width="318" height="240"
1 width="316" height="239"
1 width="304" height="256"
1 width="304" height="232"
1 width="296" height="480"
1 width="296" height="240"
1 width="292" height="231"
1 width="288" height="208"
1 width="280" height="232"
1 width="272" height="200"
1 width="264" height="240"
1 width="264" height="224"
1 width="260" height="240"
1 width="256" height="288"
1 width="256" height="234"
1 width="256" height="231"
1 width="256" height="222"
1 width="255" height="232"
1 width="254" height="240"
1 width="248" height="224"
1 width="240" height="252"
1 width="240" height="232"
1 width="240" height="216"
1 width="232" height="224"
1 width="223" height="240"
1 width="208" height="208"
1 width="192" height="22"
1 width="192" height="192"
1 width="184" height="176"
1 width="160" height="200"
--- End code ---
Again you could sort the result by the content of second or third "column" by using the "-k 2" or -k 3"-switch of sort.
Navigation
[0] Message Index
[#] Next page
Go to full version