Console dat and ini files..

[liquid8]

This topic has been brought up numerous times I'm sure - but I'd like to create some custom dat and ini files for consoles. What I'd like to accomplish with this at a minimum  is to have year, manufacturer/developer, category, number of players, and perhaps controls.

So a good starting point would seem to be the nointro dat files to start - then add year/manufacturer info into that. A separate catver.ini could be created with categories, and I guess an nplayers.ini could also be created.

I think a good way to get this going if there was a few people interested in helping out is to use something like google drive where you can create documents with multiple people being able to contribute all the info. This allows multiple people to enter information at the same time - or whenever time permits for them.

What's the issues or concerns with doing something like this other than the massive undertaking? Is there any starting point or someone that has tried to accomplish this?

[Howard_Casto]

You need to check a necro thread of mine in regards to reading header data out of roms. 

For anything snes and beyond, 90% of what you are looking for is found within the rom header.  That's how these console emulators show a little info message when the game is booting.  Sega is especially good at this... they even list the number of players and controller type and such. 

Now you have to decode some of it (lots of times you get a id number and not a readable string) but it's in there. 

I also warn you by asking you to look at controls.dat.  Yeah it's dead in the water because keeping track of that stuff is even more of a massive undertaking than you might realize.

I've got a little test program... it never went far beyond that, but you are welcome to it and more importantly the ini files which tell you where the data is. 

[Howard_Casto]

Of course the more important question is how to name it.  Unfortunately there are still multiple rom naming conventions for console roms and any ini you make needs a romanem as the key for the entry.  All the ini files are completely worthless if your roms are named differently. 

That is what I was TRYING to do with the header project... build a unique rom name from the header data, but trying getting people to change their habits... it's impossible.  On top of that some of the earlier consoles, well, they don't have headers so we don't have that option.

[liquid8]

Yep, so I found this thread by you that I will look through:
http://forum.arcadecontrols.com/index.php/topic,112732.msg1196526.html

That would be interesting and certainly time-saving if some of that can be found in there. If you still have that lying around, I'd definitely be interested in checking it out. As far as naming conventions go - I'm not looking to change that. I think no intro seems to be the most common standard for consoles now, right?

Ideally, I'd like to see a whole new info file format that frontends could use that could be dynamically expanded with even your own custom information added if you wanted. It's kind of annoying having multiple dat and ini files to manage all this info that pertains to the same set of games - but I don't think that's really worth the time at the moment since most frontends aren't being updated regularly if at all. Having this info available would be usable right now.

[Howard_Casto]

No not really.  There isn't a standard at all. 

Not that I'm suggesting that we all download roms illegally, but just for examples sake:

Let's say we download a torrent of console roms.  Well even if they have been ran through clrmamepro or a similar auditor they could be in one of the three formats (good tools, no intro or tosec).  This is because dats are available in every naming convention.  Now if I download a single rom from a particular site that always seems to come up when you google roms, it is undoubtedly in good tools format.  The hyperspin guys... they like no intro.  Too bad because the rest of the net with these vast image archives like either good tools or their own proprietary format (which is more and more leaning towards the internal cart id, thus the purpose of my now dead project). 

Tosec is almost dead at this point.  Good tools is for the most part the defacto standard, but everybody hates it and I can certainly understand why.  It's ugly afterall.    Good tools is what people want to be used, but honestly it has it's issues as well that I won't go into here. 

I'll see if I can find that for you tonight, but I'll need to do a little writeup or something.  It's an incomplete project afterall, so it isn't exactly user-friendly. 
 

[liquid8]

Well Good may be the 'norm' or not, but considering it contains all the bad dumps, hacks, etc.. I'd lean towards nointro for this type of project. The idea here is to have the descriptive (and more importantly filter-able) information for each "game", not necessarily every single rom.

In any case, many versions of roms would duplicate information ..  year, developer, category, nplayers, even history info.. I'm thinking perhaps more like a sort of 'fuzzy match' database:

Example:
nointro "ActRaiser (USA)", "ActRaiser (Europe)", "ActRaiser (France)"
good "ActRaiser (U) [!]"

might share:
entry (
   year 1991
   developer "Quintet"
   publisher "Square/Enix"
   category "Action/Adventure"
)

This way you could spit that info out in any format you needed - separate dat/ini/xml files for whatever set you prefer or one new info format with everything.. Any other standard could easily be added to match in the database, and new files could be generated off that.

[Howard_Casto]

Well then no intro isn't the naming convention for you then.  None of them are really because they don't separate the rom name from the game title and try to shoehorn a region code in that doesn't exist, making filtering difficult to impossible. 

Here's the thing that made me stop the project and the reason that none of these naming conventions are doing it properly.  You can NOT properly list the region in with the name.  Why?  Because first off the game is named differently depending upon the region and secondly which region are you referring to?  The rom region, the cart label region the box art region or the region code region, because they are all different things and yet all of these naming conventions seem to think they are group-able into one thing. 

Let's give an example just to try and explain it. 

How about Gyromite on the NES?  no intro lists it as "Gyromite (World)" good tools lists it as "Gyromite (W) [!]" Neither are really correct.  There is only one Gyromite rom in the world but it goes through various hardware adaptors and has different titles and different packaging in various regions.  Here in the states and in most parts of the world it's gyromite.  In japan it's Robot Gyro.  Even though you'll probably only find it as a world release, various versions were released throughout the world including a "big box" Scandinavian version. 

So what do you do in this case?  Make multiple copies of the rom and waste space... because if you use the file name as the game name that's what you have to do. 

Sega games are even weirder.  They are all region free and if different translations exist they are usually all included within the same rom.  For rom filtering this makes things real easy, but for naming artwork and figuring out the cart name it's really hard.

There are still rom filtering problems even ignoring the artwork/game name issue.  Say you want a full list of Games released in the US.  Well you need to add all the world games and then subtract any world games that also have a USA release.  I don't know of any FE that does that currently.  For sega games it's even weirder, because all games are probably going to have a world flag, and yet many of them weren't released in the US.

What I'm getting at is you need a roname=gamedesc relationship like mame uses.  Fuzzy logic would certainly be simpler, but front ends don't support it. 


Modern games have a rather convoluted header system that you would probably have to mimic if you wanted the list to be the end-all and be-all.

The Wii/GC/Wii U's is something like this:


[NRX1D]  --Disc Id (unique identifier, like a rom name)
Name-U=Super Mega Game
Name-J=Super Mega Game
Name-E=Super Mega Game
...ect....
Manufacturer=
Revision=
Year=

Names-U through Whatever represent the Various names of the title in different release regions.  Release region data isn't recorded (although you probably should) because they make the game universally and distribute it after the fact.  On the various packaging and labels, things are named after the disc id with a "-" plus the region code.  This solves both the artwork problem (Just search for NRX1D-U or what have you) and the game name problem (lookup Name-U) in the table. 

This sort of header is pretty much the universal standard for modern games because these companies found that having a bunch of different carts with the same IDs and often the same roms but with different product names and packaging made things impossible to keep up with.... so yeah, we are just cleaning up their mess.

CRC's/Sha's are a better unique identifier, but they have their issues as well.  First off some consoles have multiple dumping formats.....  z64/n64  sfc/sns/bns ect... so their crc's will vary even though it's the same rom.  Secondly crc checking is a feature that's seldom included in FEs... mostly because you have to unzip the rom first and that takes a lot of time (why rom verifying tools are unbearably slow). 

So I guess what I'm saying is before you jump in head first think it through.  It's more of a jumbled mess than it appears at first glance.

[liquid8]

All great points.. the fact of the matter is though, it doesn't have to be perfect. All these formats and versions and so forth.. the information I'm talking about putting together applies to most of the different versions/formats/naming conventions for the same "game". Of course things will vary (year for instance might be different depending on when it was released in a certain region).. but the category, the description, the number of players.. applies to every version.

No intro to me would be the best option because there is not as many rom variations, i.e. less entries to maintain at the start. As you said, they all use the rom name as the identifier so that is what you have to work with. I just want to see a starting point since the info doesn't exist.

Check out:
http://superfamicom.org/info/chrono-trigger

This is a great site for SNES (and some NES info) - it has most possible identifiers and naming conventions. So my suggestion would be to blanket all these together, something like:

Code: [Select]

entry {
match:
           id: SNS-ACTE-USA, SHVC-ACTJ-JPN
           internal: CHRONO TRIGGER
           good: Chrono Trigger (U), Chrono Trigger - Kurono Toriga (J)
           nointro: Chrono Trigger (USA), Chrono Trigger (Japan)
   zap: Chrono Trigger (NTSC)(Eng)(1.0), Chrono Trigger (NTSC)(Jap)(1.0)
year: "1995"
category: "RPG"
developer: "Square"
desc: "Time traveling, butterfly effecting RPG."
nplayers: 1P
}

That one entry now provides the info I'm talking about for 6 (or potentially more) 'unique' roms. If a file (or database) is put together similar to that, it could then be parsed and a script can spit out the necessary files for whatever set you need. A file like that would also help if a frontend choose to use it directly to manage different conventions across the board. Technically you could expand on that and provide region specific information in the same entry if necessary.

[Howard_Casto]

Ok I was going to include a write up but I decided to do it another way.  I'm attaching the program along with the source code and all my notes. 

You browse to a rom (unzipped) and press the parse button.  It'll then spit out all the info it can find from both the internal header and some crc-based game lists and id lookup tables I've included. 

I've pretty well mapped out the following consoles:
nes
snes
n64
ngc
virtual boy
gb
gbc (incomplete)
gba

sms
genesis/mega drive
32x
gamegear
sega cd
dreamcast

neogeocd (not really necessary since mame supports neogeo carts so well, but I did it anyway)
turbografx cd
psx (only certain storage formats and real cds)
ps2 (ditto)

Anything older than the nes doesn't have any headers or reliable gamelists to read.  Some oddball consoles like the turbo grafx don't have header info either. 

You'll note on the data printout that some data like "official game name" is weird or wrong.  That was the part of the project where I was going to manufactuer a rom name and thus the info isn't complete. 

This project isn't complete, there are probably bugs... it might not be fully useful ect... but there is a TON of data in there ready to use, particularly the ini files in the header section that basically map out the headers for the more popular consoles.

NOTE:  There are extra buttons in the program.  They don't do anything so don't use them!

[liquid8]

Thanks Howard.. this is great.

So just to clarify - what I'm trying to do here is create a console info database - but avoid creating an entry for every single unique rom - since a number of them share info like description, year, genre, etc... I'm proposing to store info by matching it with a number of rom naming conventions.

Your suggesting to use the internal id as the 'link' between rom variations - so they all have unique internal ids and those would match in any modified roms like hacks, translations or bad dumps, right?

The thing about using ids is that goes out the window if you do want to support older or "oddball" consoles. That's why I was getting at linking info by the rom names. You could still include the internal ids if they are available as yet another way to find the info. You could also include any matching crcs (multiple) if you wanted to validate the info to a specific file.

The two main things I'm struggling with:

1) How to store and relate this info in a database - the db will have to include all possible entries (nointro, good, tosec, etc.. entries) and be updated as the dat files are
2) How to deal with region differences - where the data might all match except one or two pieces of info (like year for instance). Again, I don't want separate entries for every single rom.

Once the database is setup, a web app can be made (already played around with this) that allows for input and export - it could spit out the data in any format - console.dat catver.ini, nplayer.ini, etc.. or a single file to include all info. As a bonus (and since I want one anyways) - this could be used to export frontend list files with filtering options (region, categories, etc) in place.

Here's a mock of what an entry might look like:

[Howard_Casto]

Well older roms have some things going for them.  The primary thing being that they are tiny.  This means that there aren't multiple dumping formats and that people would probably be willing to keep them in an archive unzipped as the whole set might be a couple of megs tops.  Like the NES games in the tool right now... I'm using a crc lookup for them which links them to their Nintendo cart ide which in turn links them to their full name and ect. 

So for older stuff crc is the only definitive id I can think of. 

This should be ok though.  Most of your consoles without IDs have a low game count and/or have tiny games.  Like the colecovision for example.  It's got around 230 games and that's it... INCLUDING homebrew.  Each game is only around 10kb.  So the entire set is only around 2.3 megs.  That means it's really easy for you to scan the crc and use that as a unique ID and if a naming convention is necessary, the end-user can download an entire set named properly for your dat if need-be. 

It's when we get around the NES era when the file sizes become too big to expect the end-user to just download/rename a new set.


In terms of the oddball consoles there are only two I could see you having issues with... the 2600 and tg-16.  The tg16 doesn't have any header system that I'm aware of and it supports everything from hucards to an arcade expansion board ect...  Fortunately the gamelist is pretty small, so you can probably do something with it manually. 

The 2600's library on the other hand is a doosie.  The roms are really tiny but the library is really huge.  There IS a id system in place, but it only applies to select manufacturers, mostly Atari and Activision.  There are also so many duplicate roms due to the rampant over-publishing in those days.  It just makes things confusing.  I think the stella guys have a pretty good comprehensive list going though. 

That's the thing, many of these older/quirkier consoles have a rather strong cult following.  You'll find that while you need to re-format the data, they've already collated a list for you, often with the rom crc included. 

I mean the psx gamelist for example.  Super easy to identify the disc.... there is a file at root of the disc and it's named after the ID.  I go to a psx fanboy site and they have a database posted of all known psx games and their corresponding disc ids. 

As for diverging info, the only way you can handle that is if you setup a parent/clone relationship like mame does.  Either that or do it like that Wii example I posted earlier where each region has it's own optional slot. 

[liquid8]

So still playing with ideas on how to accomplish this in the long run. In the meantime, I'm building a basic filter that will be used in the web app...



This will let you add a number of rules to filter the list, such as name contains Zelda or genre does not equal Platformer. You will be able to add multiple rules to filter the results - and since the rom names are stored in unique db tables, you can filter using different sets. These rules are also what will allow exporting of game lists based on the filters, especially once the info is in the db.

Also, I noticed that there is actually a goodmerge.. I haven't used it, but I'm curious what they use to determine which roms would be merged together. That might be a good starting point that I will look into.

[liquid8]

OK, so looking at goodmerge, it actually uses a number of regex/clone rules to determine what to group together. You can take a look at the attached file to see the rules it's using.. I added 'regex matches' to the filter tool above, which will be handy.

It also keeps a list in there of 'clones' and 'zones' where games have different names or region based names, so Castlevania for instance is:
<zoned><bias zone="U" name="Castlevania - Dracula X"/><bias zone="E" name="Castlevania - Vampire's Kiss"/><bias zone="J" name="Akumajou Dracula XX"/><clone name="Castlevania - Dracula X - Evil Trevor V.99"/></zoned>

It could be good to go off of this for name-based matching. When grouping them together in a single file, I could link all the names to the internal id for systems that support it.

[liquid8]

So looking a little more at storing this, something like...

Code: [Select]

    <games>
        <game id="CHRONO TRIGGER" nplayers="1P" genre="Role-Playing" developer="Square" publisher="Square" description="Description goes here" keywords="">
            <sets>
                <set id="SNS-ACTE-USA" region="USA" year="1996" crc="2D206BF7">
                    <rom romset="nointro">Chrono Trigger (USA)</rom>
                    <rom romset="good">Chrono Trigger (U)</rom>
                    <rom romset="zap">Chrono Trigger (NTSC)(Eng)(1.0)</rom>
                </set>
                <set id="SNS-ACTJ-JPN" region="JAPAN" year="1995" crc="4D014C20">
                    <rom romset="nointro">Chrono Trigger (Japan)</rom>
                    <rom romset="good">Chrono Trigger - Kurono Toriga (J)</rom>
                    <rom romset="zap">Chrono Trigger (NTSC)(Jap)(1.0)</rom>
                </set>
            <sets>
        </game>
    </games>

How's that look? Of course other roms can be added that would match this "game".

The idea here is:

• Game encapsulates any matching game and contains the shared attributes of all roms within.
• Set encapsulates a group of these roms, probably by region and the attributes could be overridden or additional attributes can be added.
• Rom holds the actual rom name, and could also override attributes or additional attributes could be added.

Now I'm just trying to decide how this will be stored in a db.

[Howard_Casto]

I'm not sure if you would actually want the advice I would give.  SirP and I would argue over formatting on controls.dat constantly.  Some things to this day we still don't agree on. 

I'm not a huge fan of xml, for any reason.  It's slow to parse, not easily readable by human eyes and contains a crazy amount of white space, making subsequent files huge.  At the time, the reason we switched over to xml had more to do with the fact that we ran into a logical limit  in regards to how large a file the getPrivateProfileString api call could load on win 9x machines.  Obviously this is no longer a concern, so my vote is always ini format. 

Flash forward to the present and mame's xml file is larger than some people's rom collections and takes a minute or so just to generate, much less parse.  So I guess my concerns were valid ones. 

That aside the format looks good.  I'm not a huge fan of nesting and doubled key names though.  It means the parser has to keep trying to find nodes instead of looking for specific ones.  I'm also not sure why you wish to group different roms together like that though.  As-is you are also ignoring the possibility of the same rom having a different name, depending upon the region.  This is could be an issue once you get to sega games, because often different title screen graphics and in-game text are all stored on the same rom.

[liquid8]

Any advice is good, I'm not necessarily looking for approval  To address your issues though:

The formatting is only a formality
I'm using a mysql database to keep the data, so it could be exported to whatever format. My main goal is to be able to export to ini/dat files that current frontends can use now, so it can be exported to multiple formats. I agree about XML, I also played with json formatting. Even though I am not a huge fan for readability purposes, you can minimize the file and it doesn't have the need of all the tags.

Nesting could slow down parsing
Again though, it will be exportable to whatever format you needed, so not necessarily all the data will be included in the resulting file(s). You can choose good set and it will export the info into inis/dats with only good set info.

However, I would like a frontend to be able to use a single file if it wanted... I'll play around with the actual format to see how well it might work as a file on its own. I choose this format because it can share attributes for a large or small group of duplicate games, but if necessary can override attributes all the way down to a specific rom. If you can see a way that could be done in ini format, let me know.

Different named roms will be grouped together in a game
I noticed goodmerge has a pretty good listing to go off of for this, so those would definitely be grouped together. Any names could be added in, or separated as needed.