Calling any interested coders: rom management...and something more

[Gray_Area]

Tonight while looking at my missing sets in current/recent MAME, I was thinking about rom managers. Then I came across the following:

Btw, 145 is causing all sorts of havoc with the rom managers as well.  Samples are now available in flc flavor (again, more compression) and because of it romcenter is having a kiniption fit.  Clrmampro seems to be having similar problems with some of the list xml changes due to recent EM additions. 

I'm having to bounce back between the two to rebuild my set. 

So, it really seems that a new........and much simpler......app needs to be created for this. Something like the following:

- you click on a button called  'exe' or 'target' or 'target exe'

- it automatically creates a profile

- you click another button called 'dir' or 'directory' or 'target directory'

- you press another button called 'fix'

And, hey, all your romz is correct!



What would be the icing on this cake, would be a feature that connects to your preferred source (you know, that T word!), and automagically adds the files you need, and only those!!


Now who thinks this can be done [relatively easily], and has the inclination to??

[Howard_Casto]

MAME/MESS actually have built-in rom verification, but I don't think that any of the rom managers take advantage of it.  It might be due to the excessive command-line launching, but to me, using the internal system ensures that the roms are what they say they are.

Run:

mame.exe sf2 -verifyroms

And if you've properly setup your paths in your mame.ini and your rom is correct you'll get the following message:

romset sf2 is good
1 romsets found, 1 were OK

Run it on a game with an issue (in my case a ddr game) like so:

mame.exe ddr2ml -verifyroms

And you'll get the following error messages:

ddr2ml      : ge885jaa.u1 (548 bytes) - NEEDS REDUMP
ddr2ml      : 885jab01 - NOT FOUND
romset ddr2ml [ddr2m] is bad
1 romsets found, 0 were OK.

This tells us that the romset is bad, that one of the files needs a redump (you can't do anything about that) and that the other is missing.  So basically it tells you what is wrong and what file you need.


Now obviously that isn't practical for all 18,000+ roms but luckily if you omit the rom variable, mame will go ahead and verify EVERYTHING.

It'll spit out a list something like this:

romset fb2nd is good
romset fb2ndc2 [fb2nd] is good
romset fb2ndd1 [fb2nd] is good
romset fb2ndd2 [fb2nd] is good
romset fb2ndo [fb2nd] is good
romset fb2ndv1 [fb2nd] is good
romset fb2ndv2 [fb2nd] is good
romset fb3g is good
fb4         : fb4rom2.bin (262143 bytes) - NEEDS REDUMP
romset fb4 is best available


It bascially summarises the output messages.... if a romset is ok it tacks on a "is good" at the end.  If it's bad it'll tack on a "not found" or "is bad" at the end. 

If it runs across an improperly dumped set, it'll warn about needing a redump, listing all the offending files, but at the end it will give a "is best available" letting you know that there isn't anything wrong with that rom. 


There is a similar command line switch for verifying samples btw....


So at least for determining what is wrong, MAME will do it for you.  The problem is what to do after that.....  You'll have to search for files via crc checksum, NOT names as mame roms constantly get re-named.  Then you have to actually get the missing ones from somewhere and zip em up.  Of course verify roms doesn't really check the names, so you'll have to do a second pass with listxml if you want everything named correctly. 


Long story short, figuring out what is wrong has never been the issue... fixing it has.  It's complex and slow to search through a bunch of zip files, check crcs and unzip/rezip stuff. 

[demeth]

MAME/MESS actually have built-in rom verification, but I don't think that any of the rom managers take advantage of it.  It might be due to the excessive command-line launching, but to me, using the internal system ensures that the roms are what they say they are.

You don't even have to parse the output, mame will set the exit-code depending on if it was a success or not.
from <arcanfe/resources/scripts/importers/mame.rb>

Code: [Select]

def checkrom(mamepath, rompath, romset)
 a = IO.popen("#{mamepath} -rompath #{rompath} -verifyroms #{romset} 1> /dev/null 2>/dev/null")
 a.readlines
 a.close
 
 return $?.exitstatus == 0
rescue => ex
 false
end

so something short (untested) like;

Code: [Select]

Dir["rompath/*.zip"].each{|a| b = IO.popen("/where/is/my/mame -rompath #{rompath} -verifyroms #{a}")}.readlines
 b.close
 File.rename(a, "broken/#{a[a.index('/')+1..-1]") if $?.exitstatus == 0
}

Would move all non-working romsets to a different dir. But that's, as howard said, the easy part.  ;-)

[spoot]

Current application I'm working on is slowly getting rom "fixing/auditing" features.  It will crc32 audit 47k rom/zips over my network in about 10 minutes.  It doesn't currently rename files, etc....but will shortly.  SHA1 calc will slow it down a chunk I spose, but it's currently threaded to support multiple cores so hopefully won't be too bad.

Kinda funny I added a utility two weeks ago to convert 7z to zip as mame/mess don't support it .  Then I find out that 7z is being added very soon so I'll have to add an option to batch convert zip to 7z.   

[Gray_Area]

Mainly, I'm interested in the automation feature of filling sets without having to be a pokerom. You wouldn't need to worry about re-naming, or missing files, or anything. You'd just hook up the feed bag, and wait. They have software doing it at Emumovies for everything....except, you guessed it.

[Howard_Casto]

Well the problem with that idea is you just created a rom downloader, which is highly illegal.  Nobody wants to be the Napster of emulation. 

[Gray_Area]

Well the problem with that idea is you just created a rom downloader, which is highly illegal.  Nobody wants to be the Napster of emulation. 

I don't buy that. Any torrent client can download roms.

[Dazz]

Well the problem with that idea is you just created a rom downloader, which is highly illegal.  Nobody wants to be the Napster of emulation.  

I don't buy that. Any torrent client can download roms.

You don't buy that??  What?  That's exactly what you are asking for...

[Gray_Area]

Well the problem with that idea is you just created a rom downloader, which is highly illegal.  Nobody wants to be the Napster of emulation.  

I don't buy that. Any torrent client can download roms.

You don't buy that??  What?  That's exactly what you are asking for...

Perhaps you're having trouble understanding my english: any torrent client can download roms.

What I'm asking for is an app that would download and replace only the contents of your directory.

[Loafmeister]

I don't think you are understanding what you are asking

[demeth]

Perhaps you're having trouble understanding my english: any torrent client can download roms.
What I'm asking for is an app that would download and replace only the contents of your directory.

However, ANY torrent client can download ANYthing. Bittorrent was initially used to offload the distribution of Linux isos, a perfectly reasonable approach given the hosting and funding situation at the time. He could (and did) argue that what the nasty users did was none of his concern. Hence, it was never created for the expressed purpose of infringing copyright, but that would be the case here; There's no gray area to speak off, it's pitch black -- hence, putting the developer at considerable risk. Emulation (given the current legal precendence) survives on the compatibility/curatorship role -- we need it to save outdated software requiring outdated or extinct hardware, and even that won't save you in every country. Thus, when it comes to satisfying some inherent need for collecting and managing every single little rom out there, you're on your own.

[Gray_Area]

Well, anyways:

The initial content of this thread had to do with rom management. Everyone got side-tracked on the latter *BONUS* bit. So, why can't there be a three clicks and you're done manager?

[qwench]

Maybe because it is over complicated ?
Many many cases to handle :
roms merged ? split ? unmerged ?
bios as well ?
and samples ?
chd in folder ? in zip ?
How to identify samples (they have no crc and no size) ? by their name ? confusing, or extension ? wav ? and now flc, mp3 and more ?
And identifying chd ? calculating sha-1 on the xGb files ? too long, getting it on the header ? which version ? v1 ? 2,3,4,5,... ?
Roms in a folder ? in nothing ? in a zip ? a 7z ? what next ?
Roms ? chd ? samples ? bios ? devices ? and next ? console ? computer ?
Main set or clone or bios or device or sample ?
What about roms inconsistencies ? Games with 5 times the same crc and different names ? merged roms not found in main set ? roms with Same crc in main and clone, but different names ? or same name with different crc?
How to identify roms with no crc ? sha-1? md5? or... ?
Moving roms ? from zip to 7z, zip to folder, folder to zip, folder to folder, zip to zip......
and more more more....

Good luck  

[Gray_Area]

Is anyone else getting the idea somebody is back?

[qwench]

- you click on a button called  'exe' or 'target' or 'target exe'
- you click another button called 'dir' or 'directory' or 'target directory'
- you press another button called 'fix'
And, hey, all your romz is correct!

In romcenter:
 - you click file/open and select your exe
 - you click file/add directory
 - you select directory and click 'fix'
And, hey, all your romz are correct!

[Gray_Area]

- you click on a button called  'exe' or 'target' or 'target exe'
- you click another button called 'dir' or 'directory' or 'target directory'
- you press another button called 'fix'
And, hey, all your romz is correct!

In romcenter:
 - you click file/open and select your exe
 - you click file/add directory
 - you select directory and click 'fix'
And, hey, all your romz are correct!

Yeah, but what I'm interested in is a manager that'll work with your torrent client.

[drventure]

The other issue there, is that that kind of app would pretty much require the dev to... find, download and manage quite a number of the roms out there, if not all of them (theoretically, one should be able to build an app that handles a small number of known roms for test, and that should be good enough to say it's good for all roms, but, we all know how that kind of theory plays out 

So, if you've already gone to that much trouble, the app might not be worth the added hassle.

[Gray_Area]

The other issue there, is that that kind of app would pretty much require the dev to... find, download and manage quite a number of the roms out there, if not all of them (theoretically, one should be able to build an app that handles a small number of known roms for test, and that should be good enough to say it's good for all roms, but, we all know how that kind of theory plays out 

So, if you've already gone to that much trouble, the app might not be worth the added hassle.

I thought the xml was the resource, which is always current, right? If there's a way to point utorrent at the xml and tell it to replace only the files I have, I'm all ears.