Do you merge?

[thorus]

I have multiple rom collections that have been merged reducing the amount of space required by a large amount.  Although I have not seen it mentioned much as far as support  with front ends and emulators.  Now I know I could use the pre-execution fields to pull it out but it would also require the FE to know which one in the zip file to grab.  Has anybody tried this or do you guys just un merge and buy another Hard drive?

[Howard_Casto]

You don't want your roms merged... you want them split/merged. 

Split/merged means that every single game gets it's own zip file, but any roms duplicated amoungst variants are only contained in the parent's zip.  Aside from the minimal amount of space required to make the zip file itself, you will see no real size difference between merged sets and split/merged sets.  The only difference is fes get a zip for each rom and thus they are happy. 

It's an option fro both romcenter and clrmamepro.... go fix em already.

[Minwah]

What Howard said...

Also, merged sets can cause problems for FE's recognising the clones...especially since -verifysets was removed from Mame.

[sWampy]

Are you guys sure he wasn't talking about mame merged roms, or goodmerge merged console roms? It does a great job of cramming all the roms in the smallest space immaginable, but they aren't very useful in their smallest versions, since compacting them that way uses 7-zip archives that most console emulators don't support.   To use them you would need forntend support for the format to help you.   It really rocks though, cause the program understands that Castlevania is called Kumajo Dracula or whatever in Japan, and groups them, giving the main set the name you pick as your primary language.  A full set of N64 roms shrinks from shrinks from something like 14 gigs down to 4.7.   Complete Genesis sets shrink to a single cd size.     

Great for archiving, would be wonderful if frontends/emulators supported them.

[thorus]

sWampy that is exactly what I'm talking about.  I'm debating writing my own Front end because I would like that along with some other customization for the layout and finally to put all settings into a database instead of scattered files.  But I wanted to make sure that it isn't available before I go too far into making my own.  I have not seen any support for the 7zip merged sets that I have from emulators so I was assuming it would be handled via the front end.  I haven't looked into this much for mame so really this is more of a console problem as far as I know.

[sWampy]

I'm not sure what all front ends could use these, but I bet mamewah, gameex, and dk could.   In mamewah, you could make a batch file that extracted the rom you needed from the .7z file, load the emulator, then delete the unzipped rom on exit.  In gameex, you could just give it a pre launch command that extracted the rom, and a return command to delete the rom.   In both cases you could only play one version of each rom, but I don't think that would really make a difference often.

[Minwah]

I might be missing something...but I really don't see the point if emulators don't support them...

And HDD space is cheap as chips now so space isn't really an issue.

[sWampy]

I might be missing something...but I really don't see the point if emulators don't support them...

And HDD space is cheap as chips now so space isn't really an issue.

I personally think it's cool, because you end up with say 300 zips listed in a directory instead of 900+.   I think it would be nice to have just the 300 games listed, ie just castlevania, not castlevania (e), Castlevania(U), Dracula (J), etc, but still have all the other versions available in the same directory, without having to keep 2 copies, or worry about keeping track of a bunch of different filters.  While direct frontend support isn't necessary to use them.  You can always make a batch file to just unzip the primary rom, and then launch the emulator pointing at the extracted rom, then delete on exit.   If the frontends knew about the merged sets directly, they could generate a list filtered list, and also allow you to hit a key while sitting on a parent to show a list of other versions contained in that archive.   So if there was some cool intro/ending/extra feature of the Japan/French/Whatever version you read about, you could play it without much trouble.

[Howard_Casto]

You seem to be talking about console roms, which incidentally don't have clones (each version is unique). 

Also you do not want fes that are aware of merged sets, mainly because that requires crc checks and rom verification during the scanning process which can take a LOOOONG time.  Think of how long it takes romcenter/clrmamepro to verfiy a complete mame set. 

[sWampy]

Well there are a ton of hacked console roms, that are close to being like clones in arcade sets.   But there are also misdumps, overdumps, this kind of header, that kind of header, that nobody really seems to be able to identify as the correct dump.  What goodmerge does is use 7zip's ability to make solid archives that compress the crap out of these small changes to make an archive say named game.7z that contains game.rom, game(j).rom, game(E).rom, game(e)overdump.rom. or whatever.   All the frontend support would entail would be looking for all .7z files in a directory, and being able to extract the primary rom from that file and call the emulator.  It would be nice if it had the option to list all files in the .7z so you could run any game.   http://goodmerge.sourceforge.net/Statistics.php  shows the savings that can be obtained.   For those of us with multiple cabinets sharing files over a network, having small easily to maintain console collections would be awesome.   

http://lusspace.free.fr/html/modules/news/ supports goodmerge sets now btw.

[Howard_Casto]

Well, with respect, those aren't clones.... they are unique romsets.  What goodmerge is doing then, is making some sort of bastardized mush of all the rom variants (note I didn't say clones because they aren't clones) and putting them in 7zip format, a format that is NOT universal and not all emus support (not to mention no fe that I know of supports) and just making it difficult for everyone to read... emus, fes, rommanagers ect....

Let's assume that this all wasn't issue.  You still end up with a bunch of super-compressed files, which take a long time to umcompress and would trash the heck out of your harddrive from constant use. 

It's just all-in-all a bad idea. 



Also the question arises.... why would you even need the rom variants?  Sure there might be one or two you wish to keep (so keep em) but in regards to console roms, most variants are hacked up crap or foreign language translations.  Mind you it's ok to keep em, but I don't see why you need them launchable from the fe.  Personally I just use gootools to sort by region /hack and only point to the US folder fro my console roms.  Any special case roms I wish to run I can just throw in that folder.

[sWampy]

Well, with respect, those aren't clones.... they are unique romsets.  What goodmerge is doing then, is making some sort of bastardized mush of all the rom variants (note I didn't say clones because they aren't clones) and putting them in 7zip format, a format that is NOT universal and not all emus support (not to mention no fe that I know of supports) and just making it difficult for everyone to read... emus, fes, rommanagers ect....

Let's assume that this all wasn't issue.  You still end up with a bunch of super-compressed files, which take a long time to umcompress and would trash the heck out of your harddrive from constant use. 

It's just all-in-all a bad idea. 

It's not a bastardized super compressed format, it's just a better format that zip.   It zips across the entire archive, so you get bigger more efficient hash tables, so better compression as a whole.   The format is just fairly new, but is gaining a lot of support from others, clrmamepro has supported it for a long time, and tons of no emulation support.   It open source and doesn't have a lot of the problems with copyrights and patents that zip and rar have.   It's about like png was a few years back when mame added support for it, it was new and most people were like why would you go to png, jpg is all that will ever be needed, and before that people said the same thing about gif.

Compressing might be a little slower, but decompression is just as fast as zip.   A small # of small files causes a lot less fragmentation than a large number of large files. ;-)

Also the question arises.... why would you even need the rom variants?  Sure there might be one or two you wish to keep (so keep em) but in regards to console roms, most variants are hacked up crap or foreign language translations.  Mind you it's ok to keep em, but I don't see why you need them launchable from the fe.  Personally I just use gootools to sort by region /hack and only point to the US folder fro my console roms.  Any special case roms I wish to run I can just throw in that folder.

Most people wouldn't want them all, and shouldn't need them all, no tool exists that does a good job of auditing the roms for most consoles than good tools, that make sure you have one and only one version of everygame made for a system.   With goodmerge, you are left with only one file for each game, and all the other versions of the game are stored along with the parent taking up very, very little additional space. 

I've done as you say for ages, only point at the us folder, but I then have to keep all the other versions on the drive wasting space if I want to be able to produce a list of missing games. 

To me being able to store 880 n64 games that used to take up 25.5gigs in 4.71 gigs of space, or 18 gigs of gba zipped roms in 9 gigs of space.

[Howard_Casto]

Huh?  The bastardized, mushed up mess the the idea of putting roms that aren't related at all in the same zip.  Goodmerge is doing something that doesn't make any sense, namely grouping variants in the same zip rather than keeping them in their own zip.  I wasn't referring to the 7zip format.  Better read that again.

An arcade example of this would be to zip up all the variants of dragon's lair into a single zip.  Why would you do that?  They aren't real clones and thus the roms don't depend on each other.  Also daphne can't find the roms because they are in the wrong zip so only the parent would ever be playable.  Fe's also couldn't find the "clones".  If the emu can't see the "clones" and the fe can't see the "clones", then you can't play the "clones".  At that point there isn't any point in keeping them.

That pretty much sums up what you are wanting to do.

On the topic of 7zip imho it's a complete waste of time to use a form of compression on roms that the emus can't support.  It's fine for archiving stuff you won't be acessing with anything but 7zip but that's about it.

And you are missing the point on decompressing.  Most emus support zip... so you don't have to decompress the files ahead of time.  To do what you are propsing would require fe authors to go to essentially decompress each zip to check what files are in it.  (Even when you use code to retrieve contents of a zip file, that is essentially what happens... a portion of it gets uncompressed in memory to read the contents.) Then when launching a game, the fe would have to have a table stored somewhere of what zip that particular rom is stored in... decompress the whole thing again, and copy the loose rom somewhere. 

The thrashing of your harddrive comes from the constant copying and deleting of files.... every single time the fe scans for roms, and every time the fe launches a game.

As opposed to simply launching the zip file with the emu. 


For the record.... png never really was needed in mame for snapshots. Png's, in general are of a larger file size than a comparable gif.  Now pngs are used for the artwork system, which requires an alpha channel. (Of course everyone uses them wrong and makes a seperate alpha mask when the frikkin png can contain an alpha channel, making things even larger.) I suspect that was the real reason pngs are used.



And my question on your final response is....

Yes your dat manager is going to report missing files if you delete the other revisions.  Your point is? You know which ones you don't need.  The rom managers show the country revision in the name so you can say... "Oh that is a Japan rom, I don't need to worry about that one".

[sWampy]

Huh?  The bastardized, mushed up mess the the idea of putting roms that aren't related at all in the same zip.  Goodmerge is doing something that doesn't make any sense, namely grouping variants in the same zip rather than keeping them in their own zip.  I wasn't referring to the 7zip format.  Better read that again.

The are related,

1080 Snowboarding (E) [!]
1080 Snowboarding (E) [f1]
1080 Snowboarding (E) [f2] (NTSC)
1080 Snowboarding (JU) [!]
1080 Snowboarding (JU) [f1] (DS-1)
1080 Snowboarding (JU) [f2] (PAL)
1080 Snowboarding (JU) [f3] (PAL)
1080 Snowboarding (JU) [f4] (PAL-Z64)
1080 Snowboarding (JU) [f5] (SRAM)
1080 Snowboarding (JU) [f5][t1]
1080 Snowboarding (JU) [f6] (Boot-PAL)

are all related for sure, sure you might not really need anything but the 2 that say [!] and I'd really only want the (JU)[!] one really, but why not keep them all if you can keep them all in a smaller space than just keeping the 2 in seperate zips.   

Or that

Donkey Kong Country (Competition) (U)
Donkey Kong Country (V1.0) (E) [!]
Donkey Kong Country (V1.0) (U) [!]
Donkey Kong Country (V1.1) (E)
Donkey Kong Country (V1.1) (U)
Super Donkey Kong (V1.0) (J)
Super Donkey Kong (V1.1) (J)

went together even thought they don't have the same name.

I haven't looked, but someone told me that 7zip's library was compatable/drop in replacement for zlib.dll that lots of people use in their emulators/frontends, but supports 7zip, zip, rar, solid archives, etc.   So it I would think before long, newier emulators would start supporting the format natively. 


My cabinet runs perfect disk nightly, so I highly dought that the 3 or 4 console games that might get played a day would cause a fragmation nightmare, and if they did, the machine has 2 gigs of ram, I could afford to unzip them to a ram drive if I had do.   It only has an 80 gig sata drive with windows and the emulators on it, all the roms/disk images are loaded over gig-e.

[Howard_Casto]

The are related,

1080 Snowboarding (E) [!]
1080 Snowboarding (E) [f1]
1080 Snowboarding (E) [f2] (NTSC)
1080 Snowboarding (JU) [!]
1080 Snowboarding (JU) [f1] (DS-1)
1080 Snowboarding (JU) [f2] (PAL)
1080 Snowboarding (JU) [f3] (PAL)
1080 Snowboarding (JU) [f4] (PAL-Z64)
1080 Snowboarding (JU) [f5] (SRAM)
1080 Snowboarding (JU) [f5][t1]
1080 Snowboarding (JU) [f6] (Boot-PAL)

No they aren't.  They aren't related at all.  They are the same title but totally different versions.  180 ju f1 and 1080 ju fe use totally different roms.  It's like zipping pacman and galaga together... it makes no sense. 

As I said they are variants, not clones.  A clone uses some of the roms from the parent.  Console roms only have one rom, so by default, they cannot have clones. 

By merging them together you are just making a mess.  They are seperate because they are totally seperate games.  If you don't like all those zips (why?  as you should never have to look at them, that is what a fe is for) then delete the variants you don't like but putting them together with roms that aren't releated is just silly. 

7 zip aside.... two zips don't take up more room than one... at least not enough to make any difference.  It's been a while but at one point merging the entire mame set, chds and all (and I mean real merged, not split-merged) saved 250-500 megs of space..... that's a maximum of 500 megs out of over 10 GIGS.  In other words not enough to worry about.  7 zip is the same way, two zips don't take up much more room than one zip with the same two files in it.


But getting back to 7zip......  You keep saying in the future emulators/fes should start supporting it.  Well yes, I assume they will, but what does that have to do with now?  As I said, compressing working roms in a format that isn't universally supported doesn't make a whole heck of a lot of sense. 

[sWampy]

Oh, you are so right, roms that are 90-98% identical, are the same games with only different languages and headers, aren't related at all.  Some might be recompiled, relinked with newer versions of the sdk, etc but they are mostly identical, or you wouldn't be seeing the huge space savings you see when storing them together.   Sticking them together taking up 25% of the space of them zipped separately, makes no since at all.   

Here is the example 1080.


http://www.landoncrews.com/1080.png


Notice that the 1080 Snowboarding.7z contains 25 versions of 1080, and is 11,005,952 bytes.   While the 1080 Snowboarding (E) (M4) [!].zip is 11,313,152 bytes.  If you were to take all 25 versions of 1080 snowboarding out and zip them into zip files, the total disc space would be around 282,828,800.   You are correct, it's not twice the size, I'm wrong, this makes no since at all, what a waste of time.   

[liquid8]

oh my... I have to take Howards side on this... damn. This brings on 2 points:

a) if you want to so bad, keep them that way (no use in arguing about it), but you are going to have to either, unzip them each time you want to play them (which will probably be never), or write your own frontend, which I wouldn't recommend, because in your fe you will have to do exactly what Howard said code-wise for it to work...

People like Howard, )p(, Minwah and so on have been working on their frontends for years.. they know what works and what doesn't.

and

b) Did you really own all those versions anyways?? Oh... did you even own one of them? If they really are 90-98% identical, why do you need 25 (TWENTY FIVE) versions. A lot of us tend to get fussy with keeping a "complete collection", and I sympathize with what you want to do, but think about it... it's not worth the hassle. Trust me, I tried.

liquid8

[Howard_Casto]

Oh, you are so right, roms that are 90-98% identical, are the same games with only different languages and headers, aren't related at all.  Some might be recompiled, relinked with newer versions of the sdk, etc but they are mostly identical, or you wouldn't be seeing the huge space savings you see when storing them together.   Sticking them together taking up 25% of the space of them zipped separately, makes no since at all.   

Here is the example 1080.


http://www.landoncrews.com/1080.png


Notice that the 1080 Snowboarding.7z contains 25 versions of 1080, and is 11,005,952 bytes.   While the 1080 Snowboarding (E) (M4) [!].zip is 11,313,152 bytes.  If you were to take all 25 versions of 1080 snowboarding out and zip them into zip files, the total disc space would be around 282,828,800.   You are correct, it's not twice the size, I'm wrong, this makes no since at all, what a waste of time.   

You just aren't getting it.... This has nothing to do with how similar the games are or play (which is virtually identical) they are DIFFERENT FILES!!  Do you get that?  Is it too far over your head to understand?  The emulator authors will NEVER add support for files merged in the way you are talking about because the 1080 u 1 variant doesn't share ANY FILES with 1080 u 2. 

They are both SINGLE FILES... it is PHYSICALLY IMPOSSIBLE for them to share any files because they both only have one.  It is the same for nearly every console emu.  Each game is a singular file.  It's illogical to expect emu developers to put in a lot of code and a very time-expensive zip search routine for searching for files where they don't belong. 



Maybe you don't understand how true clones work so I'll explain it.

For example let's examine the fictional roms "game.zip" and "gameb.zip"

Open up game and you see:

game-snd.bin
game-vid.bin
game.bin

Open up gameb and you see

game-snd.bin
game-vid.bin
gameb.bin


Now these two games actually have files in common.. namely game-snd and game-vid.  So merging them makes sense..... we throw out the extra copies of "game-snd.bin" and "game-vid.bin"
to make a new gameb.zip that only contains:

gameb.bin.

Now the emu has to be specifically coded to know all the parent-rom relationships or else it would literally take hours to search through thousands of zips for all the files.  The emu has to have an internal database that says "I should look for the first two roms in game.zip cause that's the parent, then I'll look for the third file, first in the parent and then in gameb.zip"  Mame does it because it is a documentation project.... emus that rip-off mame's data also do it (like zinc, raine, ect) because the data is available.  For consoles and I can't stress this enough... it is completely and totally impossible to do so.  Why?  Well for one console emus are designed to run any rom because each and every cart runs on the same hardware... namely the console.  So emu developers don't know, and for the most part don't care what roms are out thier for said system as (with a few exceptions) all games will run on their emu if they are dumped properly.  You are asking them to keep a database of 10's of thousands of roms, which would literally require hundreds of man hours just so you can save some space?  And those guys are good at coding.  On top of that you are essentially asking fe authors (which admittedly aren't as good) to do the same thing, if you want them to detect variant roms inside a zip and extract them for you in a speedy manner with no known record of how roms are related to each other or even which ones exist.




Finally....

Did you just compare multiple REGULAR zip files to a single 7 zip file?  That's comparing apples and oranges.  Compare multiple 7 zip files to a single 7 zip file or multiple zip files to a single zip file. 

And for the thousandth time... the emus simply don't support 7 zip so there's no use in using 7 zip compression for anything other than archiving.  IF and WHEN they do support 7zip THEN it's a valid argument.


And I didn't want to start a flame war but since liquid brought it up....

You shouldn't have a full set of console roms.  I don't.... I only have the roms to carts I actually own.  Why?  Because you can't use that "well these games are no longer available" or "these games are impractical to own" argument for console roms like you do arcade machines.  With the exception of odd-ball systems like the neogeo carts for old consoles are still readily available and cheap. 

While I won't get into a legal argument over it (as either way it's illegal) I do morally object to rom hoarding for the sake of rom hoarding when it comes to consoles.

I'm very sorry for yelling but you seem to be caught in an infinity loop of brining up points that don't matter because your suggestion is still too impractical regardless of the few advantages you bought up. 

Long story short.... can't use 7zip cause the emus can't use 7zip.  Shouldn't merge em cause they aren't related. 

[sWampy]

Ok, I tried to be nice,  I'll stop trying to explain it to you, you will never get it!

It's wierd I make well over 6 figures programming, and you have to resort to begging for money to add features to your buggy frontend that almost nobody uses. 

It took around 3 minutes to throw together a perl script, to rename the files, and maybe a minute and a half to make a batch file go get mamewah working with .7z files for n64, and I went from having 140 zipped roms that took up 8 gigs, weren't in a good state at all to 393 .7z files that are well organized, and only take up 5 gigs.   

Telling people they don't understand a concept that is clearly miles and miles over your head is just laughable, you couldn't even understand the question thorus originally asked.

[thorus]

Wow this has gotten into a bit of a flame war, sorry guys.  But back to my original topic it looks as if nobody uses the merge as me and sWampy are thinking. 

I'm not implying that I want to make emulators handle the roms I'm looking more so for a front end solution.  sWampy mentioned using batch routines in MameWah but how would MameWah handle which to pull out from the zip file?  I would like the option similar to mame32 where variations of game (ie 1.0, 1.1 EU) would be a sub category of the game and you could choose which you mean when you say play The Legend of Zelda.

As far as drive space goes I'm running out of space on my 300 gig drive which I assumed would be large enough.  And when I take files that fit in 1.4gig and explode into 18gig and do that 10 times over  I can see where all my space is going.  I'm still working on exactly how my directory structure will work out but I didn't like the language split since I still ended up with dupes/hacks that cluttered the directory. 

[Howard_Casto]

Ok, I tried to be nice,  I'll stop trying to explain it to you, you will never get it!

It's wierd I make well over 6 figures programming, and you have to resort to begging for money to add features to your buggy frontend that almost nobody uses. 

You know changing the subject and bringing in personal insults is generally a sign of a person who is losing an argument. 

And I've never begged for money, I wanted to get my "free___" referals finished so I posted that.  I work on my apps regardless of that.  I'm sure you find my frontend buggy, that's your opinon.  So where is your frontend?  Where are your helper apps?  You are a big, well-paid programmer  what contributions have you made to this hobby since you are so much more qualified than the rest of us?  None?  Oh I thought so.  Seems funny that the guys who aren't pros are the only ones willing to get off their high horses and actually help the community in a noticable fashion.

I also find it extremely annoying that if you make 6 figures programming, as you say, you are asking others to do your work for you.  Why are you even complaining?  Get off your lazy rump, do it yourself, release it to the public and leave everyone the hell alone!


Your personal attack on me is rude and un-called for.  You should be ashamed of yourself. 

[Goz]

Wow... I am speechless Howard calling someone rude... thats a change. 

[Howard_Casto]

Wow... I am speechless Howard calling someone rude... thats a change. 

Goz butting in where he doesn't belong... I'm... um not suprised at all.

[Kremmit]

Watch out, Goz!  He'll get your post's soul! 

[Goz]

Pfffft...