.146 Has been out for a while now... time for the conversion to 7z

[Howard_Casto]

I've been dreading this, but in order to keep from having to do it later, I'm going to go ahead and convert my roms to 7zip.  It isn't a required conversion, but seeing as how mame is only going to get larger it only makes sense to use the format with better compression.

Clrmamepro apparently supports an auto conversion, but that program always mutilates my roms. 

Over at romcenter's forums I found a batch file posted a few weeks ago by somebody named nayo.  It had a few bugs in it (which I corrected) but I've tested it and it converts things properly.  You'll need 7zip installed on your system to use this.  Just save it to a bat file, place it in your rom folder and run it.

*****************************************
@echo off
md temp
md done

:LOOP
FOR %%i IN (.\*.zip) DO (SET varx="%%~ni")
IF NOT EXIST %varx%.zip GOTO END

move %varx%.zip .\temp\
cd temp
"c:\Program Files\7-Zip\7z.exe" e %varx%.zip
del %varx%.zip
attrib -r *
attrib +a *
"c:\Program Files\7-Zip\7z.exe" a -mx9 -mmt %varx%.7z *
move %varx%.7z .\..\done\
del * /Q
cd..


GOTO LOOP
:END

rd temp
cd done
move * .\..\
cd..
rd done

*******************************

As with anything "blind" like this, I would suggest you make a backup of your roms first, but I have converted a few of mine for testing and verified them to be correct via mame.


A current full romset in mame is around 40gigs, minus the chds.  I don't know how much it will save total, but reports seem to indicate that it saves a lot more space then the mame devs estimate.... somewhere around 10 gigs....which isn't a lot in terms of the giant hard drives we have now, but it's a 25% compression boost!  There is also a simplier batch file on the site that'll compress your chds, but this seems sketchy to me and I wouldn't reccomend compressing with anything other than chdman (which is flaky atm, so I'm waiting to do the chds until .147).

http://forum.romcenter.com/viewtopic.php?f=2&t=2352

[Howard_Casto]

I thought I would keep you guys up to speed as I do the conversion process.  I started the batch file around 8:40 this morning.  It's converted about 5,700 out of the 18000 or so roms in mame.  Thus far the savings in space is around 5 gigs. 

So yeah, it's takes frikkin forever to do a complete set, but the batch file isn't super taxing on the system so it's just running in the background as I use this computer normally.


I've also been verifying the roms periodically via romenter and mame and thus far no duds that I can find. 

Stay tuned.... sometime late tonight or early tomorrow we'll find out how this little experiment did. 

[tony.silveira]

I thought I would keep you guys up to speed as I do the conversion process.  I started the batch file around 8:40 this morning.  It's converted about 5,700 out of the 18000 or so roms in mame.  Thus far the savings in space is around 5 gigs. 

So yeah, it's takes frikkin forever to do a complete set, but the batch file isn't super taxing on the system so it's just running in the background as I use this computer normally.


I've also been verifying the roms periodically via romenter and mame and thus far no duds that I can find. 

Stay tuned.... sometime late tonight or early tomorrow we'll find out how this little experiment did.

thanks for taking the plunge!

[Howard_Casto]

8,689 to go!

So I'm guestimating a full set takes around 18 hours....  certainly a while, but it isn't terrible.  Wake up early in the morning, go to work/school and it'll be done by bedtime.

[Howard_Casto]

Finished!

It's around 1am here so about 17 hours give or take 30 minutes. 

My romset isn't completely rebuilt yet (still gotta run em through romcenter) but my roms now weight in at just under 31 gigs.  That's over 10 gigs of savings!


All my roms seem to be intact and in working order.  When starting mame I can't see any significane in load times, even with larger games like sfIII. 

My verdict?  Since the effort on my part consisted of leaving my pc running and it didn't tie up my pc, I would say it was worth doing.  It isn't a huge space savings, but considering that emulators gobble up so much space, even on a huge harddrive it's worth it because 10 gigs here and there add up. 


I'm now going to go back and convert some of my ancient console rom sets from zip to 7z format using the same batch file.  If I convert every rom that supports 7z to 7z then I will reclaim 25% of my harddrive space (which has nothing but roms on it), which IS significant if you look at it that way.

[Howard_Casto]

Just did the samples folder.  It took all of one minute.  Saved about 10-20 megs of space, which is a smaller percentage of savings than the roms.  Again everything still works normally. 

I just did this to see if it would help any though.  Mame now supports flc-based samples but none have been released yet.  Flc compression is specifically for audio, so one would notice a more pronounced savings that way. 

I'll do some tests on the artwork files tomorrow. 

[capsule]

Well, deleting the roms you never play would make you gain more space... 

My 2 cents...

[Turnarcades]

I did this practice years ago on our demo machines to reduce required space... firstly the painstaking process of hand-filtering complete romsets to remove non-working, duplicates and other such crap, recompressing video previews and roms at higher rates and resizing snapshots to the exact size used in the frontend. Took months and I still revise things now as I find non-working stuff, but I shaved huge amounts of space from every platform this way, taking over 250Gb worth of stuff down to less than half that. I'm all for emulators supporting tighter compression - why wouldn't you want it?

[Howard_Casto]

Well that's my sentiment.  It certainly isn't required and it annoys the hell out of me when people think that mame roms HAVE to be in 7z and chds HAVE to be converted to v5, but why complain about the option to use a better compression method that'll save some space?

I'm an occasional mame dev (read barely ever but still sometimes).  I need the whole set, so picking and choosing isn't an option for me.  Nothing is more annoying than trying to work on a set of drivers only to find that you don't have all the roms... so you have to stop what you are doing and go download them.  So this is a big help for me.  I can work on my output adventures without having to constantly stop and without filling yet another harddrive.

Just for the record, artwork compression didn't help much (forgot to post it).  This is probably becuase pngs are already lossless compressed, so the only thing that gets packed is the text files.  So I wouldn't bother doing the art files. 

[Gray_Area]

I'm wondering whether the last update to 7z being two years ago will have any effect on it's near-future use....

[Howard_Casto]

I think the main reason 7z hasn't been updated in ages is that it doesn't need to be.  7zip is the replacement to zip, like it or not.  Zip was created in the old Fat16 days and thus it was never really geared towards large files and the cluster sizes that modern computers use.  7zip is much better suited to NTSC and file systems that support larger files. 
It actually boggles my mind that Windows only supports Zips and Cabs natively and not 7 Zip.  It would have to be a liscensing issue or something.

Until a new file system comes out, or harddrive/file sizes significantly change, there isn't any reason to update 7zip.

For anyone wondering... rar is about the same as 7zip, it's just rar is a proprietary format.

Or at least this is how I understand it.  Compression logic is one of those mind-boggling things that can confuse the heck out of anyone.     

[Haze]

you'd be better off compressing your samples as FLAC, not 7z, unless you already did ;-)

anyway, yeah, that's about what I'd expect from .7z in terms of savings

and no, it's not change for the sake of change, .zip is an old container with 32-bit limits, some more recent games actually have over 4gb of ROM, and .zip can't contain that.  Of course there is some debate over simply using CHD there instead as the games we've seen using such high capacities address it in a serial manner rather than it being memory mapped, but you never know what the future holds.

of course plain old .zip isn't going away

right now the code support in MAME is a bit ugly, but hopefully the zip/7z stuff can be abstracted by some common 'compressed file' interface at some point, at which stage integrating other formats would become easier too.

I wouldn't read too much into things like 7-zip not being updated in years tho, you can't infinitely make the compression better, and you don't want to make things more advanced and introduce incompatibilities with older versions.  The only things you'd really want to put in at this stage are optimizations, cleanups, and bug fixes (security related etc.)  Of course improving the GUI would be good too, but open source and GUIs have never really agreed with each other ;-)  Just because Winzip / Winrar can whore out new versions means nothing, they're commercial projects and need to look busy, even if they're busy doing nothing.

[Howard_Casto]

Just because Winzip / Winrar can whore out new versions means nothing, they're commercial projects and need to look busy, even if they're busy doing nothing.

I can attest to that.  I remember a year or two ago deciding to check all of my programs to make sure they were up to date.  On one computer my version of winrar was five YEARS old!  I didn't notice because it still rared fine and unrared brand new files.  So what the heck were the five years worth of updates for?

[EmBalmer]

After extracting all zip's to a "seperate folder" i ran this batch and waited, and waited 
@echo off
Echo %Date% %time% > Temp.txt
for /D %%d in (*.*) do (
   7z a "%%d.7z" ".\%%d\*"
)
type Temp.txt
Echo %Date% %time%
Pause
del Temp.txt
----------------------------------
You can add 7z to the environment "PATH"
     My Computer (right click) >  Properties > Advanced > Environment Variables
     Path           C:\WINDOWS\system32;C:\Program Files\Resource Kit;C:\Program Files\7-Zip

[Gray_Area]

you'd be better off compressing your samples as FLAC, not 7z, unless you already did ;-)

anyway, yeah, that's about what I'd expect from .7z in terms of savings

and no, it's not change for the sake of change, .zip is an old container with 32-bit limits, some more recent games actually have over 4gb of ROM, and .zip can't contain that.  Of course there is some debate over simply using CHD there instead as the games we've seen using such high capacities address it in a serial manner rather than it being memory mapped, but you never know what the future holds.

of course plain old .zip isn't going away

right now the code support in MAME is a bit ugly, but hopefully the zip/7z stuff can be abstracted by some common 'compressed file' interface at some point, at which stage integrating other formats would become easier too.

I wouldn't read too much into things like 7-zip not being updated in years tho, you can't infinitely make the compression better, and you don't want to make things more advanced and introduce incompatibilities with older versions.  The only things you'd really want to put in at this stage are optimizations, cleanups, and bug fixes (security related etc.)  Of course improving the GUI would be good too, but open source and GUIs have never really agreed with each other ;-)  Just because Winzip / Winrar can whore out new versions means nothing, they're commercial projects and need to look busy, even if they're busy doing nothing.

This is why I asked. Now, why flac?

[Howard_Casto]

wav files don't really compress the much when zipped up due to their file structure. 

flac is lossless compressed wav afaik.  Kind of like how a png has a much smaller file size than a bmp and yet they both have the same picture quality.

An mp3 or ogg wouldn't work because those degrade audio quality.

[Gray_Area]

wav files don't really compress the much when zipped up due to their file structure. 

flac is lossless compressed wav afaik. 

Ahhh. I bet that's in Wiki, but thanks for answering.

What's funny is that there are tons of people, using Vista even, who can't read flac.