Build Your Own Arcade Controls Forum

Main => Software Forum => Topic started by: MonitorGuru on November 09, 2004, 12:28:09 pm

Title: ZipMax open source updates/suggestions master thread
Post by: MonitorGuru on November 09, 2004, 12:28:09 pm

All --

I am creating this thread as a place for ZipMax enthusiasts and/or developers to either post suggestions on what they'd like to have added/changed in ZipMax, and the source code (and possibly) executable changes made (links only to exe's please... source is tiny, exe's arn't)

As eluded to in a recent ZipMax thread (asking for .INI settings), I made changes to ZipMax .5 as currently posted on Roman's ClrMame web site.  I submitted the changes nearly a year ago and from that and through various replies on Roman's web site, he doesn't want to take the time to support/update ZipMax.  My changes have not appeared yet. (Nothing against Roman, he provides a great service, but he rightfully needs to dedicate his time appropriately)

Therefore, I think since this site is more frequently visited and provides a nice discussion area, it would be good to attempt to support ZipMax here.

The changes I made were to allow you to start up one instance per machine over a network, then drag and drop files from a shared network folder onto all of the instances, and each machine would work on compressing the data, and all without any machine duplicating the efforts of another machine, or requiring you to separate the work beforehand and copy back/etc..

As soon as I locate the source code changes I made, along with ReadMe and .INI file changes, I will post them to this thread.  For now, those that want the changes will need to apply them and compile them themselves.

I will add a reply later tonight with the updates.  Feel free to use this thread to discuss ZipMax and what changes you'd like to see in it.  Please don't post long INI files *OR* discussions about configuring a specific compressor or the benefits of the compressor/settings to the thread, use the other thread for that.  (This thread should be about the core ZipMax processing program, not the things it shells out to)

PS: If you don't know what ZipMax is, check out Roman S's ClrMamePro site. However, in summary:

ZipMax is a batch processing program that when fed a list of command line compression utilities and another list of ZIP files, uncompresses the zip file, and recompresses it with evey compression utility and parameters it's told to.  Then, it will "create" the smallest total archive by looking at EACH COMPRESSED FILE inside EACH ZIP and picking the smallest one across all and building the final zip from that.  Therefore the resulting final ZIP archive will almost always be smaller than any single total archive from any individual compression utility.  (Since it works at the per file level within the zip and not the whole zip itself).  

This makes it unlike any other maximum compression utility out there that only looks at the resulting ZIP as a whole.  It has a GUI with a drag and drop interface, and is manipulated through an english .INI file.  You however will need to provide the compression utilities and configure it to use them correctly.

Title: Re:ZipMax open source updates/suggestions master thread
Post by: krick on November 09, 2004, 11:37:49 pm

I have a couple of ideas that I thought I'd throw out there.  I don't know if they would be separate utilities or functionality incorporated into zipmax.

1) You know how ZipMax compresses the zip contents multiple times with different compressors and then compares the contents to build a "best of breed" zipfile?  The problem is that we don't have access to every compression program and some out there might be better than what we are using.  Imagine if you had a tool that could just compare existing zips from different sources without re-compressing anything.  That way, if I were to acquire an already zipped rom from another source, I could just scan it to see if any of the individual parts would help make my already compressed zipfile smaller.

2) It would be nice if I had a utility that could make an ascii dump "fingerprint" of sorts of a given romset that logged each individual rom and it's compressed size.  That way, two people could just do a diff on their "fingerprint" files to see which roms were smaller in each set.  The fingerprint would be different for each version of mame and the utility would need to work with a DAT file in case someone wanted to fingerprint an incomplete set.  That way the utility could leave blank lines (or some other sort of placeholder) for missing roms.

Title: Re:ZipMax open source updates/suggestions master thread
Post by: krick on November 10, 2004, 12:05:04 am

Quote from: krick on November 09, 2004, 11:37:49 pm

2) It would be nice if I had a utility that could make an ascii dump "fingerprint" of sorts of a given romset that logged each individual rom and it's compressed size.

Well, I just found a solution for this one that I think might work pretty good with a little tweaking:

Using the PKUnzip for DOS ver 2.50...

pkunzip -vn *.zip > ziplog.txt

This generates a file full of blocks of text that look like this...

Searching ZIP: 1941.ZIP

 Length  Method   Size  Ratio    Date     Time    CRC-32  Attr  Name
 ------  ------   ----- -----    ----     ----   -------- ----  ----
 131072  DeflatN  36400  73%  03-23-2001  10:50  9deb1e75 --w-  41e_30.rom
 131072  DeflatN  29134  78%  03-23-2001  10:50  df201112 --w-  41e_31.rom
 131072  DeflatN  64791  51%  03-23-2001  10:50  d63942b3 --w-  41e_35.rom
 131072  DeflatX  62222  53%  03-23-2001  10:50  816a818f --w-  41e_36.rom
  65536  DeflatX  17727  73%  03-23-2001  10:50  0f9d8527 --w-  41_09.rom
 131072  DeflatX 116447  12%  03-23-2001  10:50  d1f15aeb --w-  41_18.rom
 131072  DeflatX  81400  38%  03-23-2001  10:50  15aec3a6 --w-  41_19.rom
 524288  DeflatN  82665  85%  03-23-2001  10:50  4e9648ca --w-  41_32.rom
 524288  DeflatN 263921  50%  03-23-2001  10:50  ff77985a --w-  41_gfx1.rom
 524288  DeflatN 178690  66%  03-23-2001  10:50  983be58f --w-  41_gfx3.rom
 524288  DeflatN 265126  50%  03-23-2001  10:50  01d1cb11 --w-  41_gfx5.rom
 524288  DeflatN 179814  66%  03-23-2001  10:50  aeaa3509 --w-  41_gfx7.rom
 ------          ------  ---                                    -------
3473408         1378337  61%                                         12

The only problem is that the individual blocks might not be in the same order.

Is there some way to sort the file list before feeding it into PKUnzip?