Main Restorations Software Audio/Jukebox/MP3 Everything Else Buy/Sell/Trade
Project Announcements Monitor/Video GroovyMAME Merit/JVL Touchscreen Meet Up Retail Vendors
Driving & Racing Woodworking Software Support Forums Consoles Project Arcade Reviews
Automated Projects Artwork Frontend Support Forums Pinball Forum Discussion Old Boards
Raspberry Pi & Dev Board controls.dat Linux Miscellaneous Arcade Wiki Discussion Old Archives
Lightguns Arcade1Up Try the site in https mode Site News

Unread posts | New Replies | Recent posts | Rules | Chatroom | Wiki | File Repository | RSS | Submit news

  

Author Topic: HiToText (Version 2010.11.4).  (Read 151511 times)

0 Members and 1 Guest are viewing this topic.

Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
HiToText (Version 2010.11.4).
« on: August 21, 2008, 12:33:29 pm »
Download links:

HiToText w/XML
HiToText.exe
HiToText.xml
What's new

Use the hiscore.dat-astic for the most up-to-date hiscore.dat file:

The source is now no longer downloadable, it can be accessed through Sourceforge SVN. Instructions can be found here.

Cananas has written up some documentation that is much easier to understand than my rambling posts when implementing an XML entry (They are only a little outdated, the majority of the information is in these documents. See my supplemental posts or the what's new from 4-28-2010 onward):

Schematics
Documentation

HiToText is a command line app that allows you to convert .hi files to text and spits it to stdout.

The current version of the hiscore.dat file we're using to generate the .hi files that we are deciphering is also attached. I'm attaching it mostly so we can ensure that the .hi files being used are the same, and if we decide to add some games that aren't currently supported. (Monster Farm Jump anyone?  ;D )

However, we need help deciphering individual games' hi scores in memory before these can ever be complete.

To decipher a game's memory, start by opening the .hi file for a game in a hex editor such as WinHex, and try and determine how the scores are stored for the game. You can look at NOPs posts below to see a good example of how to explain the game's memory format here so we can convert the explanation into code. Every game can be different, so it's going to take a lot of work, and we'll need help from more than one or two people, so if you want to share your hi scores with all of your friends for your favorite game, do us a favor and decipher it, and we'll include it for you. Below is a list of games we've currently deciphered.

I'm sure more people would like to help if they thought they could, and while recruiting my non-technical brother, I realized I should explain what he does to help me get some of these games decoded just a little bit faster.

Basically, I'm getting pretty good at actually decoding the .hi files. Games like Road Runner (Atari) would be difficult if I hadn't already put the time in on Marble Madness which use the same space saving score and name coding.

Because of this, more of my time getting a game into HiToText is usually ensuring that all the possible characters allowed in the name of the hiscore are taken into consideration. So I'm regularly beating hi scores and putting my name as: ",.-" to see what these characters map out to.

What I've got my brother doing, is beating hi scores, and using every possible character that the game allows, taking a screenshot of all the high scores, and e-mailing those screenshots and the .hi file to me, so I can spend more of my time programming and decoding the .hi file.
One important note about this, I never need all the alpha-numeric scores, usually just "ABZ", "abz", and "019", as every game so far has a pattern for these characters, that make needing "CDE" and so on pretty pointless.

For something like Captain America and the Avengers, the game only stores 5 scores at a time, but has a lot of characters that could not be on the same hiscore table at the same time, so my brother took multiple screenshots, and made copies of the captaven.hi file for those screenshots and sent me an e-mail with this information, deleted the old captaven.hi file and started again with the characters that wouldn't fit in the first hiscore table, I ended up with 4 different captaven.hi files, but was able to capture every possible character allowed with the screenshots.

I'm hoping that some of you that have games you'd like to see decoded would be willing to upload their hiscore screenshots, and .hi OR .nv files to this post. Please zip them up as this board does not actually accept .nv or .hi extensions, and converting these to .txt corrupt them.

A Google spreadsheet is listed here that shows information about what games are deciphered, what games people are working on, and what games we need help with to decipher.

Google Spreadsheet

If you'd like to have access to edit this spreadsheet, as you'd like to do work without someone duplicating it, or have information about a particularly difficult game to decipher, please post here or e-mail me and I will give you access to modify the spreadsheet.

Original post:
Code: [Select]
I'm not entirely sure if this is the right place to ask this, but since the hi score post is stickied in this forum, it made sense to me.

I am looking for a way to covert the .hi files that are generated with the hiscore.dat file into text readable values, such as "NLA, 10000, DEW, 7900".
I've tried doing the conversion myself however I am not versed enough in C to really understand how this data is being written to the .hi file.

I'm reading in the file as byte values, but after that I'm having no luck, I'm hoping someone here has, and knows how to parse these files externally.

My reasoning for this, is I'd like to create a little app that would parse these scores and submit them to a webserver I have,
so my friends and I with MAME could compete against each others scores without sharing a copy of these .hi files.

People I'd like to thank for helping this project (alphabetical order):

  • BadBoyBill - Incorporated HiToText into Hyperspin.
  • bLAZER - Decoded many games for XML format.
  • Cananas - Found Mr. Driller score table; updated the hiscore.dat and decoded many games. A HUGE HiToText contributor.
  • chillinmame - Supports the HiToText genre wheel for HyperSpin on every HiToText update
  • dna disturber - Deciphered many games, updated the hiscore.dat. A HUGE HiToText contributor.
  • headkaze - Created much of the basic structure and very good at guiding me in the right direction. Also integrated HiToText into CPWizard.
  • NOP - Decoded many .hi files and really helped motivate me to continue this project.
  • redhorse - Deciphered R-Type, and Slap Fight.
  • RetroBorg - Decoded .hi formats for Juno First, Berzerk, and River Patrol.
  • Tom Speirs - Incorporated HiToText into GameEx.
  • Wob - Deciphered many games.
  • Zallax - Deciphered U.N. Squadron, Tron, and Discs of Tron
  • And to all the testers, thanks for finding the bugs.
« Last Edit: November 04, 2010, 11:39:32 am by Fyrecrypts »
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

NOP

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 916
  • Last login:September 19, 2017, 08:22:27 pm
  • I stole my avatar.
    • winterMAME
Re: .hi file conversion.
« Reply #1 on: August 21, 2008, 01:42:03 pm »
can you post one of your .hi files here?  I'll take a look at it and I bet we can figure it out, especially if you know what it's supposed to look like to begin with.  Post all the details you can about what the initials and scores are.
« Last Edit: August 21, 2008, 01:43:55 pm by NOP »

MKChamp

  • Trade Count: (+1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 502
  • Last login:February 07, 2024, 02:45:53 pm
  • Stil around.
Re: .hi file conversion.
« Reply #2 on: August 21, 2008, 02:34:46 pm »
I think the problem is that there is no set format in the file. Games save/load the data in different structures...so, dkong.hi would be different than galaga.hi. A simple example is a game that saves 5 highscores vs. a game that saves 50.

Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #3 on: August 21, 2008, 05:08:41 pm »
I've uploaded a random .hi file I have, but I actually don't know the values as I'm at work, and not willing to run MAME. =)

I believe if I can "crack" the encoding I can come up with an algorithm (or heuristic if need be) that can get me what I want.

Edit: I just realized that you could probably find these hi scores on your own, as they are just the default hi scores for 1942.

Edit 2: Removed 1942.hi.txt attachment... It's not really necessary anymore.
« Last Edit: September 06, 2008, 08:24:22 pm by Fyrecrypts »
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

NOP

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 916
  • Last login:September 19, 2017, 08:22:27 pm
  • I stole my avatar.
    • winterMAME
Re: .hi file conversion.
« Reply #4 on: August 21, 2008, 07:44:55 pm »
well, it's starting to unwravel

The default scores for 1941 are:

40000 (c) CAPCOM
35000 ALL
30000 RIGHT   {sic}
25000 RESERVED
20000 EXEDEXES

Looking at the hi file in winhex shows that:
bytes 0 = rank?
bytes 1-4 are the score.  It doesn't appear to be byte swapped.
bytes 5-0dh (maybe?) are a representation of the ascii of the name. 
then it looks there are 3 null bytes and the scores continue. 

Here's the alphabet in hex:
0ah = a
0bh = b
0ch = C
0dh = D
0eh = E
0fh = F
10h = G
11h = H
12h = I
13h = J
14h = K
15h = L
16h = M
17h = N
18h = O
19h = P
1ah = Q
1bh = R
1ch = S
1dh = T
1eh = U
1fh = V
20h = W
21h = X
22h = Y
23h = Z
30h = space
38h = (c)

It'll certainly take some more work deciphering, but that might get something started.  We'd need to look at some more .hi files and see if any of this matches, and if not, MKChamp has accurately predicted our doom.

Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #5 on: August 22, 2008, 09:35:00 am »
Edit: Post removed (it had bad/old information)
« Last Edit: September 06, 2008, 08:25:15 pm by Fyrecrypts »
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #6 on: August 22, 2008, 01:40:58 pm »
Did a little more research on the way the hiscore.dat file is used, and the future looks bleak or long depending on how you look at it.

Every game can store the hi scores and all the other information any way they please, and every game will be different... To the amount of characters allowed for names, to the way the score is stored.

In order to do this correctly, I will need to break down how every single game in hiscore.dat stores it's hi scores in memory.

I'd probably need some help, as there's no way I could do all this on my own.
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #7 on: August 22, 2008, 02:28:34 pm »
Thank you NOP, I've got 1942 working. =)

Now to create a 1942.hi file programmatically, and I'm one step closer to my master plan.

By the way, I'm more than willing to share my code when it becomes more useful, right now it's just a few very hardcoded functions.

When it actually starts actively scanning the hi score directory, submitting scores to an online MySQL database, retrieving scores from the same database, and creating new .hi scores on it's own, it'll be worth releasing.
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

NOP

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 916
  • Last login:September 19, 2017, 08:22:27 pm
  • I stole my avatar.
    • winterMAME
Re: .hi file conversion.
« Reply #8 on: August 22, 2008, 02:48:56 pm »
bummer that all the files are different.
Not surprising at all, but disappointing for sure.  I'd think that some games from the same era, manufacturer should be the same.

This is a really cool idea, sorta like having your own private twin galaxies between your friends, all online.  It would be great if not only could a web based scoreboard be displayed for a given game, but having a way for the .hi file to be copied directly to a player's machine whenever there is a new entry.  that way you could be offline, still play to win, then upload your scores/.hi files and have the web server sort and maintain the highest.

I wonder if you could also have just a central repository of .hi files and have MAME itself load/save the file across the network?  That won't get you the online display you want, but it would provide for everyone having the latest scores available...

MKChamp

  • Trade Count: (+1)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 502
  • Last login:February 07, 2024, 02:45:53 pm
  • Stil around.
Re: .hi file conversion.
« Reply #9 on: August 22, 2008, 03:40:22 pm »
FYI...Headkaze had pinged me about a month ago with this exact same idea for a program he was pondering on making. I think the issue is that even if you parse the hiscore.dat file for each game, you don't know how that game then in turn interprets the data internally to make sense of it.

Maybe he can chim in with what he's discovered.

Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #10 on: August 22, 2008, 03:43:20 pm »
That's the idea, being able to share the scores. I thought about having it share the .hi files, as that's easy, but if two people are playing at the same time, one person's file would probably overwrite the other, losing the score. This just won't do.

My idea is to do two major things, upload new hi scores to a webserver, and two, take the top scores and recreate the .hi file for anybody with this application, and access to the webserver I specify. No hi scores will ever be lost, and they will be updated whenever someone contributes a new score.

Edit: Yeah, in this case the hiscore.dat file is pretty much useless, except to get some hints about how to parse and recreate the .hi file for each game.
« Last Edit: August 22, 2008, 03:45:34 pm by Fyrecrypts »
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

retrometro

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 487
  • Last login:February 15, 2024, 08:26:04 pm
    • Arcade, mame, retro... and the gp2x.
Re: .hi file conversion.
« Reply #11 on: August 24, 2008, 09:43:07 am »
I'm definitely interested in an app that can run reports of high scores.  It'll be a lot of work to decipher scores from each game but once that happens, I can see how it can be extended to lots of things.  For example, it would be great to see the top score(s) through a FE like MALA as you browse your games... or a report ran after your FE shutsdown so that upon boot the top scores of your favorite games are displayed.

-------- gp2x and retro... play it forever! ------------------------
http://www.youtube.com/view_play_list?p=0EE573E86D5A86E0
--------------------------------------------------------------------

NOP

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 916
  • Last login:September 19, 2017, 08:22:27 pm
  • I stole my avatar.
    • winterMAME
Re: .hi file conversion.
« Reply #12 on: August 24, 2008, 10:02:44 am »
By the way, I'm more than willing to share my code when it becomes more useful, right now it's just a few very hardcoded functions.

what are you writing in?  Should help from me come in the form of just deciphering the .hi files, or do you want code written to do the work?  I'm pretty much a C++ programmer only. (well, unless you want to work in x86 assembly, but I doubt that!)

Since this is on a per game basis, we might as well get the most wanted games sorted out first-I suspect not too many people want to share hi scores in Dockman or Disco #1, so can you post your "gotta haves" list?

Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #13 on: August 24, 2008, 10:24:29 am »
Well, I've already done a lot of the coding in C#, so I'll probably just do that.

I think getting help with just deciphering the .hi files would be just fine, no need for it to be in code. =)

I also found that I was wrong with some of my scores above, it's not just TOP... I was very tired that morning I guess. I've attached the screenshots of the scores. It looks like they'll be easier to decode than I had originally thought.

Edit: As for games we should decipher first, I'm thinking the classics really: Donkey Kong, QBert, Pac Man, Ms Pac Man, Joust, etc...
« Last Edit: September 06, 2008, 08:25:41 pm by Fyrecrypts »
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

nullb0y

  • Trade Count: (+5)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 1307
  • Last login:March 27, 2023, 04:37:32 pm
  • Who wants some?
    • antinull
Re: .hi file conversion.
« Reply #14 on: August 25, 2008, 11:28:16 pm »
So heres my big idea
Have a server that holds the highest scores
have a script that downloads the scores compares to yours and uploads yours if they are higher
Alright. Who wants some? Who's next? Huh? How 'bout it? Who wants some? Huh? Who wants to have a little? You.
  You want some more? Huuuh? You want a little? Do ya? Huh? You want some more? Huh? Huh? Hh!

Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #15 on: August 26, 2008, 01:16:41 am »
Post #1 and #11...
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

headkaze

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2943
  • Last login:August 14, 2023, 02:00:48 am
  • 0x2b|~0x2b?
Re: .hi file conversion.
« Reply #16 on: August 26, 2008, 08:19:00 am »
I'm definitely interested in an app that can run reports of high scores.  It'll be a lot of work to decipher scores from each game but once that happens, I can see how it can be extended to lots of things.  For example, it would be great to see the top score(s) through a FE like MALA as you browse your games... or a report ran after your FE shutsdown so that upon boot the top scores of your favorite games are displayed.

This was basically the idea I had and PMed MKChamp about recently. As you guys have already figured out the hiscore.dat only stores/restores a chunk of memory used to hold the hiscores, nothing else.

We would have to write another dat file which took this chunk of memory and broke it up into Player Name / Scores which would be locations in the chunk of memory for each hiscore entry.

The rest is trivial I guess writing software to support the displaying and sharing of hiscores, so yeah the hard part is getting another hiscore dat with all the locations for each hiscore entry.

Here is the hiscore.dat entry
Code: [Select]
; <gamename>:
; <cpu>:<address>:<length>:<value to wait for
; in the first byte/word>:<value to wait for in the last byte/word>

There is also a couple of interesting tutorials on how the hiscore's are obtained.

But for simplicity sake I think NOP's idea is worth a look at. The only problem I see with that is the whole system couldn't be automated. Each .hi file would have to be examined, and what if there are two top hiscores in two different .hi files, there would have to be a way to merge them into one file. So I think even with this idea, you need a new dat file with all the hiscore entries defined. Then there is hope of automating a system like this, as well as being able to display actual hiscores on the website. Also people would have to register 3 character names on the website that will be accepted. Alot of work but I think it's a good idea. Alot of the hard work would be creating this new dat file.
« Last Edit: August 26, 2008, 08:24:32 am by headkaze »

Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #17 on: August 26, 2008, 09:53:09 am »
I'm hoping to have the automate stuff done by the end of the week, and you don't want to have one person have just a 3 character name be accepted, at least I don't. If my brother comes over to play, and beats everyone's Dig Dug, I want everyone to see his initials too.

I'm solving this problem, by having the webserver information stored in a settings.xml file and anyone I want to access the hi scores can use the address I give them with the application. As for the new .dat file, I would propose this as a standard:

<gamename>:
<numOfScoresSaved>:
<nameOfStatToBeSaved>:<statToBeSaved>:<#ofbytes>:<someTypeOfByteFormat>
(Repeat the above as necessary for names/initials, rank, scores, floor, character, world, etc...).
<Any important trailing bytes>

Here's the problem though, certain games are not as easy to breakdown as the above. If you look at msword.hi you'll see that there's an extra byte around the third score that I can't account for. It's an extra 0D, I believe, right before where I would guess is the floor reached bytes, and it offsets all the other scores by that one byte. How do you account for that in the above new .dat file? In addition we'd need something to explain the format of the bytes, hi scores themselves are usually just the literal digit representation (despite the bytes being in hex). So the 4 bytes: 00-50-00-00 is 500,000 and not the hex to decimal equivalent: 5,242,880. Certain games allow lower case characters (Super Mario Bros), certain games allow the © symbol (1942), and so on...

I'm thinking it's going to need to be semi-hardcoded for each game, or at least each game that can't be described by the above schema. Like I said, I'm more than willing to share the source when it's completed. In the meantime, if you want a game included in this application, please try and break down the game's method for storing the scores in memory here. The more information you can provide the better, in particular a character mapping to bytes for each allowable character that the game uses for storing names.

I know this is going to take a lot of work, but it has the potential to be a very cool application, and one that everyone can actually help with.
« Last Edit: August 10, 2009, 12:04:34 pm by Fyrecrypts »
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

headkaze

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2943
  • Last login:August 14, 2023, 02:00:48 am
  • 0x2b|~0x2b?
Re: .hi file conversion.
« Reply #18 on: August 26, 2008, 10:27:39 am »
Fyrecrypts: Not sure what you mean by the hex issue. If your trying to figure out what bytes do what in the .hi files I suggest you take a look at tutorials on the hiscore dat website and use the Mame debugger to view the data in memory. Although I think in most cases viewing the .hi file in a hex editor (my personal favourite is HxD) should be good enough. There might be some cases where numbers are stored in big endian which will mean the bytes will need reversing. There may be some cases like this where it's difficult to describe in a dat file but it may be possible. Some ways the data is stored could be dependent on the game, the CPU and driver. Some games may use the same format as their parent ROMs.

If you write a program and hard code it, that would be fine, just make it a command line application that can accept a .hi file as a parameter and output the hiscore table to stdout in a standard format. Then with that any FE or software can use this util to grab the hiscore list from a .hi. So I'm not against hard coding it into a program, and lets face it it will be alot easier hard coding each game rather than writing some complex dat file as long as it's maintained I wouldn't even care if the source was released. Come to think of it a dll may be even better, but anyway you get the idea.

Let's not rule out the idea of a dat file though, see how hard it is hard coding it first and after a few games you might realise it would be possible or too hard to make a dat. For now I support the idea of a program and even if we do go for a dat file in the end, the program may even be modified to output the games it supports to a dat. Either way you should learn a few things about how hiscores are stored, and I have a feeling a dat file may be too hard create anyway. Also if we do go the dat file way and it becomes complex to parse it, there seems like no point going that way anyway.
« Last Edit: August 26, 2008, 10:29:57 am by headkaze »

Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #19 on: August 26, 2008, 11:51:47 am »
Edit: Post left so the rest of the thread makes sense, but just know this issue was only because .hi files converted to .txt files and then uploaded caused some extra stuff to be included in the file resulting in strange .hi files.

This is the hex issue:

Score:   00-50-00-00- (500,000)
Name:    08-0C-2C- (EGW)
Floor:   00-0C- (might just be 0C) (12)
Rank:    04- (big hulking guy)
Unknown: 00-00-00-00-00-00-

Score:   00-45-00-00- (450,000)
Name:    30-00-1A- (YAN)
Floor:   00-0B- (11)
Rank:    04- (big hulking guy)
Unknown: 00-00-00-00-00-00-

Score:   00-40-00-00- (400,000)
Name:    0E-00-04- (HAC)
Floor:   00-0D-0A- (10) (Why the extra 0D?)

Rest: 04-00-00-00-00-00-00-00-35-00-00-0C-30-1C-00-09-04-00-00-00-00-00-00-00-30-00-00-0E-1C-24-00-08-00-00-00-00-00-00-00-00-25-00-00-04-00-1E-00-07-00-00-00-00-00-00-00-00-20-00-00-04-1C-18-00-06-00-00-00-00-00-00-00-00-20-00-00-04-00-1E-00-05-00-00-00-00-00-00-00-00-20-00-00-04-1C-18-00-04-00-00-00-00-00-00-00-00-20-00-00-04-00-1E-00-04-00-00-50-00-00


As you can see there's an extra byte to describe the floor for the third score, all the other scores line up fine if you break it down in the format I have for the first two scores (4 score bytes, 3 name bytes, 2 floor bytes, 1 rank byte, 6 zero padded bytes). How do you explain in the .dat for that extra 0D?

In addition, the character representation for 'A' is 0A in 1942, but 00 in msword, and it's not just an offset, the character representation in msword for 'G' is 0C, and for 'H' is 0E, what is 0D? in 1942 0D is 'D'. So we would need to have some sort of translation table in the new .dat as well that could convert these values per game.

The app I plan on working on would have the side effect of having an xml file containing all the current codes in the .hi file, but I'm not sure if I'd do the job of breaking it out and making a standalone app/dll file that would convert a .hi file to stdout. That may end up being the best solution however.
« Last Edit: September 06, 2008, 08:28:15 pm by Fyrecrypts »
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

NOP

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 916
  • Last login:September 19, 2017, 08:22:27 pm
  • I stole my avatar.
    • winterMAME
Re: .hi file conversion.
« Reply #20 on: August 26, 2008, 12:01:11 pm »
I agree that you're pretty much going to have to hard-code each game on a case by case basis.  That's not such a bad thing, and if something is worth doing, it's usually worth doing twice, so if we start hard coding now and discover that it's really something that could be explained via a formatting xml, then we change it in later revisions.  Not that big of a deal. 

I'm all for decoding more .hi files; I've just been exceptionally busy the last few days, but count me in for the long haul.

I also like a command line switch type app, so that any FE could access it, but then I'm a DOS guy to begin with. ;)

Unknown bytes in the file I think are also OK at the moment.  Since we're only reading these files to output them to another format, and not generating our own (that's a future project) we just skip the stuff we don't understand.  It'll come around.

Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #21 on: August 26, 2008, 01:03:10 pm »
I've whipped up a quick .hi file to stdout/.txt file/whatever, and included the code in case you'd like to add/change outputs.

Edit: App removed here, but can be found in the main post.
« Last Edit: September 06, 2008, 08:29:10 pm by Fyrecrypts »
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

headkaze

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2943
  • Last login:August 14, 2023, 02:00:48 am
  • 0x2b|~0x2b?
Re: .hi file conversion.
« Reply #22 on: August 26, 2008, 05:40:22 pm »
Here is a slightly different approach. First change is to use structures to store the data which I think will make it easier in the long run. I also took more of an OOP approach by using a super class that all other game hiscore classes can inherit from. Your welcome to use the code or not but it just shows another method to accomplish the same thing. (It's also in C# my language of choice also) ;)
« Last Edit: September 06, 2008, 08:03:05 pm by headkaze »

Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #23 on: August 26, 2008, 05:58:21 pm »
That is much better than what I rigged up HeadKaze, I will be using your version from here on out.
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

headkaze

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2943
  • Last login:August 14, 2023, 02:00:48 am
  • 0x2b|~0x2b?
Re: .hi file conversion.
« Reply #24 on: August 26, 2008, 06:39:12 pm »
That is much better than what I rigged up HeadKaze, I will be using your version from here on out.

No worries, glad your a C# coder as well :) BTW When I was first trying to parse the 1942.hi file I was using the one attached to this thread. I don't know what happened but the file must have got corrupted because I couldn't parse it right using the info NOP provided. Had me stumped for a while. I ended up creating my own 1942.hi in Mame and then I could parse it. Strange, so I don't recommend anyone to use hiscore files posted here as text files. If you do upload a hiscore file it's probably best to zip them up first ;)

headkaze

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2943
  • Last login:August 14, 2023, 02:00:48 am
  • 0x2b|~0x2b?
Re: .hi file conversion.
« Reply #25 on: August 28, 2008, 02:36:33 am »
I just wanted to mention that anyone can help with this. If you read the tutorials on how to view Mame's memory in the debugger here and get yourself a copy of a nice Hex Editor like HxD all you need to do is find the offsets in the .hi file. Take a look at NOP's post on the format we need then zip up the ".hi" file along with a post with the offsets and me or Fyrecrypts can write the class to deal with the game.

If we don't get people to help do that then this project will probably die. That would be a shame because I think it's a great idea.

Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #26 on: August 28, 2008, 09:04:49 am »
I agree with Headkaze completely, there are a tremendous amount of games out there, and it would simply be just too much for even just the two or three of us that have posted in this thread.

On another note, I am fairly close to finishing up the part that actively scans your hi score directory and compares it with an online database of scores. I will post that code as soon as I'm finished as well. When that's done, I will focus more on cracking individual games.
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #27 on: August 30, 2008, 11:09:07 am »
I've discovered a little more about 1942.hi, the scores in the file can basically be arranged in any order and the game uses what we call rank to determine the order.

The last 10 bytes correspond to the high score always displayed. The first of those 10 corresponds to "today's top %", the next 8 is the score, the last byte is the start stage?

In addition, the first of the 3 bytes that we deemed unknown corresponds to the start stage as well (the default scores were all 0).

Also, I have made a lot of progress on my webserver app, and was going to release it today, but after seeing this new info, I need to change some things.

In addition there are 25 scores saved, not 24.
« Last Edit: August 30, 2008, 10:12:13 pm by Fyrecrypts »
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

retrometro

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 487
  • Last login:February 15, 2024, 08:26:04 pm
    • Arcade, mame, retro... and the gp2x.
Re: .hi file conversion.
« Reply #28 on: August 30, 2008, 05:12:15 pm »
Looking forward to the public beta.


-------- gp2x and retro... play it forever! ------------------------
http://www.youtube.com/view_play_list?p=0EE573E86D5A86E0
--------------------------------------------------------------------

headkaze

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2943
  • Last login:August 14, 2023, 02:00:48 am
  • 0x2b|~0x2b?
Re: .hi file conversion.
« Reply #29 on: August 31, 2008, 09:47:27 am »
Hey Fyrecrypts I was just thinking perhaps we should put out a request for people to post thier .hi files if they have a decent hiscore or progression in a game. If we get hiscores that are at higher levels it will probably help deduce what some bytes are for.

Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #30 on: August 31, 2008, 11:34:36 am »
Yeah, anything to help. =) It looks like using initial .hi files is going to be misleading like it was for 1942. It's going to take a combination of new and older .hi files, and watching them change to really know what's going on.

In addition, I have attached the source and executable for my webserver app. If you do not have webspace you can use with MySQL what I've given may seem kind of pointless. I debated about sharing my webspace for everyone, and just figured it was too easy to abuse, but I've included the php files that I'm using on my webserver and I'll share the database schema I used.

First, I know that what I've written can be programmed better, I didn't go through and optimize the code like I would've had it been a professional application, I work full time, and have a 6 month old son, so programming time right now is very rare. =) Second, I put absolutely no error handling, and have not included many standard features that should come with this again because of my lack of time, but that's why I share the code, so someone else with more time can improve on what's there.

Now, the database schema I used was naming tables in my database by the name of the game, in this case 1942. I had fields for each field stored in the .hi file, so for 1942 it's Rank, Score, Name, and Stage. In addition, I added my own field of TimeAdded as this is something I want to expand on later. That's pretty much it for the database schema.

Lastly, enjoy! And now we need to get the other games .hi files cracked!

Edit: App removed here, but can be found in the main post.
« Last Edit: September 06, 2008, 08:29:40 pm by Fyrecrypts »
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

NOP

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 916
  • Last login:September 19, 2017, 08:22:27 pm
  • I stole my avatar.
    • winterMAME
Re: .hi file conversion.
« Reply #31 on: September 02, 2008, 07:19:32 pm »
here's file info for galaga.hi:

Bytes 0-5  = score #1
Bytes 6-B = score #2
bytes c-11 = score #3
bytes 12-17 = score #4
bytes 18-1c* = score #5
Byte 1d = 24h
Bytes 1e-20 = 3 initials for score #1
bytes 21-23 = 3 initials for score #2
etc.
Then 6 bytes where the hi score is repeated, same formatting as byes 0-5.


* yes, that's only 5 bytes for the last score, and 6 for everyone else.  I don't quite understand how that worked out, and that 24h is not a part of the #5 score.   I'm not sure if the scores are variable size, or what the deal is to explain that. 


Scores are stored in reverse byte order, but are binary coded decimal.
In other words, the score of 605690 (my own hi score woo!) ends up being coded like this:

Byte 0: 00
Byte 1: 09
Byte 2: 06
Byte 3: 05
Byte 4: 00
Byte 5: 06


The initials use the same alphabet method as in 1942.
« Last Edit: September 02, 2008, 07:55:03 pm by NOP »

NOP

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 916
  • Last login:September 19, 2017, 08:22:27 pm
  • I stole my avatar.
    • winterMAME
Re: .hi file conversion.
« Reply #32 on: September 02, 2008, 07:54:20 pm »
donkey kong (dkong.hi)

Looks like each player's data is 22h bytes each:

bytes 0-4 = Looks like some pattern, for me it's 94,77,01,23,24
bytes 5,6 = 10h,10h
bytes 7-0c = score #1
bytes 0d, 0e = 10h,10h
bytes 0f-11 = initials #1
bytes 12-1a (9 bytes) are all 10h
bytes 1b = 3Fh
bytes 1c,1d = 00
bytes 1e-21 (for me) were 93,02,f4,76

Then it starts for #2:
bytes 22-26 = some patterny thing again - 96, 77, 02, 1e, 14
bytes 27,28 = 10h,10h
bytes 29-2e = score #2
bytes 2f-30 = 10h,10h
bytes 31-33 = initials #2
bytes 34-3c (9 bytes) = 10h
byte 3d = 3fh
bytes 3e, 3f = 00
bytes 40-43 (for me) were 76, 02, f6, 76

#3's patterny thing is: 98, 77, 03, 22, 14

I think it's obvious that the 3rd byte there is the rank.


initials are stored like this:
11h= A
12h= B
13h= C
14h= D
15h= E
16h= F
17h= G
18h= H
19h= I
1ah= J
1bh= K
1Ch= L
1dh=M
1eh=N
1fh=O
20h=P
21h=Q
22h=R
23h=S
24h=T
25h=U
26h=V
27h=W
28h=X
29h=Y
2ah=Z

Scores were stored in BCD format, not byte swapped.
So score #1 was 029300 and stored in bytes 7-0c as: 00,02,09,03,00,00


Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #33 on: September 02, 2008, 08:32:50 pm »
Good timing, I just fixed up 1942, it's complete. I also fixed a bug with some scores not being correctly uploaded. I removed the old version above, and attached the new. I'll try and get to the new games NOP deciphered tomorrow, thanks!
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

headkaze

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 2943
  • Last login:August 14, 2023, 02:00:48 am
  • 0x2b|~0x2b?
Re: .hi file conversion.
« Reply #34 on: September 02, 2008, 09:39:55 pm »
Here's a start on the galaga class. There is more to that 0x24 byte than first realised. For example have a look at a freshly created galaga.hi and every score has 0x24 on the last byte of the score. Right now I just ignore the 0x24 byte, so we will have to figure out what it means and deal with it eventually.
« Last Edit: September 06, 2008, 08:03:26 pm by headkaze »

NOP

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 916
  • Last login:September 19, 2017, 08:22:27 pm
  • I stole my avatar.
    • winterMAME
Re: .hi file conversion.
« Reply #35 on: September 02, 2008, 09:46:46 pm »
donkey kong jr (dkongjr.hi)

very similar layout:


bytes 0-4 = Same patterny pattern, for me it's 94,77,01,23,24
bytes 5,6 = 10h,10h
bytes 7-0c = score #1
bytes 0d-12 = 10h
bytes 13-15 = initials #1
bytes 16-1a (9 bytes) are all 10h
bytes 1b = 3Fh
bytes 1c,1d = 00
bytes 1e-21 (for me) were 76, 02, f4, 76

Then it starts for #2:
bytes 22-26 = some patterny thing again - 96, 77, 02, 1e, 14
bytes 27,28 = 10h,10h
bytes 29-2e = score #2
bytes 2f-34 = 10h
bytes 35-37 = initials #2
bytes 38-3c (5 bytes) = 10h
byte 3d = 3fh
bytes 3e, 3f = 00
bytes 40-43 (for me) were 39, 02, f6, 76

#3's patterny thing is: 98, 77, 03, 22, 14

so, same patterny thing as DK.  makes sense, but interesting that it doesn't change at all between these 2 games.

Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #36 on: September 03, 2008, 12:57:39 pm »
For Galaga, I'd guess that the 6th byte in the score if it's not a 6 digit score is going to be 0x24, so we can safely ignore the 6th byte in a score if it's this value, otherwise it's the hundred-thousand digit.

I am at work and can't do testing on this, but I have a feeling it's a blank hex value so when galaga is displaying the scores it doesn't show a leading 0 for 5-digit hi scores.

Edit: Each of these hex values represents some sprite graphics to display. The programmers probably decided to map 0x00 to the graphics for a '0' and so on. I believe 0x24 maps to a space (' '). Put a 0x24 in the middle of a score and I'd bet it shows up in the score as a space.
« Last Edit: September 03, 2008, 01:38:33 pm by Fyrecrypts »
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

NOP

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 916
  • Last login:September 19, 2017, 08:22:27 pm
  • I stole my avatar.
    • winterMAME
Re: .hi file conversion.
« Reply #37 on: September 03, 2008, 02:54:44 pm »
correct me if I'm wrong, but I don't think pacman and mspacman have hi score files.  there's no top 5 anyway, just the highest score since power up.  No initials are ever entered.

joust didn't seem to have one either-it does have an nvram file though, so I suspect defender and stargate, probably robotron will be similar.

IIRC, qbert also doesn't have a .hi file.

I may also be using an outdated copy of mame for my research, which could explain some of these missing files.

if you have any other requests, let me know, otherwise I'll just decipher some that look easy. ;)


Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #38 on: September 03, 2008, 02:55:45 pm »
I've decided I'm willing to open up my webspace for people to use the hi-score scanner app. If you PM me, I'll PM back the details, I'd like to do it this way so I can keep track of how many people have the info, I really don't have a ton of bandwidth so I have to pretty strictly regulate this. Sorry. =/

Edit:

NOP: Yeah, certain games don't use .hi files, I mentioned a few specific games without checking the .dat file. mspacman however does use .hi files, but like you said it only stores one score so it should be pretty easy, in fact I think the whole file is only 11 bytes.

You can see which games would create .hi files by looking in the hiscore.dat file (just open it up in any text editor). For mspacman the .dat file shows:
00:4E88:04:00:00
00:43ED:06:40:40
00:43D1:01:48:48

The third value, delimited by colons, is the only really important thing to us, as it shows how many bytes are used, all three of these memory locations let us know that there are 11 bytes in a .hi file, and probably 3 distinct pieces of information for us to use.
« Last Edit: September 03, 2008, 03:19:29 pm by Fyrecrypts »
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext

Fyrecrypts

  • Trade Count: (0)
  • Full Member
  • ***
  • Offline Offline
  • Posts: 237
  • Last login:August 19, 2015, 10:50:47 am
  • .hi guy
Re: .hi file conversion.
« Reply #39 on: September 03, 2008, 05:38:43 pm »
Corrected galaga to account for the 6th byte, and I also whipped up msword.

Edit: Finalized msword. Galaga is finalized a few posts down.

Edit 2: File removed, and put into the main post.
« Last Edit: September 06, 2008, 08:30:52 pm by Fyrecrypts »
Help us crack more .hi files here:
http://forum.arcadecontrols.com/index.php?topic=83614.0
Generate the newest hiscore.dat here:
http://www.hitotext.com/HiScoreDat/index.php
Sourceforge project page:
https://sourceforge.net/projects/hitotext