Software Support > DOS/WinCab
Libraries
Chris:
--- Quote from: ex_directory on April 03, 2008, 11:34:56 am ---Above you say your proposal for using filters in memory will be slow, are you just referring to the startup procedure or the usage of wincab? I know it costs time but I wonder if a direct performance comparison would yield the sqlite solution to be quicker?
--- End quote ---
I'm saying an unindexed filter would be slow on a large library, and the same would be true for SQLite if I don't index it either, although creating SQL indexes is a lot easier. The big issue is just having to build new database features every time I want to add something, and if I want to do it myself vs. using an SQL-based solution. It's a matter of "core competencies"... DWJukebox is a jukebox and not a database engine, so do I want to spend my time writing and debugging indexes or adding features to skins?
Really, the only reason I'd even consider it is because I was fortunate enough to think ahead and perform all database access through a couple of functions... makes it easy to drop in. But it'd be nice to be able to offer "advanced" users the ability to write filters directly in SQL... then there's no limit to what they can do.
The database is actually very flat now. Effectively there are only three tables: the master song list, a list of integers of "avalable" songs (this is the list I sort and manipulate), and a list of paths. CD skins have two more tables: a list of album artists and a list of paths to CD cover art. All of the data tables such as symbol and color names and values, etc. are hard coded; I used to load them from a datafile but then I ran into problems where someone would update the executable but not the datafile, so I just merged it all in.
ex_directory:
--- Quote from: Chris on April 03, 2008, 08:30:04 pm ---I'm saying an unindexed filter would be slow on a large library, and the same would be true for SQLite if I don't index it either, although creating SQL indexes is a lot easier. The big issue is just having to build new database features every time I want to add something, and if I want to do it myself vs. using an SQL-based solution. It's a matter of "core competencies"... DWJukebox is a jukebox and not a database engine, so do I want to spend my time writing and debugging indexes or adding features to skins?
--- End quote ---
I understand where you are coming from now. I guess the answer to the issue of amount of debugging/development on indexes would come down to the design of the helper functions etc that support functions to get indexes.
In reality, categories such as decade/genre etc would have permanent indexes that gives benefits of speed of startup (if nothing changed), switching between categories during runtime etc. But a song search etc would probably be a temporary array of song indexes created on the fly. It is probably the performance of the latter that is most important i.e. sql v your own memory based solution.
The only thing I can offer is the similarities to the program I needed to write. If I would have had to support search functions, I would have created function(s) to create indexes on the fly, i think my master array had <10000 items in it, and in terms of a single scan to check each entry for a match be it on one field match or multiple (effectively a sql query), I would have expected to be fast enough that the delay before results were displayed would not be an issue. If the performance was fast enough you could actually have all indexes created on the fly as they are required, this might only mean 1 or 2 more public functions you need to develop to provide that functionality.
We use paradox files at work to maintain a list of records in a buffer so we have state if the app crashes. This is very simple, a file, a list but you would not believe the problems this has caused us over the years. But I partly blame the developers (Borland), for that.
:soapbox: Sounds like I am saying avoid the SQLite approach, I'm not!! Just trying to offer my experiences.
FWIW, I spend ages adding ID3 stuff to my album mp3's. Vista is quite cool for playing around with that as it displays ID3 stuff as columns in windows explorer. What I also like about Vista is the ability to tag files, usually pictures with some names, e.g. Kids, Home, Friend x, Friend y. It doesn't work as well with music, but, one could tag based on stuff like who owns it (mine or the missus, kids etc), was song released, bpm (dj etc) and probably all kinds of other stuff that people have for their collections. Your SQLite solution could offer the ability to cater for this.
So anyway, we have probably talked this subject too much, I should leave you alone to develop ;D
--- Quote from: Chris on April 03, 2008, 08:30:04 pm ---Really, the only reason I'd even consider it is because I was fortunate enough to think ahead and perform all database access through a couple of functions... makes it easy to drop in. But it'd be nice to be able to offer "advanced" users the ability to write filters directly in SQL... then there's no limit to what they can do.
--- End quote ---
>:D Being devil's advocate, would offering those users ability to write sql, provide more complex filter arrangements not possibly cause more development than the couple of funtions you have available?
--- Quote from: Chris on April 03, 2008, 08:30:04 pm ---The database is actually very flat now. Effectively there are only three tables: the master song list, a list of integers of "avalable" songs (this is the list I sort and manipulate), and a list of paths. CD skins have two more tables: a list of album artists and a list of paths to CD cover art. All of the data tables such as symbol and color names and values, etc. are hard coded; I used to load them from a datafile but then I ran into problems where someone would update the executable but not the datafile, so I just merged it all in.
--- End quote ---
I had kind of worked this out, and for my 2 pennies, it all makes sense based on the problems you needed to overcome and performance you wanted from the solution.
Chris:
--- Quote from: ex_directory on April 04, 2008, 04:43:50 am ---I understand where you are coming from now. I guess the answer to the issue of amount of debugging/development on indexes would come down to the design of the helper functions etc that support functions to get indexes.
--- End quote ---
I don't have my code here in front of me, but no "fields" are accessed directly. Everything goes through a function like jb_getsongstring(intSongID,DB_FIELDNAME) or jb_getsongint(intSongID,DB_FIELDNAME). Really comonly used fields like artist have their own functions like getartist(intSongID), but getartist(intSongID) just calls jb_getsongstring(intSongID, DB_ARTIST). In the "database", each song is just a bunch of fields in brackets, like this:
--- Code: ---{145798TAKE}{Take A Chance On Me}{ABBA}{Greatest Hits}{2}{204}{545131}{Pop}{0}
--- End code ---
Those are off the top of my head and may not be in the right order but they are File ID, Title, Artist, Album, Track Number, Song Length, File Timestamp, Genre, and Play Count. So DB_ARTIST evaluates as 3, and so jb_getsongstring loops through 3 sets of brackets and returns whatever is between them. There are, of course, equivalent putsongstring and putsongint functions. So to change over to SQL, all I need to do is change jb_getsongstring to
--- Code: ---sprintf(strQuery, "SELECT %s FROM tblSongs WHERE intSongID = %d", strFieldName(intFieldID),intSongID);
sqlite3_exec(jb_db, strQuery, callback, 0, &zErrMsg);
--- End code ---
The callback function appears to actually iterate through the data, so it is slightly more complicated.
--- Quote ---In reality, categories such as decade/genre etc would have permanent indexes that gives benefits of speed of startup (if nothing changed), switching between categories during runtime etc. But a song search etc would probably be a temporary array of song indexes created on the fly. It is probably the performance of the latter that is most important i.e. sql v your own memory based solution.
--- End quote ---
Right. And it's unlikely I'll implement a true song search because that gets into the whole "on-screen keyboard" issue, although I could do it I suppose with a dynamic version of the T9w system that's used on phones. Essentialy, I'd have to figure out how many keys I have available and distribute the alphabet among them, then at startup create matching indexes. For example, using a 10-key keypad with letters distributed the way they are on a phone, "Take A Chance On Me" comes out to 825322426236663. So as the user starts typing, all matching songs thus far are displayed; four or five keystrokes should be sufficient to narrow down any song.
None of this is rocket science; filtering, indexes and search are Programming 101.
--- Quote ---in terms of a single scan to check each entry for a match be it on one field match or multiple (effectively a sql query), I would have expected to be fast enough that the delay before results were displayed would not be an issue.
--- End quote ---
Well, the tricky part here is that I need to be able to do this in a way that doesn't disrupt the play of music or animation on suboptimal hardware, which is why I'd tend toward preindexing everything. It's spending memory and prep time against run time. The nice thing is that the database is rarely changed, it's usually just added to, and additions are always at the end of the list. So I can save the index files after they're created, and just need to insert data into the appropriate indexes when a song is added. The tricky part there becomes when a song DOES change, I need to extract its data out of the affected indexes and re-add it. But that's rare. And there's the big benefit of writing my own indexes and query code; I can optimize for the most usual case.
--- Quote ---We use paradox files at work to maintain a list of records in a buffer so we have state if the app crashes. This is very simple, a file, a list but you would not believe the problems this has caused us over the years. But I partly blame the developers (Borland), for that.
--- End quote ---
Ironically I was just thinking about this last night: storing the jukebox state in a temporary file and offering to restore it on startup. I already track whether or not shutdown was expected in the log, but dumping the playlist and credit count and offering to reload them would be easy enough.
--- Quote ---:soapbox: Sounds like I am saying avoid the SQLite approach, I'm not!! Just trying to offer my experiences.
--- End quote ---
I'm leaning toward not using it right now.
--- Quote ---So anyway, we have probably talked this subject too much, I should leave you alone to develop ;D
--- End quote ---
Not at all, 80% of the features in the jukebox have come from discussions like this.
--- Quote --- >:D Being devil's advocate, would offering those users ability to write sql, provide more complex filter arrangements not possibly cause more development than the couple of funtions you have available?
--- End quote ---
More development? No, it's simple. More technical support Very likely! :P
--Chris
ex_directory:
--- Quote from: Chris on April 04, 2008, 10:29:54 am ---Well, the tricky part here is that I need to be able to do this in a way that doesn't disrupt the play of music or animation on suboptimal hardware, which is why I'd tend toward preindexing everything. It's spending memory and prep time against run time.
--- End quote ---
Sorry, your right, I have been thinking recent hardware, multi-threading for so long, I can't remember the last time I saw a real DOS prompt - not that I am really a Windows fan either. What do you believe is your user base now between DOS/Windows? I would imagine your continuing support for cross-OS is affecting design and ability to develop functionality you would like to release?
--- Quote from: Chris on April 04, 2008, 10:29:54 am ---Ironically I was just thinking about this last night: storing the jukebox state in a temporary file and offering to restore it on startup. I already track whether or not shutdown was expected in the log, but dumping the playlist and credit count and offering to reload them would be easy enough.
--- End quote ---
Sounds like you have lots of functionality ideas going on! Guess you would need to continuously keep a log of state, just dumping on shutdown may not be sufficient for power down situations.
Chris:
--- Quote from: ex_directory on April 07, 2008, 05:50:52 am ---What do you believe is your user base now between DOS/Windows? I would imagine your continuing support for cross-OS is affecting design and ability to develop functionality you would like to release?
--- End quote ---
I actually think my DOS user base is fairly high, because DWJukebox is essentially the ONLY option for a DOS user. There are a lot of Windows jukes, and let's face it, most of them are better.
DOS is actually the reason DWJukebox exists, as my first MAME cabinet was DOS-based and putting a jukebox on it was the only way my wife would allow it in the house. I am actually not a Windows programmer at all; I still code in old-fashioned procedural-style C code. So I'm really not losing anything. If a break occurs, it may be because I may try to start using Media Player to play audio rather than the library I'm using, which would give me access to WMA files and remove the patent restrictions that keep me from allowing any kind of commercial use. The DOS version would still rely on the old library and thus would not inherit the Media Player features.
--- Quote ---
--- Quote from: Chris on April 04, 2008, 10:29:54 am ---Ironically I was just thinking about this last night: storing the jukebox state in a temporary file and offering to restore it on startup. I already track whether or not shutdown was expected in the log, but dumping the playlist and credit count and offering to reload them would be easy enough.
--- End quote ---
Sounds like you have lots of functionality ideas going on! Guess you would need to continuously keep a log of state, just dumping on shutdown may not be sufficient for power down situations.
--- End quote ---
Or for serious crash situations that I can't recover from to run the AtExit functions. But all of the state information would fit in about 2-3K of space and it only needs to be written out when either a song is added to or removed from the queue (by playing or deletion), or a credit is added.
Navigation
[0] Message Index
[#] Next page
[*] Previous page
Go to full version